After a lot of soul searching I decided to take Option #6 and kill the product. I could've figured out a way to make options 1-5 work if I truly believed in the product and it's mission. Deep down though, sending large files wasn't something I was truly passionate about.
Founder-product fit is a bitch. If you're not enthusiastic (and I mean really enthusiastic) about some fundamental aspect of the product you're delivering, it's not a good product for you.
However, to answer the title question, when should you kill your startup? Probably a lot sooner than you think. Some entrepreneurs give up too early, but they are rare. Most tend to give up far too late - and when they finally throw in the towel and start something new (which is a lot more exciting and fun than flogging a moribund horse), they usually feel relief that they finally made the decision, and regret that they didn't make it sooner.
If you're asking yourself whether you should kill your startup, you probably should kill your startup.