It's hard to make this call. It involves taking responsibility (as well as admitting failure). I favour giving up on moribund ideas rather than flogging them until you're dead (impatience kills startups, patience kills human beings).
Vinicius Vacanti proposes a well thought out framework: first, define your success metric, then keep iterating.
The key takeaway is that if every iteration is improving your success metrics, keep iterating. You only stop iterating when you can't seem to improve it any further.
Read more good stuff here.