Unfortunately, I think a lot of it comes down to experience. If you feel like you're beating your head against a wall and you have no other ideas, it's time to take your baby outside and shoot itget rid of the idea.
On the bright side: you can continue to come up with other ideas. The real limiting step here is not usually ideas, but time to work on them. If you kill one game idea, it gives you the time to work on your others. If you have no other ideas... well, then I guess you keep working on your current one until a new idea occurs to you that you want to try.