I usually write systems for pen-and-paper RPGs, so bear that in mind when reading this.
First, I look at what kind of experience I want to create. Is there something I want to emulate? What sort of pace to I want to keep? How punishing should mistakes be? Should a lot of it be up to luck? You don't need (or want?) hard numbers here, but an overview.
Next, I like to think of how the player interacts with the situation.
Do they pick from an a list of options in order? Do they expend resources? Do they have to pick their moments? Do they try to change the situation to suit them?
I find them quite handy. Map out your combat loop a little. Again, hard numbers are for another pass.
Another thing is to plan out how quantities such as strength, damage, action points and so on link together. This is not the combat cycle, but a map of dependancies.
Write out a sample combat using your flow chart. Make up values that feel right, or use placeholders. What happens? Where did the player get to have an influence? Are there any parts where they could get bored, or where things spiral out of control? Could a lucky hit lock the opponent down completely?
Back to the Charts
Look back at the chart and make any changes you think fit. If you can't follow it at this point, before the implementation intricacies, then it's probably going to be a bit much for a player (remember, you know it better than anyone!). Try pin down a few values.
Narrow Things Down
I like to jot down a few ideas for containers, data structures (well, the pen-and-paper implementations of them), and ways to tie these back to the feel. I also look at single things which could swing the battle in radically different directions, in terms of experience - it may be worth changing them.
Don't get too Attached
Sometimes an idea is really awesome, but hard to reconcile with the rest of the system. Rather than discard it entirely it may be worth smashing it to bits, working out why it works, then rebuilding it.
The Grunt Work
Make a few prototype items, enemies, and so on. Concentrate on modelling archetypes.
More complexity can be added later, as can more subtle differences.
Hope that helps.