I'm working on two RPG battle mechanic ideas. I haven't chosen a game engine yet. Before starting to write code, how should I plan their details?
Should I draw and write on paper? Should I write a specification document? What game mechanic planning processes have others used?
The first (unasked) question is, should you be designing the battle mechanics of an RPG before any code has been written, in the first place? Probably, yes -- but be aware that if you're making an RPG on your own or in a small team, it is not the mechanics or game engine that is the challenging part, but the massive loads of sheer content you have to create in terms of every single level, character, NPC, dialogue tree, weapon, spell, treasure chest... I don't know you or your team, but I run into a lot of students who want to make RPGs because they love to play them, so my first instinct is to make sure you know what you're getting into!
Second, yes, I'd say that getting the game mechanics down on paper is a good first step. A good second step would be putting them in a state where you can actually test them "by hand" without writing code -- paper prototyping, in other words. Play out your combat system (or a simplified version if needed) as a tabletop game and see if it gives the player interesting decisions to make, and modify the mechanics accordingly until you reach the point where it works well. When the mechanics are fun, work on the balancing part, figuring out the exact formulas and numbers in such a way that the game is neither too easy nor too hard from a pure "stats" perspective. That would be a good time to work on the AI as well, to make sure that the enemies are choosing their moves in a way that dovetails with the stats-based challenge level. You can do all of this without writing a line of code, and it's a lot easier to change your mind and try out new mechanics when you can do it in five seconds with a pencil.
You should plan... as much as you need to. What are your goals with the planning? Usability? Edge case testing? Fun factor? Who is your audience for these plans? Is this something you're going to implement yourself later or are you going to be handing this off to someone else?
My recommendation is to look at it from the perspective of the user. What actions can they take at any given point in time. Maybe draw out some storyboards.
If you need to put some more specific thoughts down on paper, sure start doing that. But you should make your system flexible and data-driven enough that you shouldn't necessarily have to sit down and say "spell X does Y damage" before you start making the system since that would probably be best determined through playtesting.
If it's simple enough that you can do so, You should play it out on paper.
Even if it's complicated, with multiple distances, stances, cover, hit locations, what-have-you, you should see if there are any obvious pitfalls, and doing it with pen and paper is probably the fastest way. Run a combat scenario with your battle idea and see how it goes.
Sure, writing a document about it is probably a good idea, it will help you formalize and analyse your ideas, but do not feel constrained by the document.
I usually write systems for pen-and-paper RPGs, so bear that in mind when reading this.
First, I look at what kind of experience I want to create. Is there something I want to emulate? What sort of pace to I want to keep? How punishing should mistakes be? Should a lot of it be up to luck? You don't need (or want?) hard numbers here, but an overview.
Next, I like to think of how the player interacts with the situation. Do they pick from an a list of options in order? Do they expend resources? Do they have to pick their moments? Do they try to change the situation to suit them?
I find them quite handy. Map out your combat loop a little. Again, hard numbers are for another pass. Another thing is to plan out how quantities such as strength, damage, action points and so on link together. This is not the combat cycle, but a map of dependancies.
Write out a sample combat using your flow chart. Make up values that feel right, or use placeholders. What happens? Where did the player get to have an influence? Are there any parts where they could get bored, or where things spiral out of control? Could a lucky hit lock the opponent down completely?
Look back at the chart and make any changes you think fit. If you can't follow it at this point, before the implementation intricacies, then it's probably going to be a bit much for a player (remember, you know it better than anyone!). Try pin down a few values.
I like to jot down a few ideas for containers, data structures (well, the pen-and-paper implementations of them), and ways to tie these back to the feel. I also look at single things which could swing the battle in radically different directions, in terms of experience - it may be worth changing them.
Sometimes an idea is really awesome, but hard to reconcile with the rest of the system. Rather than discard it entirely it may be worth smashing it to bits, working out why it works, then rebuilding it.
Make a few prototype items, enemies, and so on. Concentrate on modelling archetypes. More complexity can be added later, as can more subtle differences.
Hope that helps.
Since the site doesn't let me upvote or comment Ian Schreiber, +1 to him.
If you intend to test the battle mechanics, a pen&paper test is the way to go. It is also the method recommended by Game Design Workshop: A Playcentric Approach to Creating Innovative Games.
Once you have that done, then you may try to implement it in an actual game (or just look for people interested in making a game to work together).
