I have two RPG battle mechanic ideas that I am working on, before starting writing code (the engine is undecided), how should I plan the details of it? Should I draw and write on paper? or should I write a specification document? Any example of a game mechanic planning process that others were using?
The first (unasked) question is, should you be designing the battle mechanics of an RPG before any code has been written, in the first place? Probably, yes -- but be aware that if you're making an RPG on your own or in a small team, it is not the mechanics or game engine that is the challenging part, but the massive loads of sheer content you have to create in terms of every single level, character, NPC, dialogue tree, weapon, spell, treasure chest... I don't know you or your team, but I run into a lot of students who want to make RPGs because they love to play them, so my first instinct is to make sure you know what you're getting into!
Second, yes, I'd say that getting the game mechanics down on paper is a good first step. A good second step would be putting them in a state where you can actually test them "by hand" without writing code -- paper prototyping, in other words. Play out your combat system (or a simplified version if needed) as a tabletop game and see if it gives the player interesting decisions to make, and modify the mechanics accordingly until you reach the point where it works well. When the mechanics are fun, work on the balancing part, figuring out the exact formulas and numbers in such a way that the game is neither too easy nor too hard from a pure "stats" perspective. That would be a good time to work on the AI as well, to make sure that the enemies are choosing their moves in a way that dovetails with the stats-based challenge level. You can do all of this without writing a line of code, and it's a lot easier to change your mind and try out new mechanics when you can do it in five seconds with a pencil.
You should plan... as much as you need to. What are your goals with the planning? Usability? Edge case testing? Fun factor? Who is your audience for these plans? Is this something you're going to implement yourself later or are you going to be handing this off to someone else?
My recommendation is to look at it from the perspective of the user. What actions can they take at any given point in time. Maybe draw out some storyboards.
If you need to put some more specific thoughts down on paper, sure start doing that. But you should make your system flexible and data-driven enough that you shouldn't necessarily have to sit down and say "spell X does Y damage" before you start making the system since that would probably be best determined through playtesting.