It sounds like you've got at least two threads of execution, with your rendering on its own thread. If that's the case, then you actually have two frame rates to worry about. You'll want both to be as fast as possible. However, it also depends on the kind of game you're building.
Are you building a first-person shooter, where small drops in frame rate could give an opponent an advantage? If so, then you'll want to make sure your average fps is high enough, but also worry about your worst-case frame times. Are you building a board game? If so, the occasional frame time spike isn't going to kill the user experience.
In my own games, my process is usually something like this:
- Run a profiler on the code
- Look at the average frame rate. If it's too low on average, bring the averate fps up by optimizing the slow stuff.
- Once average fps is high enough, look for worst-case frame times (frames where you see big spikes in computational or rendering time). Try to optimize those worst-case scenarios to improve worst-case frame time.
If you're at 30 fps most of the time, but you spike to 200ms every 10 seconds, that's going to cause problems. But if you're averaging 15 fps, bring your average fps up first.
So, the short answer would probably be: optimize whatever makes the biggest improvement to the user experience first.