In 'normal' business programming optimization step is often left until really needed. Meaning you should not optmize until it is really needed.
Remember what Donald Knuth said "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil"
When is the time to optimize to make sure I am not wasting effort. Should I do it a method level? Class level? Module level?
Also what should my measurement of optimization? Ticks? Frame Rate? Total Time?
Where I've worked, we always use multiple levels of profiling; if you see a problem, you just move down the list a bit more until you figure out what's going on:
(There's a fun story from a past year's GDC of a graphics programmer who took four pictures of himself -- happy, indifferent, annoyed, and angry -- and displayed an appropriate picture in the corner of the internal builds based on the framerate. The content creators quickly learned not to turn on complicated shaders for all of their objects and environments: they'd make the programmer angry. Behold the power of feedback.)
Note you can also do fun things like graph the "profile bars" continuously, so you can see spike patterns ("we're losing a frame every 7 frames") or the like.
To answer your question directly, though: in my experience, while it's tempting (and often rewarding -- I usually learn something) to rewrite single functions/modules to optimize number of instructions or icache or dcache performance, and we do actually need to do this sometimes when we've got a particularly obnoxious performance problem, the vast majority of the performance issues we deal with on a regular basis come down to design. For example:
You hear it until it becomes obnoxious in any university-level computer science courses, but: it really is all about the data structures and algorithms. Spending some time on algorithm and data flow design is going to get you more bang for the buck in general. (Make sure you've read the excellent Pitfalls of Object Oriented Programming slides from a Sony Developer Services fellow for some insight here.) This doesn't "feel" like optimization; it's mostly time spent with a whiteboard or UML tool or creating many prototypes, rather than making current code run faster. But it's generally much more worthwhile.
And another useful heuristic: if you're close to your engine's "core", it may be worth some extra effort and experimentation to optimize (e.g. vectorize those matrix multiplies!). The further from core, the less you should be worrying about that unless one of your profiling tools tells you otherwise.
Also remember however "premature pessimization". While there's no need to go hardcore on every line of code, there's justification for realizing you are actually working on a game, which has real-time performance implications.
While everyone tells you to measure and optimize the hot-spots, that technique won't show you performance that is lost in hidden places. For example, if every '+' operation in your code would take twice as long as it should, it won't show up as a hot-spot and thus you will never optimize it or even realize, however since it's being used allover the place it might cost you a lot of performance. You'd be surprised how many of those cycles trickle away without ever being detected. So be aware of what you do.
Apart from that, I tend to profile regularly to get an idea of what's there, and how much time is left per frame. To me time per frame is the most logical as it tells me directly where I'm at with framerate goals. Also try to find out where peaks are and what causes them - I prefer a stable framerate over a high framerate with spikes.
Once a game is ready to be released (either final or a beta), or it is noticeably slow, that's probably the best time to profile your app. Of course, you can always run the profiler at any point; but yes, premature optimization is the root of all evil. Unfounded optimization, too; you need actual data to show that some code is slow, before you should go try to "optimize" it. A profiler does that for you.
If you don't know about a profiler, learn it! Here's a good blog post demonstrating the usefulness of a profiler.
Most of game code optimization comes down to reducing the CPU cycles that you need for each frame. One way to do this is to just optimize every routine as you write it, and make sure it's as fast as possible. However, there is a common saying that 90% of the CPU cycles are spent in 10% of the code. This means that directing all of your optimization work to these bottleneck routines will have 10x the effect of optimizating everything uniformly. So how do you identify these routines? Profiling makes it easy.
Otherwise, if your small game is running at 200 FPS even though it has an inefficient algorithm in it, do you really have a reason to optimize? You should have a good idea of your target machine's specs, and make sure the game runs well on that machine, but anything beyond that is (arguably) wasted time that could be better spent coding or polishing the game.
I find it useful to build profiling in. Even if you're not actively optimising it's good to have an idea on what is limiting your performance at any given time. Many games have some kind of overlayable HUD which displays a simple graphical chart (usually just a coloured bar) showing how long various parts of the game loop are taking each frame.
It would be a bad idea to leave performance analysis and optimisation to too late a late stage. If you've already built the game and you're 200% over your CPU budget and you can't find that through optimisation, you're screwed.
You need to know what the budgets are for graphics, physics, etc., as you write. You can't do that if you have no idea what your performance is going to be, and you can't guess at that without know both what your performance is, and how much slack there might be.
So build in some performance stats from day one.
As to when to tackle stuff - again, probably best not to leave it too late, lest you have to refactor half your engine. On the other hand, don't get too wrapped up in optimising stuff to squeeze out every cycle if you think you might change the algorithm entirely tomorrow, or if you haven't put real game data through it.
Pick off the low hanging fruit as you go along, tackle the big stuff periodically, and you should be fine.
If look at Knuth's quote in its context he goes on to explain, that we should optimize but with tools, like a profiler.
You should constantly profile and memory profile your application after the very basic architecture gets laid.
Profiling will not just help you increase the speed, it will help you find bugs. If your program suddenly drastically changes speed, this is usually because of a bug. If your not profiling it might go unnoticed.
The trick to optimizing is to do it by design. Don't wait till the last minute. Make sure the design of your program gives you the performance you need without really on nasty inner loop tricks.
For my project, I usually apply some VERY needed optimizations in my base engine. For example I always like to implement a good solid SIMD implementation using SSE2 and 3DNow! This ensures my floating point math is on cue with where I want it to be. Another good practice is to make a habit out of optimizations as you code instead of going back. Most of the time these little practices are just as time consuming as what you were coding anyway. Before coding a feature, make sure you research the most efficient way to do it.
Bottom line, in my opinion, its HARDER to make your code more efficient after it already suck.
I would say the easiest way would be using your common sense - if something looks like it's running slow, then have a look at it. See if it is a bottleneck.
Use a profiler to have a look at the speed functions are taking and how often they're being called.
There's absolutely no point optimizing or spending time trying to optimize something that doesn't need it.
If your code is running slow, then run a profiler and see what exactly is causing it to run slower. Or you could be proactive and already have a profiler running before you start to notice performance problems.
You will want to optimize when your framerate drops to a point that the game starts to suffer. Your most likely culprit will be your CPU being used up too much (100%).
You should optimize your code... as often as you need to.
What I've done in the past is just continuously run the game with profiling turned on (at the very least a framerate counter on screen at all times). If the game is getting slow (below your target framerate on your min spec machine, for example), turn on the profiler and see if any hot spots show up.
Sometimes it isn't the code. A lot of the problems I've run into the past have been gpu-oriented (granted, this was on the iPhone). Fillrate issues, too many draw calls, not batching enough geometry, inefficient shaders...
Other than inefficient algorithms for hard problems (i.e. pathfinding, physics), I've very rarely run into issues where the code itself was the culprit. And those hard problems should be things you spend a lot of your effort on getting the algorithm right and not worrying about smaller things.
For me is the best follow good prepared data model. And optimalization-before main step forward. I mean before start implementing something big new. Other reason for optimalization is when I'm loosing control on resources , App needs lot of CPU load/GPU load or memory and i don't know why :) or it's too much.
