Is it only a matter of code optimization?
Do game developers plan to start small on a new hardware, to have space to evolve while creating a series of that game?
Are computer graphics breakthroughs common enough to happen in a console lifetime?
There is indeed an optimization part in this. The more programmers get used to a console hardware, the more they learn how to squeeze graphical quality into it.
But it's far from being the only reason:
That's an interesting question. I don't think game developers intentionally leave space for evolution when working on a new console. To me this is only incidental. But I stand to be corrected, maybe some big studios do this strategically after all, I'd be curious to see any evidence of that (testimony, anybody?).
They most certainly do not!
When I first joined the games industry, I asked a producer from [giant publisher you've definitely heard of] this very question. He told me that every console game his company developed used 100% of the platform's available resources. While this was probably not the literal truth, it's close enough.
If anything, console games spend most of their development using closer to 120-150% of the platform resources. The last few months of development are spent frantically trying to crunch everything down to "only" 100% before the ship date. If subsequent titles need more CPU cycles / RAM / disk space / etc., the developers first have to figure out how to optimize their current technology -- find more efficient algorithms, use better data compression, etc. Thus, rapid evolution by necessity!
It makes perfect sense, actually; what competitive advantage would there be in explicitly not trying to reach the full potential of the hardware, given your current knowledge of the system? Would you trust your competitors to show the same restraint?
New hardware generally just offers the ability to do more than before. Most graphical improvements come from new techniques, new optimisations, and better art.
It's worth knowing that the 'new' techniques that get used are rarely new - they are typically 20 year old techniques that previously were too slow for real-time rendering. These typically get translated into real-time code many years later when a programmer finds a way to do it efficiently with the hardware of the day.
Since Consoles pretty much have one specification. And not like PC where you have tons of different variations. Developers can optimize their games better. But this doesn't mean it will do space magic in the long run. It will reach a cap. ( currently the 360/Ps3 have 512 mb in total, which to many developers seems to be an annoyance)
Consoles are fairly outdated after 6 months. PCs always grow faster in hardware specs. But games on PC never really get optimized for one hardware set. They have to keep in mind that there are also people with lower specs ( hence the video and other options ). But since developers working on console games can optimize in the longrun they can do neat tricks with the hardware.
PS3/360 games look fairly good in this time of their life cycles. But just imagine if a developer focused on one specified powerbeast of a pc.
In the beginning of a new consoles life developers need to adapt to the hardware. You can't just slap code from one platform to the other. That is why you don't see the games we have now in the beginning of the life cycle. The production pipeline grows overtime. It's not something you can get right the first time. A lot of optimizations need to be made and it just costs time. When you see better features in a sequel it's because the development of the technology was growing during the first game.
Real breakthroughs usually happen with new hardware. But developers can be creative at times that allows them to do things on limited hardware.