When a new processor generation is released, most websites report that game engines and programs need to be optimized for the new hardware. I do not quite understand why. A processor usually has an architecture that defines what kind of instruction set it uses. The one we all use nowadays is amd_x86_64 . Why would any program or compiler need to be updated if all processors use this same architecture? Surely there are features WITHIN the pipeline of the new processor that optimizes the execution of machine code, but why would the machine code itself need to be changed if the architecture did not?
Because different generations of the same architecture can have different instruction sets.
For example, Streaming SIMD Extensions are probably the best-known x86 instruction set, but yet, and despite there just being one x86 architecture, there exists SSE, SSE2, SSE3, and SSE4.
Each of these generations can include new instructions that provide faster ways of performing certain operations. An example that would be relevant to games might be dot product instructions.
So if a game engine is compiled for a previous generation of an architecture, it will not have support for these newer instructions. Similarly, it may be necessary to optimize the engine for newer instructions; SSE4, for example, has support for dot product instructions that work on array-of-structs data. An optimization that could take advantage of these newer instructions would be to change your data layout to array-of-structs.
Maximus's answer is correct, I just want to give another piece of the story:
The hardware itself changes in a way you need to change how you code, regardless of newly introduced instructions.
Increased or decreased amounts of cache means you need to worry less or more about cache optimization/cache invalidation being issues. More cache means with small data you can focus less on making sure data is contiguous with out running into performance concerns. Less cache means this could be an issue, and very little cache means with some large data structures it won't matter any way.
New levels of cache means you need to think more about how you organize even larger sets of data (L1, vs L2, vs L3 vs L4).
More cores means you need to think about how you are going to multi thread applications better, and how your application scales in a multi process environment.
Faster clocks means you need to start thinking about memory latency more than you need to think about CPU computation speed as a bottleneck of your system.
The number of FPUs on a system may no longer match the number of integer ALUs per core any more (AMD had/has architectures like this).
The number of clock cycles it takes to compute an operation my have decreased, or increased.
The number of registers available changed.
All of these have very real performance impact on programs which made assumptions about underlying architecture in previous hardware with the same ISA, either positive or negative.
Even beyond gross changes like support for new instructions, microprocessor manufacturers are constantly modifying their designs to improve performance, and every new design can have different relative performance for each instruction or technique. Maybe you wrote some carefully optimized branchless code for the Model X, but the Model Y has an improved branch-predictor that reduces the misprediction penalty for the non-branchless version of the code (which also frees up a register to be used somewhere else). Maybe the Model Y supports greater parallelism of a certain high-latency instruction, so that now an unrolled loop of that instruction gets you better throughput, while on the Model X a shorter sequence was better.
Any problem can be solved in many ways, and every program is an interlocking collection of trade-offs and resource allocations, from the point of optimization. Even small changes in the availability of those resources or the cost of a given piece of code in terms of those resources, can have a cascade effect that gives a substantial performance advantage to one piece of code or another. Even if an upgraded chip has "more of everything", how much more of each thing can swing the balance.