I've done this professionally. For about a year, I was the guy responsible for managing rendering performance for a single "AAA" 3D game across almost the complete line of iOS devices.
I want to stress this: for the duration of the development of the game, this was exactly all of my job. I did this 8-10 hours per day, five days a week, for a year. I wasn't implementing the game (apart from some shaders); I was only maintaining and tuning rendering performance across a broad set of devices.
This is my way of saying: it's not easy. It takes a serious investment of time to really maintain compatibility across a wide range of devices.
The way that I did it was to build a series of in-game graphic testbeds. Things like level flythroughs, animation tests, and so on. These tests would run, and the devices would log out statistics about how long things took to render, how long they took to process, etc. I'd run these tests on about six different devices every day, as development continued. And then I'd graph these blobs of data, so I could compare how long something took to render on an iPhone3 vs. an iPhone 3GS vs. an iPhone4 vs. an iPhone4S, and I'd have to figure out how to make the iPhone4 run as fast as the 3GS, and how to boost the speed of the 3, and so on. I did research to figure out exactly what operations were slow on each device, so that I could make custom rendering changes for each model.
Often this meant using different shaders. Or disabling shaders entirely. Or tuning the far clip distance or being more or less aggressive in occlusion testing or anything else I could think of to boost performance on the slower devices. And after every change, I'd run the suite of tests again, to see if the change I'd made had actually had the effect I expected it to. It was a slow, laborious process, and required a lot of attention to avoid making mistakes and making things worse.
Incidentally, the other developers loved these graphs; they immediately showed on which days somebody had done something that hurt performance, which meant that we didn't have the usual last-minute crunching to find performance, since we (mostly) caught everything within a day or so of it going in.
In terms of pipeline, I used two separate pipelines; one which used shaders, and one which used the fixed-function pipeline. I'm not sure if I recommend that these days, but it's what I did. (The fixed-function pipeline, in my case, was legacy code which already existed. The shader pipeline was new. I don't think I'd bother writing a fixed-function pipeline from scratch, today, unless I really needed to support something that didn't have shader support.)
The device-specific changes were all placed in game code and in data files, not in the pipeline (so an iPhone4 would load a simpler shader than an iPhone4S would, for example, and an iPhone3 would move to lower LODs at a closer distance, etc), but they'd all be rendering through either the shader pipeline or the fixed-function pipeline, neither of which cared about the particular device they were running on.