The lowest level that makes sense from my point of view is something that talks about the resources involved in rendering - vb/ib, render surfaces, textures, shaders, state blocks, etc.
The problem here is that some of these need to be in different formats, depending on the API - that's where it gets a bit tricky. The easiest way around it is to pre-process static resources for the respective API. For dynamic ones, use only shaders to generate them - that makes it fairly straightforward to stay in native formats.
All you then do on the higher level is set up pipelines with attached resources and hand them to the GPU. You'll find that not everything can be abstracted out nicely in that way, especially if you take advantage of hardware-specific tricks. But it's a good start.
(Sidenote: if you treat platform-specific tricks as a special kind of resource, you can push this whole concept quite far. )
So in a way, you'll create two things: A hardware resource manager, plus a toolkit to set up a DAG of these resources.