I assume you want your library to work for more than one specific videocard. So you will need to make assumptions about the minimal set of features current videocards support. You can assume that this minimal set is basically the instructions that DirectX and OpenGL require (so that's strike one for writing your own library).
Of course you could use these instructions and use them as you want in your library. However you still need a way to talk to the GPU, now I don't think there is a way to do this (without severe reverse engineering and guaranteed differences between different models) without using the driver provided by the GPU vendor (AMD, nVidia or Intel usually) (strike two). Now of course you will have to talk to the driver in a way it understands... and well unfortunately for you the driver understands only DirectX and OpenGL (strike three, you're out!)
So theoretically there is a way to do what you want, but it would involve writing your own driver for a GPU which, without the help of nVidia or another GPU vendor is neigh impossible, and the end gain would be neligible since you would still have only the instructions nVidia and other GPU vendors found useful for implementing OpenGL and DirectX.
Since your question has a lot of overlap with this one on stackoverflow you might want to read the answers there as well.
If you tell us why you are not content with DirectX and OpenGL we might be able to help you with that underlying problem.