It depends if you want to produce finished games or just learn.
If you actually want to make a game, making your own game engine from scratch just isn't feasible in any real timeframe unless you have an very narrow, simplified game and even then your still looking at quite a while.
Making a 3D game engine from scratch is a huge undertaking. They have heaps of stuff and most of it is complex.
Here is a bunch of stuff that a 'game' might contain:
- Your basic windowing toolkit (how to get a blank window in something like QT, SFML, GLFW, GLUT, SDL, WGL, etc... Or maybe a canvas in WebGL).
- Basic 3D api (draw a triangle).
- Integrating 2D stuff (you want menus and huds?). Simple basic stuff like overlaying SFML/SDL. Flash based UI? Integrate webkit for a HTML5 overlay? Integrate QT widgets?
- Handling input with your windowing system, what control types are there. Do you want to bind keys at runtime?
- More complex 3D api. Vertex Buffer Object, Vertex Array Objects. Rendering to a texture.
- Basic 3D knowledge such as Matrix and vector multiplications. How the transform matrices work. ModelView/Projection matrices.
- Learning an algebra math library so you don't have to do that all yourself from scratch (GLM for example).
- Loading your game from disk (Settings, World/Level data. Need to make your own file formats and loaders).
- Spacial geometry, How do you store all those object in 3D space. Just in one vector which is ok for smaller number of stuff or Octrees? BSPs?.
- Scenegraphs and rendering pipelines. How do you cut out the stuff you don't want rendered? Bounding boxes? Occlusion queries?
- Mesh Loading. How is the data stored? A simple OBJ, Parsing COLLADA XML and somehow producing something relevant to your gameengine/3dapi, your own optimized custom binary format? (which you will need a tool to produce unless your typing in vertices by hand). A 3rd party mesh loading library like assimp?).
- Texture loading (probably just use a simple lib, but which one? (I like DevIL with OpenGL) What format jpg? tga? dds?).
- Content creation. Will you just dump a bunch of placeholder meshes in or are you going to learn a 3D program like Blender, Maya or 3DS max. Are you going to animate them?
- Shaders. How to compile, bind values to them and use them. Then a whole new language either GLSL, HLSL or Cg for writing the shades themselves. There a pain to debug.
- Shading algorithms. Perpixel. Calculations for Blinn, Phong. Want toon shaders?
- Shadows. Want shadows, render the depth component of the scene from the view point of the light. How do you do that, binding frame buffer objects?
- Visual Effects. Glow can be quite nice and isn't much more than bluring. Depth of Field? Bloom? HDR? Antialiasing?
- Font rendering (Learn another 3rd party lib? Does your widget system have something?).
- Scripting? Lua? AngleScript? Python? Binding them to your language. Writing wrappers for everything.
- Best class structure for game data. Inheritance or component design pattern?, your probably going to have to mix and match quite a bit for a game. How about named runtime properties so you can do things like "player1.health=1" in a script (some languages this is quite easy but they have overhead as a result).
- Saving/Loading. How do you know what needs to be saved/loaded? Just do the whole level and waste everything or set stuff as 'dirty' and just save the dirty stuff.
- Networking, basic UDP networking, RPC calls (so a player can send that they moved). How do do you know what has changed and needs to be resent? Just have a few fixed values you sync or let anything change and keep a list somehow?
- Physics? Your own basic inbuilt physics engine? Or something like Bullet.
- Sound? OpenAL? OpenSL? DirectSound?
- Multithreading? Just the renderer?, a few choice threadable stuff (sound, physics) or the whole engine? How do you deal with multiple threads trying to read and write to the same stuff? Using locking mutexes with cost overhead or design a streamlines FIFO message passing que system.
- Optimizing. Are you going to use a flexible but more resource intensive language or something like C++ where you can play with pointers and pass by reference. How do you minimize dynamic memory allocation.
- Now how the hell do you pack that all together. So your script engine can access named runtime properties and cause the network to send packets with updated values while ensuring mutual exclusion and displaying it to the screen.
Giant imposing list aside, just about all of that is all insanely fun to learn, you programming will likely improve heaps as a result and much of it is useful in other areas or programming. Obviously a lot of that can be cut down depending on what you are trying to accomplish.
You will probably want to split up all that stuff into small projects. Much of it might not be 3D at all (like the networking and data structures). And lots of it isn't specific to 3D games either (networking, scripting , etc...).
One big problem with learning 3D is you need to make it interactive a lot more in order to understand it and have anything useful. Having a scripting engine for example can be great since you can just spawn in stuff at runtime, it break you out of just having a 3D rotating mesh on the screen and actually having something that looks like a game. Once you have the script engine maybe you can just store your levels as scripts rather than trying to invent your own data structures. Dumping in a physics engine will give you collision detection which is much harder in 3D than in 2D as well as all the cool physics interactions you will get. Choose a more inefficient scripting langauge that will let you introspect object properties.
Perhaps you could start off learning the api by porting your 2D game to it as a bunch of 2d squares. Then maybe you can look at trying to 3Difiy parts of it. Use a 3d mesh for the main character, then other ones, then 3Dify the tiles and background. Even that is kind of heavy since you need to learn things like animation and probably shading.
Another idea is a massive cutback on stuff. Some is obvious extras like multithreading. But maybe you shouldn't bother with any kind of lighting, shadows or texturing. Choose an art style that works with everything being flat like a cartoon (although actual cell/toon shaders also require learning different lighting algorithms). It would also save on making a whole bunch of textures and your models could probably have lower polygons counts. Or wireframe (although that might actually be more work since OpenGL core profile doesn't support actual wire-frame rendering any more, you either have to draw a bunch of lines which means either several redraws of your meshes or having different line based geometry layouts rather than the traditional triangle ones for your meshes or use a special wireframe shader.
On the flip side using a premade game engine allows you to skip much of the stuff in that list and you can concentrate on making an actual game and it's content. Just place an object as X,Y,Z and let the engine figure it all out. Of course you will be restricted by what you can do to what the engine supports or what you can make it support.
