I am just a beginner in 3D games, my preferred platform is Android. I posted a question in some other forum about "What to use: OpenGL or Unity3d?" They all emphasized Unity3D and using its built-in features and tools, which I would have to code myself if I were to use OpenGL. But then someone says most of the complex games require custom engines. Why would I use an engine like Unity3D over OpenGL?
I think you've answered this question yourself already. You said "I'm just a beginner in 3D games". Further you said "... Unity3D and using it's built in features and tools, which I would have to code myself if I were to use OpenGL".
Essentially this means if you're using OpenGL alone, you're going to be writing a game engine, then, potentially years later, writing your game. If you want to focus on a game, utilize the resources available to you to get you as close to that goal as possible. Using a game engine is the best way focus your efforts on creating a game, instead of writing all the stuff required to create a game.
See this related question: What's the difference between a Library and an Engine and this one What is a game framework versus a game engine?
Because beginners, by definition, don't know what they're doing.
OpenGL has a huge learning curve. Sure, you can issue a few glBegin commands, get all excited about putting some triangles on the screen, then go off the deep end with something more complex that you're nowhere near ready to take on yet. That's not a great way to learn. An analogy (slightly exaggerated) might be learning some basic high-school physics then being put in charge of the Large Hadron Collider. It's not sink-or-swim, it's swim-or-leave-a-large-radioactive-hole-in-the-ground.
It's assumed that most people asking the "I want to make a game, what should I use?" question actually do want to make a game (otherwise they would have asked a different question, surely?) So the answer is tailored to the question, and using a pre-existing toolkit, framework or engine is one way of helping them achieve that ambition without becoming distracted by agonizing over whether a UBO or glUniform calls is preferable for updating a single matrix.
This is all about taking one step at a time, making the learning process easier, and giving you something rewarding at the end of it. Trying to learn how to make a game and how to properly use a mature and powerful 3D API at the same time is probably too much for a beginner. This is like the Dunning–Kruger effect minus the negative connotations. You don't know what you don't know, so you don't really have any kind of realistic picture of exactly what you'd be taking on.
On the other hand if your objective is to learn OpenGL, then you don't actually want to make a game! You want to learn OpenGL, and doing it in the context of making a game may be a fun way of learning, but making a game is a means to an end, not an end in itself (and certainly not the only way of learning OpenGL). You'll probably never release it, and you'll almost definitely look back on it in a few years time and be horribly embarassed by some of the stupid things you did.
It really depends on what you want to get out of this project. If your main goal is to focus on game design, then you're probably better off with a pre-built engine, because that will get you to the game-making part a lot quicker. If your goal is to learn about the programming portion of making games (especially if you're trying to prepare for a job in the industry), then you'll probably find working with OpenGL will be a more valuable experience. Keep in mind that neither of the preceding statements are true 100% of the time.
There really is no rule when deciding on this sort of thing. Most of the responses you hear in that sort of question are opinions, and very few of those will be completely relevant to your own project and skill set.
The difference is simple. Unity is a carriage with four round wheels that can get you places. OpenGL is sawdust that you could set up a shop and potentially use to mould an improved and more modern carriage and wheels. That is if you invest many hours of your time into learning what the people before you have done first.
Your goal as a beginner should be to first study existing carriages and their designs. There is not point in throwing everything out the window and starting from scratch. First learn what already exists and get a thorough knowledge of it. If you don't, how do you know of it's limitations and if you don't know the limitations, how could you break through them? What is the point in making something mediocre that is useless and being ignorant to all the vibrant existing tools that are already available?
imho the steps of a fledgling game dev are:
- Master a modern framework / Engine and get a deep understanding of it (This will also help you learn of the different aspects of game design, graphics and OpenGL are not all there is).
- Make a small game with it and learn more in the process.
- Organically encounter obstacles and limitations in the engine.
- Learn more about the subject.
- Work to improve the areas that you deem needs improving.
Working solo on your own on making something that has already been made will not make you a valuable developer to anyone because you will not be productive in truth or a good learner for that matter, you will just copy existing bits of code and maybe learn to read and understand them. As the old Pokemon saying goes, "It's not very effective...".
The worst part is that this kind of approach is not easy to keep up with because there is not gratification in it so you are running on raw will power which has been scientifically proven to be a limited resource to human beings. You need to do something that provides daily gratification or your will probably not be able to keep it up.
So to sum things up, an Engine is emphasized because you can get things done with an Engine and you are far more likely to succeed that way. Even great game creators relied on someone else’s previous work and such is the nature of progress. You need to understand the big picture and know how to play with the Lego pieces provided by others before you make your own custom Lego pieces.
Assuming your goal is to create a game, the guy who talked about custom engines being required was wrong. If anything, complex games require a mature engine that's been designed, implemented, optimized, tested and is maintained by professionals who have been in the industry for long - or one that can match their quality (at least for the purposes of a given game), if that's possible.
Large game developers do build in-house engines, but they still reuse them heavily for their future games. But keep in mind that their budgets and team sizes and lots of other factors are vastly different than independent developers' - it may make much more business sense for them to build their own engine, even if existing ones don't fall short technologically.
Of course, there's still the misconception, because a lot of features don't exist out-of-the-box in existing engines, that they can't be implemented in that engine at all, but that's usually inaccurate.
Keep in mind that some of the most innovative modern games, like Portal and Antichamber, were built using existing engines. Some rather complex online games, like Aion and TERA were built using existing engines. Some of the most visually beautiful games, like Bioshock Infinite and Thief, were built using existing engines. And, of course, some weren't. But the point is: nothing I've ever seen really required a custom engine.
On the other hand, if your goal is to learn concepts of graphics programming, of course OpenGL is the way to go. But even then, keep in mind that being familiar with a few existing engines will help you tremendously with understanding what you're doing and what you should be aiming for.