You are correct. There is no need to use an engine to make your game. But in many cases it's just a matter of "why reinvent the wheel?"
There are for example frameworks such as lib gdx that abstracts away the most low level OpenGL stuff, so that you can focus on the things that actually matter.
I really recommend this blogpost by Dennis Gustafsson to get an insight in to what frameworks and engines does for you! It also gives some insight into what you really have to do if you want to cram out the last bit of performance from the hardware you are working with, which often is the reason some people stay away from "real engines". http://tuxedolabs.blogspot.se/2013/01/development-environment.html
With all this said, if it makes you excited to write things from the ground up, you should absolutely try it!