When talking about game engine, I guess you are thinking about anything that is not the render engine, sound engine, physics engine, input engine, etc.
So you might be thinking about game architecture, AI, or design pattern/systems etc, which are used to structurate a software into managing data and other components to create some sort of scene/thater/illusion, thus producing a manageable sets of rules to tweak to create an actual "video game".
This is the actual top layer of a video game, and I even tried to learn a little about it, it's vast, complicated and requires a lot of knowledge into all other subjects. Imagine you want to recreate a alternate reality from nothing, keet track of and computing every each event, and regenerate accurate data from those, at whatever moment: for every kind of game there is some kind of subset game-engine, and I don't think it's actually possible to make a generic engine which is mathematically fast enough. Even for a game where you control a single character, there are still a lot of things which can vary and can still require the game to change the whole architecture.
The gaming industry is not as well-served by open-source as other fields of computer sciences, just because entertainment has different kinds of license types, which involve artistic licenses: game product is only paid by a customer, and AFAIK, there is only few engines that seems good enough to work for you :
Hord3d, NeoAxis, blendelf, Panda3D. But I can't assure you they will be as much easy to use as tools like Unity, Torque, C4 and others; keep in mind that a game engine is, with the graphical renderer, I think the most difficult type of software you'll find in game-making.