Open source and commercial are not mutually exclusive! Do you know anything about free software? There's a difference between free as in software and free as in beer.
For instance, Quake 3 was a hugely popular, very technical, 3D, high performance, commercial game engine. It sold as its own game, and then Id sold it out to other developers to make lots of games. They didn't do it immediately, but the quake 3 engine is now open source under the GNU public licence v2. Lots of indie / open-source games are being built upon the open sourced Quake 3 engine.
There's tons of other modern games that are open source as well, just look around. Myst URU Online just announced that they are going to be releasing the source code to both the client, the tools, and a server. Thats a modern, totally 3D game. There are tons of others.
"What is the main problem? Engine(Programmable part) or just textures/models?"
I'm not sure you even know what it is you're asking. Are you talking about open source, or are you talking about indie games? There's a HUGE difference.
Also, have you heard of mods?