The easiest way for the developers (who tend to make games for commercial purpose) to make community patches is to give the players an SDK/API or whatever has been created strictly for the game. Of course a normal player will not have access to the 3D Studio Max, but the developer can provide something else, like the 3d item prospector, 3d world builder etc.
Bethesda Softworks has provided the Skyrim community with the Skyrim Creation Kit which is a combinatin of all tools that were used to assamble the game, by level designers, game designers.
It has complete solutions for the AI system along with a complex NPC talk-action dialogue system. That way the players can do anything to the game, no source code is needed. As the matter of fact the Skyrim uses Havok Physics engine, and I am sure that distributing the expensive Havok Engine soucecode would make the Intel corporation rather unhappy.
The community creates various patches that are shared over the net. Even though the company has no money from the additional player-made patches, it encourages players who play on a pirate game version, to buy one because it gives you access to the community mod patches along with the Creation Kit. Bethesda used some of the ideas/solutions provided by players in mod packages upon creation of the official game patch, which was free. And later they have used some of the mods in creating the official Skyrim expansions such as the "Dawnguard" or "Dragonborn" which cost about $15 each.
It doesn't have to be opensource to encourage players to create using the SDK (Software Development Kit) you provide. Thus delivering source code isn't mandatory to create the game fully "modable". The best proof of that is Skyrim - 4+ large official patches and 15,000+ community made patches with 16+ patch combo's with major bug fixes done by enthusiasts.