I dont think the question should be "How to force players" but rather, "how to make very appealing to players" to kill low level monsters. The more natural and logical way it is, the better.
You could take example of the pokemon series (I personally think, its one of the best RPGs ever made). You couldnt die. You would be just moved to nearest center (that is similar).
You wanted to kill low level monsters to advance your pokemons and thus allowing take down the big boss and gain badge (story goal motivation). You wanted to "catch them all" including the low level ones (collection motivation). And lastly you needed a skill to advance (cut, swim...). Those are hard limitation (unlock restriction motivation), plus you could visit previously inaccessible areas for big rewards, you can go as far as make visible big shiny swords across a obstacle.
To break it down: you want to kill small ones, to kill the big one and get "key" to next area, all appearing natural and very RPG. You could hide it just as you suggested - give boss hp regen as dps check.
So, actually best is combination - hard restriction between big zones (acts) while soft ones inside it (gaining strength to kill final boss and unlock new act, collections, sets,...).