Is it really efficient to load all your objects in the physical or collision world just to test them for intersection?
Yes it is efficient... for the programmer. There are lots of physics engines around and it's far easier to just use one - than to strip one down, or implement raycasting yourself.
Talking about code (in fairly rough terms):
You've got the code to create physics objects (you're not going to coldet/raycast against fully triangulated models, are you?) in a data structure that is efficient for doing coldet/raycast (you're not going to bruteforce it, are you?). And then some wasted code for various physical properties (and depending on the physics engine you can probably avoid much of this). Plus the CPU and memory requirements for the same (relatively small).
Then you've got the raycasting code. You need that in any case.
And finally the rest of the physics simulation code - which just sits in its binaries, never executed, never loaded to cache. Code doesn't take up much space anyway.
So the "wastage" here is utterly minuscule. Especially compared to the massive effort involved in replacing or stripping down half a physics engine.