I have implemented something similar in the past and I can tell you that casting a box along the camera direction will not work (unless you are using an ortho camera). For a perspective camera, you would actually need to cast a frustum :). The reason for this is because the four points that make up the rectangle in screen space (as they show up on the screen), if you were to just extrude them, for a perspective camera, they would be squashed by the perspective effect. So you would need to compensate for that by not only extruding but also enlarging them as they get further away.
The way I did it was to always keep a record of the objects which are visible to the camera. Those are the objects which you can see on the screen. Now when you need to check if the selection rectangle, intersects the objects you could write something like this (I assume you're using C++):
for(int objIndex = 0; objIndex < numVisibleObjects; ++objIndex)
GameObject gameObject = visibleObjects[objIndex];
AABB worldAABB = gameObject.GetWorldAABB();
Rect2D screenRect = worldAABB.GetScreenRect();
// Object is inside selection rectangle. Do stuff.
So basicly you loop through each visible object, retrieve its world AABB, and then call GetScreenRect on the AABB to return a 2D rectangle which encloses the AABB in screen space. This can be done by looping through the AABB world corner points, transform them in screen space and find the min and max points, in screen space. From there you can use that to build the rectangle.
After you have this rectangle, you can just check if it intersects the selection rectangle. If it does, it means the object is inside the selection rectangle.
Now all the above implies that you will have to implement a mechanism that is able to detect visible objects at any one time. This can get a bit more complex, but I think in the beginning, if you are using a small number of objects in the scene, you could just loop through them all and use the above code snippet (or something like it) to see how it works for you.
I hope this helps.