Most of your efficiency benefit can come from checking against a loose bound for your whole shape (box or sphere known to contain it). When you find that the ray "might" intersect it (because it intersects a sphere or box that's not much larger), then you pretty much need to check each face.
You can skip faces whose normals point away from the ray, since a different face of your closed solid would have to catch it first, if at all.
In some very specific cases, you can rule out whole faces. For example, consider unit cube centered on (0,0,0). If you're casting any ray from (0,0,100), the only face it could hit, if any, is the Z-positive one.
You mentioned "simple 3d shapes". For a complex shape with many faces, you might do broad rejection (sphere or box) for smaller pieces or individual faces of the shape.