Ok, so, voxels. Are you using a game object for each voxel? If so, there's your problem.
Minecraft (and clones) don't actually hold an object in memory that represents each individual block. They have an object that describes the block and then an array of IDs--usually integers, aka Block IDs (Stone is 1, dirt is 3, water is 9...)--used tor represent which block is in each location (and then a lot of lookups are performed). From there it computes a single mesh for an entire chunk based on visible sides (this is why X-ray cheats work). That way your entire render distance out to 160 meters is made up of only 441 actual objects (for the ground) each with 16k verts (depending).
This is a much easier problem to solve when it comes to occlusion culling. Namely: is it in front of the camera...or behind? And basically say "fsk it" about the underground spaces because the savings are slim.