No high-performance voxel renderer draws individual cubes or cube faces.
To get good performance, groups of voxels are assembled together into a single mesh, called a "Chunk".
A chunk is typically something like 16x16x16 voxels, but might be larger or smaller as you need. It contains only the visible faces of voxels in that chunk, stitched together into a single mesh that can be drawn with a single draw call.
Compare that to drawing a solid 16x16x16 block of voxels using your current method:
- iterate over all 16x16x16 voxels (4096 iterations)
- check all 6 faces of each one (24 576 iterations)
- draw those that border empty space (1 536 draw calls)
- check if chunk is in camera view frustum (1 check per chunk)
- if so, draw the whole thing (1 draw call per chunk)
We've cut our work down by several orders of magnitude!
This means less work for your CPU to do in preparing the drawing instructions for each frame, and each draw call gives more for the GPU to churn through in parallel, making use of the dedicated hardware it has for this purpose.
When a voxel's state changes, you re-compute the visible faces in the affected chunk, and update its mesh accordingly. As you point out, this happens much more rarely than we render a frame, so we can skip that work for most chunks, most of the time, and just keep re-using our cached meshes.
You can also use your chunks for a form of occlusion culling. A single voxel doesn't occlude much, but a chunk's worth of voxels might be enough to completely block the view through one of the far sides. If it does, you can skip drawing whole chunks worth of content behind it, saving yourself a lot of work and overdraw.
Taking a look at the demo applet from Tommo's Advanced Cave Culling Algorithm article linked above, you can see the kind of savings chunks can offer. The dark shaded chunks are behind the camera frustum, so they get skipped. The green lines trace the order in which chunks are evaluated, stepping outward from the camera. The pink shaded chunks are completely hidden from view by closer chunks, and can be skipped too. So here the engine has to draw only 19 chunks, instead of millions of individual faces.