There may be a bug in your raytracing, though there's a broader issue with your lighting here.
In general, this kind of lighting artifact is caused by not properly tessellating your faces. If each face is a single quad of only four vertices, then there is no lighting calculation in the middle of the face or in the middle of the edges. You end up with roughly triangular lighting artifacts along the edges of complex features in your terrain. This is because a single occluded vertex affects a large amount of the surface while ideally it should only affect a small area.
The solution is to further tessellate your faces, giving them more than four vertices. Your faces could be four quads (16 vertices) for example. The artifacts will be smaller. Tessellate further if desired. Find an acceptable balance between visual quality and the speed of baking the lighting.
Graphics are almost always a trade-off between an acceptable loss of quality (increase in artifacts) and performance. Even if all the baking is done offline (your terrain never changes), you might decide that making a designer wait 2 hours for lights to bake is not worth the increase in quality. Maybe you have two light baking quality levels, one for quick iteration and one for final production. At runtime, you can have options for users to get a visual quality they like that performs well on their hardware.
You can go a long way with different techniques, too. For example, if your only lights are one you carry and the Sun, you can ignore ray casting the light you carry (any unlit surfaces will also be occluded surfaces) and use a shadow map instead of ray casting for good-lucking shading from the Sun. Again, tessellating surfaces improves lighting results, but it removes the massive overhead of raycasting to each vertex in the CPU.
You can also try doing dome GPGPU ray casting to improve baking performance after raising the vertex count. Or at least parallelize the baking on the CPU so today's multi-core CPUs can chew through all the raycasting much faster.
Doing delayed lighting is also an option. After changing the voxels, do X ms of lighting calculations each frame until it's up to date, radiating out from the view position. This way you can bound the banking time per frame to maintain smooth framerates, even if the total process takes way longer than a single frame.
Short version: use smaller quads and more vertices to get smoother lighting, and find solutions to any performance problems that creates.