I am implementing PCSS and I am using a directional light. The directional light volume is clamped to the camera frustum (as is typical with directional lights). So when the camera rotates, the directional light volume can find a tighter bound on the camera frustum in some cases, and then all of the depth values can increase/decrease.
This causes the blocker search to find different blocker distances from the light source, and so the penumbra sizes change based on the camera rotation.
Has anyone dealt with this problem before? How can I compensate for changes to the light depth map (aka shadow map) when the camera rotates?
Here are some images demonstrating two camera views, visualizing the penumbra values (brighter indicates where a larger penumbra should go) and the corresponding light depth maps (shadow maps):