If you're looking for soft shadows, then you have to go back to the reason why shadows aren't hard in the real world.
Soft shadows are a by-product of lights that have a finite area. In order to handle this properly, you have to actually check how much of a light source is visible; it's not just a binary on/off.
Your 4x4 kernel is a good approximation for soft shadows, but only for directional lights. Each sample is effectively approximating a light of a particular apparent size. And since directional lights are infinitely far away, all objects will see the same apparent size of the light. So a fixed, 4x4 kernel can approximate soft shadows for a particular area of the light.
For a point light, that approximation no longer makes sense. You can't use a fixed kernel to get soft shadows, because the apparent size of the point light changes depending on where you are.
For this, you need to compute the apparent area of the light (based on the light's size), then compute a number of directions within this area which you feel is representative of the area. Or just whatever you feel you can get away with without killing your performance. You use these directions as the normals in your shadow tests. Done.
Now all of this is an approximation (or really just a hack), because they don't really provide soft shadows at all. The problem with these approximations is that they're basically changing the place in the world that they're looking for shadows from, rather than changing the place on the light where they're looking for shadows from. For a directional light, it's really one and the same, but not so for a point light. To get soft shadowing more accurately for point lights, you would need to render multiple shadow maps from multiple different positions.
Or just go with the approximation; it's probably loads cheaper.