is all the shadow casting expensive?
That all depends on how much detail you're willing to drop. How accurate do you want this to be?
Fully accurate = casting through all individual branches and twigs = fine grained 3D volume grid or CPU-intensive geometric approach = costly. Self occlusion = potentially costly.
Desired ground-dappling resolution will also affect the number of rays to cast per unit area, affecting cost.
Some possible solutions...
- Treat the canopy as though it were flat, and store as a bit map (2D boolean array with each entry being shadowed/occluded or lit/unoccluded). Start the raycast at the level of this canopy bitmap, and cast the few meters / units to the ground. Unless occluded by trunks, rays will always reach the ground, and the distance to cast each ray will be fairly short.
- Use a quadtree instead of a fixed-resolution 2D array / bitmap. This will allow you to only do many raycasts in areas around each tree's canopy; in open glades, the quadtree will use a coarser resolution, reducing the number of rays. This is conceptually similar to the concept of orthogonal ray packets, where several rays are treated as a single ray in certain special circumstances.
- Enforce a maximum tree / canopy height to ensure the number of ray steps stays reasonable.
- Reduce the number of "dapples" per square meter, as much as you can while still allowing for a fair visual impact. You can adapt this into a LoD approach that depends on how far camera is.
Ultimately, you either do this using pure geometric approaches, or you break your space down into a regular 3D grid (voxels), even if that is only for purposes of calculating lighting. They both have their pros and cons. Voxels do make this process a little easier for those not very mathematically inclined.
Your other questions
We don't encourage multiple questions in one (and these are indeed totally separate problems), but briefly, you don't mention what sort of lighting you're using: forward vs. deferred, screen space vs world space, so it is really impossible to answer that. As for the most accurate pattern production - if I were you, I'd analyse my trees' volumes into an octree, then simply flatten the z aspect of the octree in order to get your flat canopy bitmap from which you can then raycast the forest floor.
P.S. Raycasting a canopy space accurately is a task challenging enough for the best procedural programmers using native languages and GPGPU tech; Unity is no match for these sorts of problems.