Raycasting projection is fundamentally different from regular triangle projection that is commonly used in most games today. Forget what you know about vertex shader projection at this point.
There is no need for you to render polygons at all - meaning you typically don't have a vertex shader stage for this (it's what called passthrough vertex shader, i.e. it doesn't do much). That also means the vertex shader doesn't really manage any transformations via model, view and project matrices, in the usual OpenGL/DirectX sense. (See more on what the vertex shader does do, below.)
Each ray is oriented by the CPU or compute GPU (not the vertex shader), according to where the player is looking, with some offset such that each ray represents a pixel on the screen; "look direction" is the centremost ray/pixel of the screen, while all other rays have a slight angular offset from that:
Now we "cast" the rays through the world and find where they collide with obstacles, whereupon we read the colour and write it to the texture position assigned to this ray.
Finally, the texture containing these colours is rendered to a fullscreen, textured quad which is the only thing that your vertex shader will actually render - you don't need to transform the whole world space. Instead you have transformed every ray individually at an earlier stage in the pipeline!
So because we are using individual rays to read the world, we don't do the usual OpenGL thing of projection matrices to transform triangles. Instead each ray is multiplied by some matrix or modified in some other way, and is then fired into the world to retrieve the colour at its screen pixel.