How do I use the engine's vertex shader while still allowing the programmer to provide his/her own vertex shader for other calculations in their game? What is the normal approach here?
There is no normal approach because engines generally don't let you do that.
Generally speaking, engines take one of two approaches: either the user provides none of a shader or all of a shader. Never parts of a shader.
Engines that allow the user to provide shaders will establish some form of convention for communication between the application and the shader. They'll say that the position must be named "position" or whatever, and you're expected to provide vertex shaders with that attribute name. The various transformation matrices will similarly have specific uniform names and types, and the user providing the shader must adhere to those names and types.
Some engines allow you to create shaders from raw components via some system. But even there, it's the system that's creating those shaders. If that system has some mechanism for inserting user-provided fragments of code, then it will be specific to that system, and it likely won't be in GLSL anymore. And even then, those fragments will have to adhere to a specific convention based on the needs of that system.
In any case, if you're truly writing an engine for yourself or a single team that you're supporting, you really shouldn't bother with this. Just use shaders directly.