Unreal has built in systems for stuff like lighting, for example. If a developer wanted to create and use their own lighting system in Unreal, how do they go about doing that? Or are these systems already so customizable that creating a custom system isn't necessary?
We use the tools made available to us by the engine.
Major engines tend to try to cater to a reasonably large demographic of developers, which means trying to offer as many knobs and dials to turn as possible. If knobs and dials start to become cumbersome, engines typically support extension mechanisms such as writing plugins, replacement shaders, ways to disable or replace entire major subsystems or integrations, et cetera. This tends to get very engine specific.
At the lowest possible level, we simply(*) change the source code to do what we need to do. Many engines these days permit source code access, either relatively directly (like Unreal) or if you arrange a special licensing deal (like Unity). The latter is usually expensive (the kind of expensive where they don't tell you the price until you call them kind of expensive), but usually fit into the budget of the larger-scale games that would really be needing to change this kind of stuff.
(*) It's of course not usually simple, but fundamentally it's still just programming, almost exactly as if you were writing such a system from scratch, except you also have the benefit of being able to leverage the rest of the code provided by the engine and the challenge of having to fit it into the rest of the assumptions provided by the engine.
I work with Unity, and you do not need to create new systems for physics or lightning, as they are highly customizable with a lot of different variables that you can play with. These systems are also so advanced that creating a replica will be, in itself, a big project.
Having said that, if you want to create your own physical world based on new rules, you can do that; and for that project, the existing systems will only help in achieving the end result in a better and faster way.
This answer holds true for other big game engines, like the Unreal Engine or CryEngine. For game engines that do not have their own physics system, existing physics engines such as Box2D can work.