Don't run the Jenkins agent using Windows Services. Instead, launch the agent manually using CLI.
This seems to be a bug with Unity, I've been unsuccessfully trying to create a small repro case that I could submit to Unity.
I also encountered this problem. For me it happened when I was running multiple Unity instances at once on a single Jenkins node what the Jenkins agent was running as Windows Service.
The problem turned out to be that the amount of Windows Desktop Heap available to services, as opposed to applications, was too small. I worked around it by increasing the ...
Another condition for this error message occurring repeatedly is when multiple Unity builds are running at the same time on Jenkins.
One possible workaround solution for this is to reduce the # of executors in Jenkins to 1 (from 2 in my case) so that only 1 Unity build can occur at a time.
I have Unity 2017.3 installed, and I did not have this issue ...