How to free the GPU from a long task?

Question

I'm developing an application that uses the GPU. Some tasks might be rather long (like a fragment shader with a loop). I have the impression that I can make the entire OS visually freeze by asking the GPU to perform a job that doesn't end. Of course, I'm not doing this on purpose, but when that accidentally happens, I have to force power off my machine and this is very difficult to debug.

Some questions:

1. Is this behaviour normal? Or could this be a bug in the hardware or driver?
2. Is there a technique to recover from this without force stopping my machine?
3. Since the OS is still functioning (only not rendering), I could try to ssh into my machine and kill the process. Would that free the GPU from its job? Update: I tried to ssh into my machine and sudo kill -9 *pid*, but didn't work.

I would like to know if there is an OS independent answer for this. If there isn't, my setup is macOS, with OpenGL (running on Intel HD Graphics 3000).

Answer 1

The short answer is: you can't right now. (possible work-around below)

Newer GPUs can do preemtion (context-switching) to prevent the entire machine from freezing when running long GPU programs, older GPUs don't support this.

Your OS also needs to support this as well as the specific drivers.

For example Windows 8 WDDM 1.2 requires this support in drivers ( https://msdn.microsoft.com/en-us/library/windows/hardware/jj553428(v=vs.85).aspx )

But even then you can't stop the GPU yourself. When this is supported your app can be killed externally but it does not allow your own application to directly abort the GPU operation.

---

A possible work-around would be to break down your shader loop into multiple shader passes and synchronize draw calls with the CPU (greatly reducing overall performance) so that the CPU can stop sending draw commands and also allowing the CPU to yield to other programs.

Note that it is possible the GPU driver will not yield, instead batching up all commands, until a frame buffer access is done or some kind of feedback information is requested from the driver.

Reading 1 pixel from the frame buffer between passes should be sufficient to cause a pipeline stall (GPU yield).

Using OpenCL instead is another option. OpenCL execution kernels are expected to take a fairly long time compared to pixel/fragment shaders so it is more likely that the drivers will support preemption and/or the driver will not use all available cores for OpenCL tasks, leaving some for the OpenGL tasks so the drawing can still crawl along.

There is no guarantee any of these work-arounds will work on all GPUs and all drivers. Some drivers are very performance-aggressive and will not yield for a long time to maximize the frame rate of the application with focus.