Using Python what is the best way to perform heavy tasks in the background?

Question

I'm generating large amounts of procedural map data on the fly, however my game engine doesn't rely on them to render the scene and I'd like to build it on in the background and pop it into the world when it's done (as seen in Minecraft).

I've tried using threadding.Thread but this seems to interfere with PyGame and the game-loop, even with a low amount of threads.

I'm currently trying to get everything happening using the multiprocessing module - although there's a fair amount of work here, and I don't want to find myself in the same case as with Thread.

Is this possible with python? Or should I stop rendering big chunks of data in one go, and rewrite my generation so that it can do lots of smaller jobs spread out over frames in the main thread?

Answer 1

One of Python's huge weaknesses is the existence of a global interpreter lock. Essentially all python code is executed while holding this lock. The consequence is that multi-threaded python applications are effectively incapable of parallelism. Threads allow the convenience of a multi-threaded programming model, but multi-threaded python applications will not be able to take advantage of multiple processors.

To my knowledge, there are two ways of attaining parallelism in Python applications:

Probably the 'simplest' is to use multiple processes. (Using the multiprocessing module) Each process has its own address space, and its own global interpreter lock, so they can run in parallel. The downside of this is that inter-process communication is a PITA in general. Probably the easiest is if the process just writes its output to a file, and the 'main' process pulls data from the file after the subprocess has exited. You might communicate over the process's stdin and stdout (requiring marshalling and unmarshalling of text streams, etc.) You could communicate using sockets, which would let you distribute work over a network for free as a bonus, but which would be even more of a PITA than stdin/stdout. The operating system should be able to provide shared memory and shared synchronization primitives, but I'm not sure if any of those are exposed by any python libs.

Alternatively, you can take advantage of the fact that the global interpreter lock is released for calls to native code. If you're willing to implement your background work in C/C++, you can dynamically link it and invoke it from python. Work done in calls to native functions will be able to run in parallel with python code. Sharing data is more straightforward, but you have to write a bunch of C++, compile it, and get it to dynamically link.

If you hate C/C++, go with multiprocessing. If your background work comes in big monolithic chunks which you're going to end up reading from and writing to disk anyway, go with multiprocessing. If your background work comes in a bunch of bits and pieces which need to be glued together in interesting ways, implement the bits and pieces in DLLs and use python threads for the glue.

Answer 2

So about Python threading, it doesn't do any work concurrently since multiple threads within the same process can only run on 1 core. So all threads within the same process in competition for the same bit of processing power causing the performance issues you described.

The multiprocessing class will utilize multiple cores, since the distributed of the work is then handled by the OS. Sharing data becomes a bit of a problem but I recomend Python's Pipe class.

But to directly answer your question the multiprocessing class will not cause the performance issues threading class is causing you.