pygame performance issues

Question

I'm trying to make a tactics game using python and the infamous pygame library. I use threading for networking and pathfinding. The game uses a p2p lockstep architecture, with one of the players acting as a host. The server starts 2 threads (send / receive) for every connected player. All the connected clients also start 4 threads during their lifetime, 2 for networking, one for the main game loop and a new temporary thread for each pathfinding request.

My problem is that the program uses a lot of CPU and memory, even for very small maps, even when not blitting any sprites with transparency or running any pathfinding algorithms the program is slow and CPU intensive.

The map in my game is built out of tiles, which are 96x96 pixel pygame surfaces, which map sprite images are blitted to

# draw the map
for element in self.atlas.all_objects:
   try:
       self.map_surf.blit(self.map_sprites[element.sprite], element.rect)
   except:
       pass

The map, along with any units are finally blitted to the screen, simply

self.screen.blit(self.map_surf, self.map_surf_rect)

A very small map surface, with 8x8 tiles will make my CPU (amd 8 core 3.10ghz) run at 25%, with 70mb memory used per client (the cpu-fan goes crazy). The frame rate is still normal at this point.

A slightly larger map, 16x16 tiles shows a drop in frame rate, down from the desired 60 to around 48.

A map of what I'd want to be small/normal size for a real game, 40x40 tiles, brings the fps down to 12 and uses a surprising 215mb memory (all the spritesheets combined are 22mb, and I'm only using a basic green tile for testing), with the same CPU utilization as before.

On the same 40x40 map, replacing the image sprites with blank 1x1 pygame.Surface objects results in a frame rate of 27 (it's set to 60)

My basic question is this: Is the problem likely to be in my code, or is pygame itself horribly inefficient? If the latter, is there anything short of learning to code c++ that can replace pygame for these purposes?

If the problem is more likely to be in my code, how can I find out what it is?

EDIT: I profiled my code, and here are some of the highest numbers

map size 40 x 40 tiles:

4606119 function calls (4606113 primitive calls) in 145.974 seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

127128    0.225    0.000    0.225    0.000 vec2d.py:18(__len__)
218530    1.406    0.000    1.406    0.000 geometrical.py:64(__init__)
253534    0.544    0.000    0.544    0.000 vec2d.py:21(__getitem__)
1400934    2.309    0.000    2.309    0.000 {method 'append' of 'list' objects}
4626   20.189    0.004   20.189    0.004 {method 'fill' of 'pygame.Surface' objects}
2086545   64.417    0.000   64.435    0.000 {method 'blit' of 'pygame.Surface' objects}
1289   10.843    0.008   95.724    0.074 bullet_client.py:714(draw)

So it appears a lot of time is spent appending items to lists, but I'm sure not what kind of other, more efficient method of organizing my game there is. As expected however, blitting pygame surfaces trumps the other functions by a huge factor, as shown by the cumtime (whatever that is).

So I guess the question is the same, can pygame be replaced by something, or might there be a more shallow problem in my code still?

Answer 1

Ensure your worker threads aren't sitting there spinning and forcing unnecessary time sharing across your CPU. This can be solved using manual resets and signals, rather than a spin lock.

Additionally, consider profiling major portions of your code to identify hot spots, my research brings up the fact that python itself includes a respectable profiler called cProfile (https://stackoverflow.com/questions/582336/how-can-you-profile-a-python-script). Use this to profile individual systems, such as your main update, rendering code, path finding algorithms, etc.

Finally, any interpreted language is going to be relatively slow compared to a direct compiled language, but your game hardly sounds intensive enough that it should be causing this problem.

Extra note When rendering anything, consider how to minimize the number of render calls. Render calls through a graphics API are expensive to make, and thus all production rendering pipelines provide methods for sorting, and batching entity render calls. An example would be rendering of a particle system. Rather than loop through each particle within a system and allow it to render itself, a system (or collection of systems) would be batched together to limit the number of texture, geometry and buffer transfers to the GPU.