So, fundamentally, the problem here is that to coerce from OpenGL to Direct3D requires bouncing from GPU memory back to system memory and back to GPU memory once more.
You might seriously consider porting the OpenGL DLLs to Direct3D, or the Direct3D application to OpenGL. Ooops, missed that this was over the network anyways.
If you do not, there are two aspects to this performance problem to consider:
1) This is fundamentally going to eat up a lot of bandwidth. You've already discovered the only real technique to help with that: Only call glGetTexImage if you really, really need that data. If the texture didn't change, reuse the result of your last read. If you don't need the texture this frame, wait until the next. Etc.
2) This is fundamentally going to introduce some latency. Worse, your current code introduces a "sync point". GPUs operate asynchronously, a little behind where the CPU is at in terms of drawing your scene - the GPU may very well still be drawing e.g. the previous frame when your CPU is issuing draw commands for the 'current' frame. "glGetTexImage" demands the results for the current frame immediately - forcing the CPU to wait for the GPU to catch up, "synchronizing" with where the CPU is at (hence the term.)
You can reduce this problem by making the readback "asynchronous" - that is, you tell the GPU/driver to start reading the texture back into CPU memory once it's ready, instead of blocking the CPU, and then check the results a frame or two later. This lets you keep the CPU and GPU busy simultaneously. With OpenGL, you can use glReadPixels with a Pixel Buffer Object (PBO) that you copied the texture into (directly from GPU memory to GPU memory) to do this.
Straight from the horse's mouth, here's an article from nVidia on doing this, containing some code snippets, performance benchmarks across GPUs and texture formats, etc.: