What you need to do is separate terrain from live blocks. For example you could store the live blocks in a dictionary that uses a 3d point as key. And then unload the terrain. This way your live blocks stay in memory in a way you see fit, and you can still look them up based on position, but the terrain is stored on disk for later retrieval.

This will increase memory a bit, but you can't avoid it entirely.

What you need to do is separate terrain from live blocks. For example you could store the live blocks in a dictionary that uses a point as key. And then unload the terrain. This way your live blocks stay in memory in a way you see fit, and you can still look them up based on position, but the terrain is stored on disk for later retrieval.

This will increase memory a bit, but you can't avoid it entirely.