the app crashes when it reaches 1.5Gb.
This strongly suggests that you're not representing your tiles correctly, as this would mean that each tile is ~80 bytes in size.
What you need to understand is that there needs to be a separation between the gameplay concept of a tile, and the visual tile that the user sees. These two concepts are not the same thing.
Take Terraria for example. The smallest Terraria world takes up 4200x1200 tiles, which is 5 million tiles. Now, how much memory does it take to represent that world?
Well, each tile has a foreground layer, a background layer (the background walls), a "wire layer" where wires go, and a "furniture layer" where furniture items go. How much memory does each tile take up? Again, we're just talking conceptually, not visually.
A foreground tile could easily be stored in an unsigned short. There aren't more than 65536 foreground tile types, so there's no point in using more memory than this. The background tiles could easily be in an unsigned byte, as there are fewer than 256 different types of background tiles. The wire layer is purely binary: either a tile has a wire in it or it does not. So that's one bit per tile. And the furniture layer could again be an unsigned byte, depending on how many possible different pieces of furniture there are.
Total memory size per tile: 2 bytes + 1 byte + 1 bit + 1 byte: 4 bytes + 1 bit. Thus, the total size for a small Terraria map is 20790000 bytes, or ~20MB. (note: these calculations are based on Terraria 1.1. The game has expanded a lot since then, but even modern Terraria could fit within 8 bytes per tile location, or ~40MB. Still quite tolerable).
You should never have this representation stored as arrays of C# classes. They should be arrays of integers or something similar. A C# struct would work as well.
Now, when it comes time to draw part of a map (note the emphasis), Terraria needs to convert these conceptual tiles into actual tiles. Each tile needs to actually pick a foreground image, background image, an optional furniture image, and have a wire image. This is where XNA comes in with its various sprite sheets and such.
What you need to do is to convert the visible part of your conceptual map into actual XNA sprite sheet tiles. You should not be trying to convert the entire thing at once. Each tile you store should just be an index saying that "I'm tile type X," where X is an integer. You use that integer index to fetch which sprite you use to display it. And you use XNA's sprite sheets to make this faster than just drawing individual quads.
Now the visible region of tiles needs to be broken up into various chunks, so that you're not constantly building sprite sheets whenever the camera moves. So you might have 64x64 chunks of the world as sprite sheets. Whichever 64x64 chunks of the world are visible from the player's current camera position are the chunks you draw. Any other chunks don't even have sprite sheets; if a chunk falls off the screen, you throw that sheet out (note: you don't really delete it; you keep it around and respecify it for a new chunk that may become visible later).
I'd like to have whole map processed at least on server app (and on client if possible).
Your server does not need to know or care about the visual representation of tiles. All it needs to care about is the conceptual representation. The user adds a tile here, so it changes that tile index.