You can implement this using Layers.
Create two new layers "Ghost" and "Player".
Assign the "Ghost" layer to the players who are ghosts.
Assign the "Player" layer to the players who are not ghosts and to any other objects which are not supposed to interact with ghost players.
Keep anything which is supposed to interact with players and ghosts equally (like walls, trees and terrain) on the already existing "Default" layer.
Set the layer collision matrix under Edit -> Project Settings -> Physics -> Physics Manager as follows:
- "Ghost" collides with "Default" but not with "Player" and "Ghost" (unless, of course, you want ghosts to interact with each other)
- "Player" collides with "Default" and "Player" but not with "Ghost"