The Unity default gravity of 9.81 units / second² assumes that one unit is one meter and the game takes place on Earth.
If these assumptions are not true, you would theoretically have to adjust it accordingly. If one unit in your game is actually just 10cm, then you would have to increase gravity to 98.1 in order to be physically correct.
But the question is: Do you actually want the gravity in your game to be physically correct? Human intuition often has a wrong idea of how gravity acts on objects which are very large or very small. People intuitively expect that a humanoid character grown to the size of a skyscraper or shrunk to the size of an ant would move and fall just like a regular sized person in regular gravity, even though that would not be the case.
So keeping the gravity at the default value might actually feel more correct to your players even though it is not. And besides, even games which are actually meant to have a realistic scale and realistic physics sometimes adjust the gravity. Because even though 9.81 m/s² is correct in theory, it might still feel wrong in the context of a particular game. Sometimes gravity is changed globally, sometimes just for specific objects (the jumping mechanics of player-characters in particular rarely obey the laws of physics). The only way to find the perfect gravity value for your game is rigorous playtesting.
I am looking forward to playing your game.