There are essentially two things to consider when thinking about the drawback of a larger texture size; performance and build size.
Large textures are more complex to render so have higher requirements of the hardware.
Large textures also make the build size much larger which may be an issue when considering the mobile market - mobile devices have less storage and, in the case of smart phones, the users are likely to want to download the application on a mobile broadband connection - 3g/4g etc - where build size often makes a difference in a person's decision to download an application.
These two factors need to be balanced with what you consider a minimum graphical quality on each platform.
The first thing to note in unity is that it's possible to have platform specific import settings in which you can dictate the maximum resolution for textures on a given platform. This is controlled in the texture importer window. The settings can also be toggled by scripts.
This page on reducing the build size discusses adjusting texture quality. Essentially it discusses the fact that textures tend to be the largest part of the build, so if the build is too big you should change the texture import settings to set a maximum size.
If performance is the only issue, the easiest way to adjust for that is to change the quality settings. It has a texture quality option in which you can tell the system to use full, half, quarter or eighth resolution. So for different platforms you could just set this value to something else. I don't think this affects the build size however.