Unity is a black box. Nobody outside Unity can tell how it really works without doing a lot of reverse engineering.
My educated guess is that Unity saves some sort of metadata along with the raw bytes in a file, and then loads that data straight to GPU memory. I did this with my own engine, with addition that I gzipped the binary data before saving, and then unzipped it when loading. I do that for all assets regardless if they are textures or anything else, so it is not really texture-spesific thing.
Loading compressed data will be quite a bit faster than reading larger amount of data. If Unity do not apply some sort of compression on the raw binary data, then it surely will be slower to read the data from a disc drive. But if the compression algorithm is too complex uncompressing might take more time, and usually it will always take more memory than just copying the raw bytes straight to the GPU. Choosing between these is a design decision they have made.
As it says in the documentation of Texture2D.LoadImage() the function takes PNG or JPG compressed bytes. Loading these images will prbably be faster, but it will get more complicated since you have to set the texture parameters every time you load the images. Also if you want to store compressed image in the GPU memory, you'll have to call Texture2D.Compress() on each texture you want to compress, which will probably make the process slower. That is a tradeoff that you might want to consider; using compressed textures will usually make your frames render faster since less texture reading is needed, but the image quality is then also a little bit lower.
Also mipmapping should be taken in to account, but I did not find any API calls regarding to manually controlling them. So I have to make another guess that unity generates mipmaps for the texture when you use Texture2D.LoadImage(), you wanted them or not. If you did not want them, then it is about 1/3 wasted texture memory. Loading textures using the standard pipeline of Unity, you can control mipmaps a lot better.