I am making a game engine with C++ and OpenGL. Currently I'm stuck making the asset explorer. I would like to be able to import assets into an engine project and then when I open the project just use them in various game scenes (like in Unity). However the only thing I am doing right now is loading everything from the 'Assets' folder of the opened editor project. That is incredibly slow even when done asynchronously with multithreading. I don't understand how does Unity load assets so fast. I've noticed that it takes a while to import the assets for the first time but then it's very fast anytime you open the project after the first import. Is there any way to cache loaded assets? And should I generate all the buffers for OpenGL right on the asset import or should I wait for the asset to be 'used'?
I ended up solving the problem (thanks to DMGregory's hints) by generating meta files for each asset. The meta files contain the path to the preview image used in the asset explorer window, and import settings. Only when the asset is 'inspected' I fully load it into memory. That way the engine start's niceley without having to wait for files to load everytime, excpet for the first import where the meta files are generated.
If I understand correctly, you have a game engine with a GUI, and you want to display to the programmer the assets that the current project is using?
A lot of GUI programs (I believe unity too) create small thumbnails for images, which are a lot smaller in size than the original assets, so they can be displaying without issue. How big or small those should be depend on what your GUI looks like. You can make them as small as 50x50 and it will still be recognisable.
Assuming that most assets are 200x200 or more, that means each asset now needs x4 times less resources, which can make loading faster but also reduced the amount of memory your engine needs to run. Keep in mind that even if the texture is resized for display purposes, when compiling your game you take the original texture, so the game is not affected by this change.
You may want to save the cache the resized images the first time this happens, this is why most programs take a while the first time they run and run way faster afterwards.
If you want to account for project scalability, you'd probably want to separate your assets into "categories" as well (maybe like per scene?) and load a specific group of assets when needed and not all assets at all times.
Note: Numbers are random-ish, you can use different numbers based on the needs of your project. The image was taken from Wikipedia - Cat intelligence