Unity just packs the thing that you are really using. So for example if you have 100 audio files and you just have one scene with one audio source where 1 audio file is assigned, then only this particular audio file will be delivered in your build. However when you have a script that has a reference to those 100 audio (in order to e.g. assign them randomly to the audio source), then all 100 files will be packed in the build.
Like DMGregory commented there is one exception: The "Ressources" Folder. All the files inside that folder will be packed in your build, since they are dynamically loadable at runtime, by name (so there is no direct reference in the scenes).