Okay, found the solution. One of my elements had a script on it.
So that script was calling other scripts, which resulted in a spider web of dependencies, which also included a lot of seemingly unrelated assets.
I just exported the asset without the script and carried over the said script manually.
My scripts are pretty connected I guess.
I'm still open ...