Yes, you can open source your own code and your own assets independently of any third party assets that your project may need or may not need. However, the whole project cannot be made open source that way, only part of it. You cannot include the third party assets that are under a restrictive license.
In order to do that, I would recommend you to just leave them out of the Git repository. Either modify the .gitignore file to exclude the third party assets from the Git but keeping them locally, or put them in a Git submodule, which you do not publish (that would give you the advantage of being able to version them), or do just not add them to the Git repository.
It should be explained somewhere in your project that certain assets are required for the project to run, but are not included in the repository. It's as simple as that.
For binary distributions, it would depend on how you package a Unity app. However, without the third party assets your project will not work right out of the box, unless these assets are made optional. Maybe that could be accomplished easily? Try to make the usage of these assets optional, maybe with some kind of fallback assets if they are not existing. But even without replacements, as long as the building process doesn't fail and the app doesn't crash without the optional assets, this would be a very comfortable solution for others to share and remix your project.