Thanks to @Hesamom I found a solution:
All you need to do is to add AsyncOperation.allowSceneActivation = false
so it won't activeactivate itself right after loading, and then change it to true
once you did all you wanted (usually at the last line of coroutine)