Note: Variant Prefiltering landed in 2023.1.0a14 and is being backported to other LTS versions. It's not perfect yet (a lot of variants are processed anyway), but you will notice a reduction of time of about one or two orders of magnitude after having it.
Currently, the issue with too many variants and abysmal build times can happen because of two reasons: