With regards to the DX API, your intuition is correct. Variable refresh rate (vrr) displays require tearing to be enabled, meaning Vsync is off. Conversely if Vsync is on, you no longer have a variable refresh rate. So the expectation should be that enabling VSync on a variable refresh monitor caps the refresh rate. This assumes that doing so is supported - ideally if it wasn't supported then it probably shouldn't be an option presented to the user.
Microsoft has some documentation & examples for supporting variable refresh rate displays, including a few points not explicitly called out in their sample code that may require specific attention. In particular, they point out that
Disabling vsync does not necessarily uncap your frame rate: developers also need to make sure Present
calls are not throttled by other timing events...
Regarding Nvidia's control panel settings, things are different. When used with G-Sync (Nvidia's implementation for vrr), the "Vertical sync" option in the control panel no longer acts as V-Sync on/off. When used together the to G-Sync module will compensate for frame time variances output by the system and will fall back on fixed refresh rate if the output exceeds the monitor's upper range. More info on that specifically can be found under this FAQ (in the section "Wait, why should I enable V-SYNC with G-SYNC again? And why am I still seeing tearing with G-SYNC enabled and V-SYNC disabled? Isn’t G-SYNC suppose to fix that?"). Again, this is with regards to how the end user configures Nvidia's control panel whereas the earlier info is with respect to how DX configured by the programmer.