Implementing a scrollable Panel in XNA/MonoGame

Question

I am trying to create a scrolling Panel for my game. The code I've shown below works but there is a problem.

I read Andrew's answer here and decided to implement the scrolling effect by using a Translation Matrix. However my Controls (Buttons, ComboBoxes, etc.) have a fixed Rectangle on the screen (the Rectangle I'm using to check if the cursor is over the Control). As I found out the Matrix translation will only move the texture of the Control and not the Rectangle.

The Problem:

So if I scroll down and click where the button was initially in the panel, it will trigger the Click event even though the button texture is "outside" the scissorRectangle.

What I considered:

1. I thought I should disable interaction with Controls outside of the scissorRectangle, but what about controls that are still partially visible?

2. If Matrix Translation won't work I thought about iterating over the Panel's _controls List and sending the offset to each individual Control (ugh), where it will adjust its Rectangle position. :/

I've got the following code for my Panel:

// ...
private RasterizerState _rasterizerState = new RasterizerState() { ScissorTestEnable = true };
private Vector2 _scrollOffset = Vector2.Zero;
private Matrix _matrix = Matrix.Identity;
// ...

In Update():

_matrix = Matrix.CreateTranslation(new Vector3(_scrollOffset, 0));
if (Enabled && BorderBox.Intersects(inputManager.MouseRect))
{
    if (inputManager.ScrolledDown())
    {
        if (_combinedRowHeight - (ContentBox.Height -_scrollOffset.Y) > -_combinedTextHeight)
        {
            _scrollOffset.Y -= 14;
        }
    }
    else if (inputManager.ScrolledUp())
    {
        if (_scrollOffset.Y < 0)
        {
            _scrollOffset.Y += 14;
        }
    }
}

And then in Draw():

// Save the previous RasterizerState
RasterizerState oldState = spriteBatch.GraphicsDevice.RasterizerState;
// Save the initial scissor rectangle
Rectangle currentScissorRect = spriteBatch.GraphicsDevice.ScissorRectangle;

spriteBatch.End();

spriteBatch.Begin();
// Draw Panel
spriteBatch.End();

spriteBatch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend, rasterizerState: _rasterizerState, transformMatrix: _matrix);

spriteBatch.GraphicsDevice.ScissorRectangle = ContentBox;

if (_drawControls)
{
    for (int i = 0; i < _rows.Count; i++)
    {
        _rows[i].Draw(spriteBatch);
    }
}

spriteBatch.End();

// Restore the scissor rectangle
spriteBatch.GraphicsDevice.ScissorRectangle = currentScissorRect;

// Restore the RasterizerState
spriteBatch.Begin(rasterizerState: oldState);

So is there no other way? Should I just send the offset to Controls and modify their position? Can't the matrix approach be used for interactable Controls?