How to select only inventory slots, and not deselect them when clicking somewhere else?

Question

I'm trying to make a selection system in my inventory to do things to the selected item in a UI window using this selection system.

The problem is that whenever I click on anything other than the inventory item, it de-selects it.

I would like to de-select the item only if I click on an empty slot, and not de-select it outside of the UI or on another UI image.

public void OnPointerDown(PointerEventData eventData)
{
    if (eventData.button == PointerEventData.InputButton.Left)
    {
        if (transform.childCount != 0)
        {
           
            EventSystem.current.SetSelectedGameObject(gameObject, eventData); //selecting the inventory slot
        }
    }
}
public void OnSelect(BaseEventData eventData)
{
    InventoryItem itemInSlot = gameObject.GetComponentInChildren<InventoryItem>(); //getting inventory item from slot
    Debug.Log("Selected" + itemInSlot.name);
    itemInSlot.isSelected = true;

    tooltipBar.SetActive(true);
}

public void OnDeselect(BaseEventData eventData)
{
    InventoryItem itemInSlot = gameObject.GetComponentInChildren<InventoryItem>();
    Debug.Log("De-Selected" + itemInSlot.name);
    itemInSlot.isSelected = false;

    tooltipBar.SetActive(false);
}