If I rotate my button by -180 on the Y axis I can't even click on it. If for example I rotate it 180 on the Z axis everything is fine. Why does this happen and how can I fix it ?
Easy fix is to set your Source Image to None, reset y rotation to 0, Color full transparent, Raycast Target Check. Then you add your Image as a child, set rotation y 180 and Uncheck Raycast Target (not needed anymore because the image is flipped, the raycast will pass trought it anyway). Set your child Image as Target Graphic in the Button component and there you go.