I have implemented an isometric grid with rotation in 2 directions in Game Maker. I did this by first calculating the points of the grid in 3D space and then converting them to 2D points through equations I derived from rotation matrices. I use a single object to draw the grid, and would like to be able to click on a tile to select it.
Here are the equations I use to convert the initial 3D points (x, y, z) to isometric 2D points (iso_x, iso_y), where b and a are the angles I'm rotating the grid by:
iso_x = cos_b * x - sin_b * y
iso_y = cos_a * sin_b * x + cos_a * cos_b * y + sin_a * z
I attempted to rework the equations to calculate the tile from mouse coordinates, but I've run into the problem where I have 2 knowns (mouse_x, mouse_y) and 3 unknowns (x, y, z), which will make implementing height difficult.
So I guess I'm looking for suggestions on how to do tile picking better, especially when it comes to choosing tiles with height. If you need extra information, I will try to provide it.
I want to keep the rotations, so moving to sprite-based tiles probably won't work because the shape of the tiles will change as the grid rotates.