I think this might be easier with a visual aide. Here are some different things we might mean when we ask for "tiles within a given radius"
From the OP's responses above, it sounds like the top-left version is desired (Euclidean - screen space). That seems to have generated a bit of confusion since most tile-based games consider ranges in tile space.
No matter which metric you use, a useful strategy is to limit your search area.
You can take advantage of the fact that in all of these metrics, the desired tiles occur in runs with no gaps in the middle (unless your tiles are sparse). I'll assume your tiles are densely packed in rows & columns (either screen-aligned or tile axis-aligned - it changes the details but not the main idea).
- Determine the least row that could possibly overlap your radius.
- Determine the greatest row that could possibly overlap your radius.
- For each row in that range, determine the least & greatest columns that could possibly overlap your radius.
- Return all tiles within these bounds.
As an example, I'm going to use the tile space Euclidean metric (top right), for simplicity.
for(row = ceil(sourceRow - radius) to floor(sourceRow + radius))
rowDifference = row - sourceRow
columnRange = sqrt(radius * radius - rowDifference * rowDifference)
for(column = ceil(sourceColumn - columnRange) to floor(sourceColumn + columnRange))
yield GetTile(row, column)
(Please excuse the sloppy pseudocode and lack of bounds-checking to ensure we're within the map bounds - those should be easy to add)
This version does only O(radius) distance calculations (linear in the radius, rather than quadratic as with methods that test every tile in a neighbourhood, or the brute force O(n^2) which tests every tile on the map)
If that's not directly applicable, it shouldn't be hard to adapt the basic strategy to a different metric/tile arrangement.