I understand the advantage of hexagonal tiles over square ones. But why aren't octagons used instead? I would think they would provide better, more natural movement in eight directions.
I was thinking about using that kind of map in some game, but I haven't seen any games using it, so I wonder if I missed something obviously flawed about using it?
Octogons:
Hexagons:
The gaps in the octogons make for an unappealing game world.
Typically, if you wanted to allow for eight directions of movement, you would just use squares.
To summarize and elaborate upon what has been said in other answers and in comments, triangles, squares and hexagons are the only mathematically possible regular tilings aka regular tessellations of the Euclidean plane. So yeah, this sucks. Triangles are completely useless here, squares suck because you can't move diagonally without having a somewhat unwieldy factor of 1.4142135623730950488016887242096980785696718753769480 ... give or take; and hexagons suck because you can't even move straight in both directions. Don't get me wrong, I still prefer them over squares within the constraints of the crappy reality mathematics left us with and go Civ5 for finally switching to hex grids. But still, if it were possible to tessellate with octagons, nobody would ever take a second look at hexagons.
You could say "Well, I don't care if there are gaps. I just pretend they aren't there." You'd get the truncated square tiling which is called square tiling not because there are little square gaps but because those octagons are in fact just glorified squares in terms of tiling the plane. Those little squares are what's left from truncating the corners off the squares that would actually tile the plane and in game terms, the reason to not use squares in the first place was to have an equal distance for straight and diagonal moves and this is what you don't have here. Diagonal moves have to bridge the same distance between tile centers as they would with square tiles. Conversely, if you pretend your magic digital space had actual holes, you can of course do that but what's the difference from just using square tiles and making diagonal moves just as expensive as straight ones?
Now this all wouldn't be so bad if there were really good alternatives that aren't Euclidean. Often, our grid is on some kind of planet anyway, so why not use an elliptic geometry, i.e. the surface of a sphere? Unfortunately, spheres are even much, much worse when it comes to regular tilings. Where in the plane you can at least use as many or as little tiles as you like, on spheres there are five arrangements, the Platonic solids. That's it. And only two of them don't use triangles. https://en.wikipedia.org/wiki/Spherical_polyhedra
However, the hyperbolic plane really rocks when it comes to tessellations. There aren't just three, in fact there's an infinite number of regular tessellations, including an octagonal one.
The only problem is that the hyperbolic plane isn't something as nice as a flat surface or a sphere but basically the surface of a Pringle. You'd need one hell of a story hook to justify a game on a Pringle ;)
Still, the octagonal tiling is so elegant and the Poincaré disc looks so awesome that I'm really surprised it's almost never been done (previously I said "never been done" here but then I read MartianInvader's comment pointing to HyperRogue).
Implementation-wise, while I've never done it myself, it should be fairly straightforward to implement this with today's 3D architectures, since a Poincaré disc view can be constructed by putting everything on the surface of a hyperboloid and doing a perspective projection (see Relation to the hyperboloid model).
Just one more thing to conclude this, in case you think about doing a grid-based space game and going to three dimensions, hoping that things might look rosier there ... better just give up. Not only would you need a regular convex polyhedron with 14 faces which doesn't exist, the only way to tessellate 3D Euclidean space with regular convex polyhedra is with cubes. Booooring. In hyperbolic space you can at least get something vaguely like the analogon to a hex grid by tessellating with dodecahedra (i.e. 12-faced polyhedra; that's almost 14, right?) but now you're in total brainfuck land and still haven't got the counterpart to an octagonal tiling:
Beautiful as hell? Oh my God, yes! Would I panic beyond measure if alien spaceships came after me in this and I was expected to react in a sensible way? You bet I would. This is probably the reason why most people just use either cubes or hexagonal prismatic stacks.
The author of HyperRogue here.
HyperRogue actually uses a tesselation made of hexagons and heptagons, here is the reason why this particular tesselation has been chosen, instead of only octagons or heptagons, for example: Hyperbolic geometry in Hyperbolic Rogue Basically, the octagons are too big.
Also some consequences of using hyperbolic geometry in a game (what works in hyperbolic and do not work in Euclidean, and vice versa) are listed in that post.
And yes, as Christian guessed, HyperRogue internally uses the hyperboloid model.
I am not allowed to comment on Christian's answer, but there is a tesselation of the 3D space with 14-faced polyhedra: Bitruncated Cubic Honeycomb (why 14 faces, anyway?)
Basically what you want is a monohedral tesselation (or tiling), that is a coverage of the entire plane (assuming 2d) with a single shape where the tiles do neither overlap nor leave gaps.
There are lots of shapes with which this can be done but when we introduce other constraints, usually orientation should stay the same or they should conform to a natural movement direction, basically only squares and hexagons remain.
Take the triangle for examples (which you might know from the tesselation of 3d objects). To fill the gaps between two triangels another triangle has to be inserted, but flipped upside down. This is obviously a hassle to generate when dealing with sprites for example since a seamless connection is important. Also triangular movement sucks.
The most natural, with regard to movement at least, is the square which happens to be the most frequently used. Hexagons are the next best thing and allow for more direct approach to a higher number of movement directions, i.e. not over the corner movement like 8-way movement on squares does. Usually they are used in more tactical games where the increase in movement is important.
Anyway, if you want to read up more, take a look at http://euler.slu.edu/escher/index.php/Tessellations_by_Polygons.
