So I have always wondered how one makes a nice (not so squarish) 2d tile map, is it possible? all games now days I think use textured polygons...but my game engine (and engine) doesn't support that to my knowledge. But it does support nice TMX files generated by mapeditor.org's Tiled Map Editor. Though in my game I want nice twisting and turning caverns to traverse ... I was wondering some ideas on such a process... is it in the art style? The type of tile engine? both? So what are some common techniques?
The quick answer is: Both.
The human brain is very good at recognizing axis aligned repeating patterns.. Very very good.. however, rotate a textures repeating edges 30-60 degrees and you will find it hides the repeating edges very well. Now this is more of a discussion for 3D environments than 2D but the same basic principals apply. If you can make it so your the repeating edges of your Texture are NOT the same edges as the repeating edges of your tiles, it will make them a bit nicer.
Another way that this is accomplished in the realm of 3D is multi-texturing. Now this is a much easier thing to do in a 2D environment. Adding in multiple 'tile layers' that render on top of each other would achieve a more varied set of tiles with out requiring a significant amount of new tiles to be created. You would want around 5 layers or so total, 3 layers that are completely behind the player and then 2-3 layers that would render on top of the player, or objects in the scene in general.
The farthest back layer would be the most repetitious and would be the generic surface you are walking on (I am assuming from what your question said that you are a top down or angle down view of the environment like Diablo or so). The next layer would still be commonly repeating but would make each area a bit 'different'. This could be collections of loose rubble, piles of bones, monster tracks and things like this that would be on top of the floor. The third background layer may or may not be needed and would be mostly for specialized tile sets, like a trickle of water or the like.
The foreground layers would be used to add a bit more uniqueness to your view. You could have a character walk behind hanging lamps down the corridors, cobwebs that hint at a large number of spiders in the area or even a completely obscuring object such as a roof brace or the like.
Now the answer is both in that as soon as you have adapted your tile engine to allow multiple layers like this, you will need the artwork to fill both of the layers.. And the key for the artist will to be to create the basic patters and the unique patterns that can be used together in multiple combinations and still look good (and not completely suck up all the memory you have to work with :))
EDIT: From the comment given this gives an entirely new approach, hehe. What is said above still applies, but since we are dealing with large heights (perception wise if we are flying :)) you might benefit from a bit of parallax scrolling (where the tile layers do not scroll at the same rate as each other). This would potentially give you the ability to do deep crevices, or a break through the cloudy sky you are flying through to see the world below. As for the complication implied, its really just a repetition of what you are currently doing.. Instead of rendering a single set of tiles at a time you will do it a few times.. It should not be too much of a stretch from where it sounds like your current game and engine are :)
Anywho, hope this was enough of an answer to help you progress or at the least have new questions to ask.