How to draw tile edges when you don't know where they're going.

This is more of an art question than a programing one but still game development.

I have a tile engine that makes a map randomly from tiles, each tile is a square 3x3 grid. The problem is that while the elements on each tile work well together such as having forests along the top three squares and grass on the other 6, the engine could put the forrests against anything such as rivers, grass, mountains or more forest.

how can i draw the edges of the tiles so the look good no matter what they are places against ?

You are should look into auto tiling. There are two prevailing methods as to how to implement this.

The first assigns a bit value to each tile type, tile_bit, and a bit value to each tile surrounding the location, loc_bit. Multiplying tile_bit by loc_bit will give a unique value for every combination of surrounding locations. This very simple to do with only two or three types of tiles. More details here : http://www.saltgames.com/2010/a-bitwise-method-for-applying-tilemaps/

The second method instead divides each tile up into for sections. These four sections each get a bit value based on their neighbor tile's value. Then you can draw the appropriate tile that matches up with the four bit values. More details here : http://www.codeproject.com/Articles/106884/Implementing-Auto-tiling-Functionality-in-a-Tile-M

Both of these articles assume that you have every combination of transition drawn. This gives you greater control over the look of your map. However, this can get extremely tedious if you have a many tile types. Instead you can dynamics generate edges.

These articles talk about just having generic predrawn edge tiles and assembling them based on your map: http://www.xnaresources.com/default.asp?page=Tutorial:TileEngineSeries:6

http://www.gamedev.net/page/resources/_/technical/game-programming/tilemap-based-game-techniques-handling-terrai-r934

This other topic also talks about even generating edges instead of drawing them. However I think this will just lead to a very crude edge most of the time: Map tile terrain transitions with 3-4 different types

It depends I suppose on how the different regions are rendered? (I.e, Do you generate a forest, then a river, then grass and glue them together? If that is the case you could manually provide the tiles that should go in between? A nice tile that indicates a tree next to a river and so on?

You could even provide a small region in between that interpolates between region A & B and makes the transition smooth. Let the forest end and provide some grass until the river region starts.