To briefly answer your main question first, the main advantages of a procedurally generated game world are that:
The world can be huge, much larger than any manually designed game world could possibly be.
The world (or at least parts of it) can be regenerated for each game, potentially increasing replay value, since the player will always have something new to discover.
Conversely, the main disadvantages of procedural generation are that:
Depending on the generation methods used, it can be hard to make sure that the world is always playable, i.e. that the player can't get stuck with no way to proceed just because they gut unlucky with the world generation.
Even if you can avoid generating completely unplayable worlds, random world generation can lead to a highly variable difficulty level: sometimes, the player might find everything they need just conveniently lined up for the taking, sometimes they might go a long time without finding anything useful.
Over long periods, such random fluctuations do tend to average out, but if the game difficulty is e.g. heavily front-loaded (typical for survival games), a bad start could ruin a game while a very lucky start could rob it of all challenge.
Poorly done procedural generation can make the world feel monotone and boring: after you've seen a dozen randomly generated mazes, they all start to look the same, even if they differ in details.
Also, if the game world is completely regenerated for each game, there will be no fixed locations or features that the player could recognize from previous games. Such lack of familiarity could make it harder for the player to engage with the game on an emotional level, or to share experiences with other players.
There's also one thing that you could count in either list: in a game where the world is randomly regenerated for each game, there can be no such thing as a "walkthrough" guide, since every playthrough will be different.
(This unrepeatability could also be a problem for testing and debugging, although it can be at least partially avoided by allowing the RNG seed to be specified, at least in debug mode, and/or by providing "cheat" command that can be used to fast-forward play to different stages.)
However, if you look at the disadvantages list, you'll see that it mostly describes the disadvantages of pure procedural generation; a lot of them can be avoided by mixing manually and procedurally generated content.
For example, the player could always start in fixed, manually designed town, perhaps surrounded by a wider more or less fixed home region, but places further away from town could be randomly generated to always provide unfamiliar regions for the player to explore.
There could also be other manually designed towns or other locations placed (more or less) randomly in the world, such that the player will have to find them, but will know what to expect once they get there. Conversely, prominent features such as mountains could be manually placed (at least in the vicinity of the fixed starting location), but the details of their terrain could be randomized.
As for random placement causing deadlocks and difficulty variation, that can be addressed by implementing various consistency and balance checks on top of your procedural world generation algorithm. For example, if the player needs a particular item to be able to cross water, you could include some code making sure that at least one such item is placed so that the player can reach it without crossing water.
Similarly, you might also want to tweak your item placement rules so that no wands of balance-breaking awesomeness are placed before the player has reached level x, and maybe to ensure that at least one is placed in a fixed location where the player can reach it before getting to the stage where they're going to need it to survive.
By the way, a common way to make procedural and manually generated content mesh more or less seamlessly is to start the manual generation process by using the procedural world generator (maybe suitably tweaked, e.g. to generate a village or a mountain peak where you want it) to initialize the region you're designing, and then manually tweaking it to add and adjust the details you want. (This can also allow very compact level storage, by only saving the manual changes and the generator seed and parameters used.)
Other ways to combine manual and procedural content include leaving blank spots in your manually designed levels, to be filled in by randomized content (e.g. a castle could include a randomly generated maze in the dungeons, with only the outline and the exits fixed), or the manual design could specify only the broad outline of some parts of the level (e.g. the placement of houses in a town) and leave the details (such as the exact appearance of each house) to be picked randomly.
In fact, even many games that have entirely fixed game world use some form of procedural generation as part of their level design process, since there are some aspects of world generation (like producing natural-looking terrain, or placing individual trees in a forest) that are difficult and/or tedious to do by hand, but relatively easy to automate.
Conversely, most procedural world generators will use at least some manually designed objects and elements to build the world out of. One could even easily have multiple levels of nested random / manual generation: for example, a procedurally generated world could include a manually generated town surrounded by procedurally generated woods with a manually drawn outline, with procedurally generated trees bearing manually designed leaves.
Also note that procedural content generation isn't necessarily incompatible with a fixed world: you could just pick a fixed RNG seed and use it to generate your world. This can be useful if you want a huge world for the players to explore, but want to keep it the same for every game and every player.
Note that, if you do this (or possibly even if you don't), you really should design your world generator to work in a hierarchical fashion, using multiple RNG instances, such that e.g. the overall map generator would keep a single RNG instance that it would use to generate subregions, assigning to each subregion a different seed value that the region generator would then use to seed a separate RNG instance that it'd use to generate the region, and so on. This is to avoid the "butterfly effect", where changing even the tiniest detail of the tiniest part of the map could throw the RNG out of sync and cause everything else in the world to be completely different.
Another important way to avoid the butterfly effect, particularly if you're generating the world "on the fly" as the player explores it, is to avoid normal RNGs entirely (except for processes that happen "instantaneously" from the player's viewpoint, like generating a new level when the player enters it) and instead use random number generation methods that don't store any internal state. For example, when choosing the seed for a subregion the player is about to enter, the overall world generator could take the coordinates of the subregion (and its own overall seed) and feed them to a hash function to generate the subregion seed. That way, each region will always look the same, regardless of the order in which the player enters them.
Ps. After all that digression, let me briefly address your final question about the actual code used for procedural generation. Alas, I don't think it's really answerable in any meaningful sense without a lot more detail than you've provided, since there are literally as many different ways to do procedural world generation as there are games using them.
For example, the great grand-daddy of all procedurally generated exploration games is probably Rogue, whose level generation algorithm simply consisted of randomly placing a bunch of rectangular rooms within a larger rectangular level, and connecting those rooms with passages (and placing a staircase to the next level in one of them). Its diverse successors, from Nethack to the Diablo series, have elaborated on this system in many ways, but most have retained the basic idea of distinct levels consisting of rooms, dungeons and mazes more or less randomly placed on a grid map.
Conversely, for games featuring open outdoor settings, you'll probably want to start with some kind of fractal terrain generation algorithm and build other features (rivers, forests, towns, etc.) on top of that.
Or you could do something like what Minecraft does, and generate a random 3D procedural texture determining which parts of your landscape are ground and air respectively (scaled and biased so that lower blocks are more likely to be ground), and then overlay features like water, soil/rock, vegetation, etc. on top of it.
Of course, none of the above applies if you're, say, writing a game of space exploration, in which case you'll just need to generate a bunch of randomly placed star systems according to some density function, and then generate a random set of stars and planets in each of them. Or maybe you're setting you players to explore a non-Euclidean landscape, in which case you'll be facing a completely different set of challenges (such as the fact that the volume within a given radius grows exponentially rather than polynomially, making it really hard to keep a large map in memory).
