A JAR is simply a ZIP file, containing the various class files and other metadata used - it's a completely standardized format.
ZIP files can be extracted, and the class files can be edited as necessary for the modification. The modified files are then added back into the archive, which makes the server use the modified files instead of the original ones.
Technically, Minecraft mods are really "hacked" versions of the original code: the actual binary class files have to be modified to let the game know about the new stuff; there's no official mod support yet (although Notch has stated that they'll work on it during Beta).
Of course, that isn't a very nice way of doing things if you actively design for mods. Instead, you would design some sort of plugin architecture which allows custom code to be run where appropriate. Exactly how you would go about this depends on what you want your mods to be able to do - e.g., if security is a concern, you probably want to design the mod interface to use some sort of scripting, so you can restrict mods from doing all sorts of things to the user's computer.