(You may to know about the term "monkey patching" or "duck punching" if for nothing other than the humorous mental image.)
That aside: if your goal is decreasing iteration time for "behavior" changes, try some approaches that get you most of the way there, and combine nicely to enable more of this in the future.
(This will go out on a bit of a tangent, but I promise it'll return!)
- Start with data, and start small: reload at boundaries ("levels" or the like), then work your way up to using OS functionality to get file change notifications or simply poll regularly.
- (For bonus points and lower load times (again, decreasing iteration time) look into data baking.)
- Scripts are data, and allow you to iterate behavior. If you use a scripting language, you now have the notifications / ability to reload those scripts, interpreted or compiled. You can also hook your interpreter to an in-game console, a network socket, or the like for increased runtime flexibility.
- Code may be data too: your compiler may support overlays, shared libraries, DLLs, or the like. So you can now choose a "safe" time to unload and reload an overlay or DLL, whether manual or automatic. The other answers go into detail here. Do note that some variants of this may mess with cryptographic signature vecification, NX (no-execute) bit, or similar security mechanisms.
- Consider a deep, versioned save/load system. If you can save and restore your state robustly even in the face of code changes, you can shutdown your game and restart it with new logic at the exact same point. Easier said than done, but it is doable, and it's markedly easier and more portable than poking memory to change instructions.
- Depending on the structure and determinism of your game you may be able to do recording and playback. If that recording is just over "game commands" (think of a card game, for example), you can alter all the rendering code you want, and replay the recording to see your changes. For some games this is as "easy" as recording some starting parameters (e.g. a random seed) and then user actions. For some it is much more complicated.
- Make efforts to reduce compile time. In combination with the aforementioned save/load or record/playback systems, or even with overlays or DLLs, this may decrease your turnaround more than any single other thing.
Many of these points are beneficial even if you don't get all the way to reloading either data or code.
On a large PC RTS (~120 person team, mostly C++), there was an incredibly deep state saving system, that was used for at least three purposes:
- A "shallow" save was fed not to disk but to a CRC engine to ensure that the multiplayer games remained in lock-step simulation one CRC every 10-30 frames; this ensured no-one was cheating and caught desync bugs a few frames later
- If and when a multiplayer desync bug occurred, an extra-deep save was performed every frame, and again fed to the CRC engine, but this time the CRC engine would generate many CRCs, each for smaller batches of bytes. In this manner, it could tell you exactly which portion of state had started to diverge within the last frame. We caught a nasty "default floating point mode" difference between AMD and Intel processors using this.
- A normal depth save might not save e.g. the exact frame of animation your unit was playing, but it'd get the position, health, etc of all of your units, allowing you to save and resume at any time during gameplay.
I've since used deterministic record/playback on a C++ and Lua card game for the DS. We hooked into the API we designed for the AI (on the C++ side) and recorded all of the user and AI actions. We used this functionality in game (to provide a replay for the player), but also to diagnose issues: when there was a crash or odd behavior, all we had to do was get the save file and play it back in a debug build.
I've also since used overlays more than a few times, and we combined it with our "automatically spider this directory and upload new content to the handheld" system. All we'd have to do is leave the cutscene/level/whatever and come back in, and not only the new data (sprites, level layout, etc) would load but also any new code in the overlay. Unfortunately that's getting much harder with more recent handhelds due to copy protection and anti-hacking mechanisms that treat code specially. We still do it for lua scripts though.
Last but not least: you can (and I have, in various very small specific circumstances) do a bit of duck punching by patching instruction opcodes directly. This works best if you're on a fixed platform and compiler, though, and because it's nearly unmaintainable, very bug-prone, and limited in what you can accomplish quickly, I mostly only use it to re-route code while debugging. It does teach you a hell of a lot about your instruction set architecture in a hurry, though.