How can a resource manager have dynamic loading/unloading without creating dangling pointers?

Question

I have a resource manager that manages the lifetime of resources in memory. A "resource" is something like a texture, mesh, et cetera. I also have an entity class that, the base class for things in my game world, which has pointers to individual resources.

I'm thinking about giving my resource manager a memory quota so it can free and reload resources as it needs to in order to satisfy this quota, and I'm wondering how I could go about this without ending up with dangling pointers in my entity objects.

I'll need the ability to free up a mesh or texture at any time in the resource manager.

I could use shared_ptr in the resource manager and weak_ptr in the entities (as long as I don't expose a shared_ptr out side of the resource manager, I'd still get the ability to free them on demand), but I am more interested in how you would do it without those classes, for example before C++11 existed or before Boost's smart pointer library existed.

Answer 1

Instead of holding raw pointers, hold some other lightweight reference to the resource. A shared_ptr and corresponding weak_ptr instances are one such mechanism, but you can easily implement your own if you don't have access to those types (or their earlier Boost equivalents) or there are other reasons why those types would not be appropriate for your scenario (such as being too heavyweight).

This kind of 'handle' system can be as simple as handing out integers which refer to internal slots in the resource system, which can then be used to dynamically reload a resource if access to it is requested while it's been unloaded.

In practice you probably want something a little more robust, at least a way to treat the handle "like" a pointer and provide operator -> overloads, for example, by wrapping it in a Handle type. That Handle type can just contain a pointer to a "control block" stored within the resource manager. The control block has the raw resource reference and a boolean flag that indicates if the resource is alive or not, and some kind of key that can be used to reload the resource if needed (I assume a string in the example below). A very simple example of this might look like:

template <typename Resource>
struct Handle {
    Resource * operator -> () }
        return m_owner->retrieve_resource(m_resource_id);
    }

private:
    int m_resource_id;
    resource_manager * m_owner;
};

namespace detail {
    struct control_block {
      Resource * resource;
      bool is_alive;
      std::string key;
    };
};

struct resource_manager {
    Resource * retrieve_resource (int id) {
        auto block = m_resources[id];
        if (block->is_alive) {
          return block->resource;
        } else {
          block->resource = reload_resource(key);
          block->is_alive = true;
          return block->resource;
        }
    }

private:
    std::vector<detail::control_block> m_resources;
};

In this implementation, the resource manager just stores a flat array of control blocks which can be referenced by an integer, and provides a method to recover a resource pointer by ID, reloading that resource if needed. The Handle class is just a useful wrapper around that behavior. Obviously this is a very simple example, though. If you want something more complete, consider Scott Bilas' article on a generic handle-based resource manager. It takes a somewhat different approach, but is well worth the read.

(Be warned that a need your question implies may mean your resource utilization is very chaotic in nature, and that isn't necessarily a good thing for performance, so you might want to consider that.)

Answer 2

Smart pointers or handle classes, just like C++11. You could write your own long before C++11. I would even argue that the shared_ptr in C++11 is probably the wrong model entirely.

All you do is make a type that holds a reference to a resource. This reference could be a pointer with some kind of lifetime management or it could just be some kind of numeric identifier used as a key into a data structure (array index, map key, whatever). Use that to access resources. Similar to how weak_ptr works, except you shouldn't need to worry about "locking" them with a shared_ptr first. Resources should only ever be released after all logic and rendering for the frame is done and there's no possibility of dangling pointers remaining (assuming you use the appropriate handles everywhere in your non-temporary data structures).

Note that just paging out resources randomly is a bad idea. If an object exists and says it needs a texture, it needs that texture. If you're running out of memory, either (a) target better hardware, (b) make a smaller game, or (c) implement a more sophisicated resource management system that can more accurately determine the current working set and appropriate times for objects to release or reacquire their resources (using level-of-detail, area-of-interest, or so on).

Answer 3

I've recently redesigned a hobby game engine to use handles for nearly all dynamically allocated objects and I highly recommend this architecture. A shared_ptr is somewhat of an anti-pattern that should be avoided if there's an alternative. I admit that I also originally tried shared_ptr too, but they do not provide centralized control over their memory. After my resource manager lent a shared pointer to the "outside world", I have no power to free that memory even if I'm certain that the object should be destroyed (for example if a game sprite/character dies). I also understand your problem with dangling pointers because using raw pointers/reference is actually quite risky in general. Below I'll provide an elegant solution that works whether you plan to use this system as either a resource OR cache manager. Some of the inspiration for my solution with handles came from here and studying Rust's "reference borrowing" paradigm.

Terminology of the components

Your design should include 3 basic components (classes):

• Handle
• Slot
• Store

These will be used to manage whatever Resource objects you wish.

Resource

Starting with the Resource, this is a base class of an object you wish to dynamically allocate/delete and access via a handle. You will likely have hundreds to millions of these resources in your game. The raw pointer to the Resource's memory address will only be accessible via the Store's API (and optionally directly via the Handle API) upon request.

Handle

The Handle class should be a cheap simple object that can be packed into an equivalent single 32/64 bit integer value for (de)serialization. They should act as a unique identifier "key" or "UID" to your resource. I think of them as "safe-pointers" because when being accessed, they'll return nullptr if the memory they were referencing was freed (rather than a dangling pointer).

I typically define a Handle class with just 2 integer members: index and token which could also be template typename integer types rather than hard-coded uint32_t, as you see fit:

template <typename Resource>
class Handle {
    uint32_t  index;
    uint32_t  token;
public:
    Resource* get() const; //Optional, but very useful
    void      delete();
    void      clear();
    Resource* detach() const;
    uint64_t  asInt() const;
    virtual uint8_t type() const = 0; //To override
};

This is just pseudo code to give you a basic idea of the API. It is neither exhaustive nor demonstrating the implementation.

Index

The integer index identifies a unique Slot in the Store's vector to obtain the resource from. Its bit-width must be large enough to store the number of objects your program expects to need simultaneously (length of the vector). This acts as a "virtual address" allowing fast O(1) lookup to obtain the resource from the Store's vector (explained more below).

This demonstrates yet another excellent feature of handles: since they use an integer index as a kind of "virtual address", the handles remain valid even when running the program on a different process later via (de)serialization (storing on disk), or if sharing them across multiple clients over network (for multiplayer games)! You'll need some extra work to ensure your Store's Slots remain synchronized across all clients simultaneously.

You should also reserve the 0'th slot index as a NULL slot which is read-only (always returns a nullptr Resource).

Token

The token is an integer that gets incremented each time a Slot's Resource* is overwritten to prevent a stale Handle from accessing a Resource* which has been deleted. This is how dangling pointers are completely averted and allows a Slot to be reused after their Resource* is deleted. A nice feature is that all handles pointing to that Slot become stale/invalid automatically. The token bit-width should be large enough to accommodate the number of times you expect resources to be deleted per Slot without exhausting the token. I'll explain more info later on exhausted tokens.

Type

Optionally, the Handle could know its type, specifically when (de)serializing the Handle. I would recommend having a purely virtual method Handle::type() and deriving a sub-class for the handful of resource types: SpriteHandle, DataHandle, EntryHandle, NodeHandle, etc. You'll quickly see how nicely this helps you detect any compile-time issues with mismatching handle types in your code. Ensure that each sub-class has a unique static integer/enum type and that the type's bit-width is large enough (probably an 8-bit unsigned is fine).

Note that you'll need to subclass the Store and Slot classes accordingly for each Handle sub-class (the examples above use templates to achieve this).

Suggested Handle Methods

• get(): [Optional] Directly return the raw pointer, for example with a get() method similar to a smart pointer. Here the implementation of the Handle must know about the Store from which it can obtain the Resource*, in other words it simply calls the cumbersome Store::get(handle) within.

This may require handling circular dependencies with forward declarations. Otherwise, it works really well when each type of Handle is associated with 1 corresponding Store to obtain the resources from.

• delete(): Tell the slot to directly free the memory it contains (and nullify its pointer). Note that this MUST be called at some point, otherwise the resource will remain in the store indefinitely ("soft" memory leak).

• clear(): Flags the handle as referencing the NULL slot, which is as simple as setting Slot::index=0.

• detach(): Returns the raw pointer AND frees the slot (returning ownership of the allocated memory back to the call site). Be careful that something else takes responsibility for freeing the memory externally.

• asInt()/asLong(): Returns this handle represented as a simple unique integer, basically for (de)serialization. This is highly beneficial for persistent data when saving an entire Store to file, for example.

Slot

template <typename Resource>
class Slot {
    uint32_t  token;
    Resource* resource;
public:
    Resource* get(uint32_t token);
};

The Slot class is not part of the externally accessible API, but internally connects the Handle to the Store. The Slot also needs an equivalent integer token (matching bit-width to the one in the Handle class) and a raw pointer to the Resource*. It's entirely responsible for ownership of its raw pointer Resource* memory and giving the outside world temporary access to it. Therefore, it must delete the pointer when being destructed or overwritten. Furthermore, whenever a Slot is "overwritten" by receiving a new Resource*, its Slot::token must increment by one. By incrementing the Slot::token, any handles "out in the wild" have just automatically become invalidated (stale) and will never be able to access the freed Resource* again.

Note: A Slot must only be accessed via a Handle, and the values of both tokens must match (Handle::token == Slot::token), otherwise the handle is considered "stale" and the slot should return nullptr rather than the raw pointer Resource*.

This clear responsibility makes the system very robust, and although memory can still "soft leak", you'll easily be able to determine (while debugging) which slots in the store you've forgotten to delete resources for.

Slot "Hot-Swapping"

Another great feature is that you can hot-swap a Resource* without invalidating any existing handles! Leave the Slot::token unchanged and simply replace the Slot::resource* pointer. Now all your corresponding handles "out in the wild" are automatically updated to obtain the new resource!

Store

template <typename Handle, typename Resource>
class Store {
    std::vector<Slot<Resource>> slots;
public:
    Handle    add(Resource* res);
    Resource* get(const Handle& handle);
};

Lastly, the Store class is responsible for storing a vector of Slots and determining which empty Slot to assign new Resource*s to. There are some implementation-specific techniques to optimize this search (such as memorizing the lowest index that is available then searching upward). Just be aware that it may potentially be time-consuming (O(N)), or complex to reach something closer to O(log(N)). The assumption is that you won't need to create them as often as you'll need to access them.

If you have a multiplayer game, you should add functionality to keep Stores synchronized across multiple clients.

Since a Handle directly knows the vector index to obtain the corresponding Slot, there remains only performing a check that the token matches between the Handle and Slot before returning the raw pointer Resource*. This is a very fast O(1) operation similar to that of a smart pointer, and guarantees that a "stale handle" will never retrieve a new resource which has since overwritten the old Resource* in the Slot, nor will it ever return a dangling pointer.

Note: the vector should never erase() or shift any of its elements as this would corrupt any existing handle indexes (i.e. virtual addresses), pointing them to the wrong slots. The vector should typically only grow in size as more slots are needed.

Exhausted slots

If a Slot::token ever reaches its maximum unsigned integer value then that Slot has become "exhausted" and needs to be disabled. Otherwise, if the integer overflows, you'll run the (very low) risk of a stale handle being able to access a Resource* that it should not (token collision). Make the token bit-width large enough so that you won't typically encounter this issue. Once a Slot is disabled, the store will no longer be able to assign new Resource*s to it. If you know with certainty that there are no handles remaining "in the wild" that point to the disabled slot, then you could re-enable that slot and reset its token back to 0, but I'll leave that as an exercise for the readers to solve.

Hint: A "reference counter" stored in the Slot might be a useful approach if you're interested in knowing how many handles are actually "out in the wild". This is similar to how a std::shared_ptr works in C++. Each time a handle is lent out or cloned, the Slot::counter is incremented, and each time a handle's destructor is called it decrements the reference counter. But this also adds complexity to the Handle class which I haven't found to be useful or necessary (yet).

Usage

Remember, Handles are cheap objects (similar in size to raw pointers) and its primary method handle.get() is O(1) fast (comparable to a smart pointer's get()), so don't be afraid to call it frequently. The idea is that Handles should replace raw pointers as much as possible in your code! Treat them like "safe-pointers" and do not try to get() their raw pointers until the latest possible moment:

1. "Borrow" a raw pointer Resource* with Handle::get().
2. Check if the raw pointer is null (most likely a stale handle).
3. Safely de-reference and use the Resource& as necessary.
4. Immediately "forget" about the raw pointer.

In conclusion, do not retain the raw pointer for long, because the longer you keep it memorized somewhere, the more likely it becomes dangling. You can still pass the raw pointer/reference as argument to nested methods, but the main goal is that you're only temporarily borrowing it and must NOT memorize it outside the scope of where it was originally obtained. If you need the raw pointer again later, just call Handle.get() again. Normalize keeping the Handles memorized for as long as you need, knowing that it could become stale at any moment and expect that its returned resource could be nullptr.

Note: If you have multi-threaded application then that's a special case where it might be possible for the raw pointer to become dangling within the scope that it was obtained (if another thread deletes the raw pointer simultaneously).

Best of luck with your implementation!