Are there any cases where I should not use object pools but instead rely on
Destroy? (Or more generally, outside of Unity, creating objects on a per-instance basis instead of with a pool)?
It depends why you were using object pooling in the first place. Pooling mainly addresses problems with objects that are expensive to construct and which benefit from locality of reference.
Whenever you have objects that aren't expensive to construct and/or don't need to have high locality of reference, there might not be a lot of benefit to using a pool, and indeed you may start to incur some disadvantages:
Pools generally work on a fixed-block-size principle, and if that fixed size greatly exceeds the number of concurrently alive instances, you're wasting memory.
Similarly, if you can't reasonably bound the maximum number of instances, you're left with either not being able to use a pool, or use one that "grows" by allocating another fixed-size block. This starts to undermine some of the initial advantages of using a pool.
If the cost of resetting a "released" slot in a pool (so it can be used by a future instance) is higher than the cost of initial construction, that's a mark against using a pool for such objects.
If you're using a pool for improved locality of reference you generally want to swap released objects with the last used slot so you have a clear separation in the pool between "alive" and "dead" slots. If the cost of that swap is high, or if some other aspect of the object semantics require the ordering within the pool to be preserved, pools might be a poor choice.