I wouldn't expect significant performance disadvantages to using a ConcurrentDictionary in your case, but as with anything concurrent I'd recommend tests/profiling.
Also, I haven't the foggiest idea what the big engines are using. Sorry.
That said, let's check your 4 uses against the information here: http://arbel.net/2013/02/03/best-practices-for-using-concurrentdictionary/
(Interesting read, esp. for cautionary notes.)
- Updating should be no different, though you'll want to think about whether it's an issue for you that the enumerator can change during the loop. (The enumerators aren't snapshots.)
- You may need some bookkeeping to control updates on mid-update-loop additions. It seems like these additions won't cause significant contention thanks to the multiple-lock design, but you'll want to verify that for yourself.
- Retrieving "by index" seems a little fishy to me, depending on what exactly you have in mind. Linq seems fine.
- Removing should be fine, though again the enumerator can change so be careful with the parallel loop.
It seems to me that your biggest risk in using a ConcurrentDictionary is, unsurprisingly, bookkeeping/correctness. That comes with the territory, especially if you're shifting back and forth between distributed and centralized (main thread) logic.
As long as you carefully compare your usage/assumptions against what the CD is doing, the change should yield a solid scalability win over your HashSet implementation. Considering the additional complexity, though, if you haven't yet verified that the HashSet is a significant bottleneck, you may want to do that first.