I would recommend having both.
This defines which entities are attached to which components. In my engine, there is no such thing as an "entity", but any component can have child components. This graph is not used to draw or update components, but is only used for logical queries and events that are supposed to effect component sub-trees in the graph. The graph is just made of weak references to components (in C++, pointers. In other languages, unique identifiers), and doesn't contain the components itself.
For example, I have an "onDeath" event that uses the component graph to tell which components need to be destroyed when a parent component has died.
Component Lists and Systems
Parallel to the component graph is a set of component lists. For instance, there is a list of all the
Physics components in the game, a list of all the
Sprite components, and so on. These lists are aggregated into a big map of component type to component list called the
ComponentManager is responsible for updating and drawing all components, one component type at a time. This is far faster than iterating through the component graph, because it has much better cache performance, and allows you to batch state of components before drawing/updating them.