Note that I'm not sure how objective this answer can be. I feel it's objective enough for SE, but then few people think their opinion on a subjective matter is anything less than absolutely correct. :)
Note also that the terms "component-based design" and "entity system" are not clearly defined in the industry, so it's quite possible that different people have different ideas of what these terms mean.
The second system you mention is called an "entity system" these days. The idea is that you only have components, and they all share some kind of of shared identifier to link them together. Even systems that use a "container object" often work this way, with the object simply being a distinct concept from a component and allows some extra features that the components do not.
Some people also use the entity system moniker to differentiate between between "logic components" and "data components". The difference there is whether your component types have any methods that implement logic or if they just contain data.
Personally, I find the idea of an "object-free" entity system to be completely broken, and to my knowledge it has never been used for a "real" game (certainly nothing large and complex). There are all kinds of pieces of data that are not related to any particular component but rather are shared by all components (i.e. they're properties of the game object itself).
However, I also find strong advantages to preferring data components over logic components. Data components allow a lot more flexibility, better performance, etc. For instance, your PhysicsComponent should not be responsible for moving the object. There should be a PhysicsSystem that has an efficient collection of all PhysicsComponents and which updates all of them together in one pass. This is more efficient, reduces errors in the simulation, etc. The same goes for graphics (you absolutely DO NOT want to have each object drawing itself, that breaks a lot of opportunities for batching and instancing and is horrifically inefficient on modern hardware!) and so on. Your components should contain the data the systems need, and otherwise just serve as a way to register the object with the appropriate systems.
That said, again there are problems with an academic puritanical approach. There always, always are. If you outright ban logic components, you're going to hate life when it comes time to implement actual game logic. Game logic is hairy, dirty, and hardly ever able to be fit into a clean universal architecture. You're going to want to have a game object that behaves in some special way when some event happens. Which is easier: hardcoding all the possible behaviors into a BehaviorSystem and selecting them based on some property of the object, or just attaching a component to the objects that need a particular behavior? Even most entity systems allow behavior components, they just lie and call them something else; like a script attached via a ScriptComponent, in which the script is really just another component (but without all the support and integration that real components could potentially get, and without the ability to attach multiple independent scripts easily).
There are hence some good ideas in the entity component systems being bandied about, but you're best off taking those ideas and applying them to a foundation you're more comfortable with. The same goes for inheritance: component based design says you should avoid inheritance with your objects and favor aggregation, but no experienced developer worth working with is going to tell you that it's sensible to outright ban inheritance in a system. There are just far too many places that inheritance makes things easier, cleaner, and faster. The trick is to use it when it makes sense and to use something else when it does not make sense. Learning those situations comes with experience, not reading blogs or forums or StackExchange answers, and your best bet is to just program, program, and program some more. :)
Some reasons you might want to consider a "traditional" component based OOP design:
If you want to know if a particular object id exists, you don't want to have to search all component pools for that id.
If you want to make an object with delayed deletion (you always want delayed deletion) you don't want to have to go and mark every component (and then open up a race condition where some system might try to add a new component to a "deleted" object which then would not receive that deletion flag).
If you want to register your object with a live game editor, including metadata about creation time, name, etc., you need a place to store all that. This could just be an extra component, but then you need to have special code to always enforce that this component is created for every game object, and to ensure it is never deleted unless the whole object is. Much easier to just have this be part of an explicit game object instance.
You indeed may want a collection of all components for a single object stored somewhere for various reasons, and a real game object instance is an ideal place for that.
If you plan on having high-performance networking, you're going to need a centralized system for communicating changes between components, linking updates (you're likely going to have a single highly-compressed "packet" type to update position and velocity, despite those being properties of two different components), and so on. While you will most certainly have a Networking component, if you don't have a game object class you're just going to end up duplicating all that functionality into the Networking component anyway.
Message broadcasting, while not necessarily the best idea in the first place, works a lot better with a central object to manage where to broadcast messages to. Otherwise you are again stuck trying to deliver the message to every possible component pool for a particular object, even though maybe only 5% of the components are used by that particular game object.
Sometimes you have behavior that works in a composable manner, and it's just way easier to use a component system to attach those logic objects to a game object, rather than trying to shoehorn everything into a separate system.
Some reasons you might consider a pure entity system design:
Works better with databases and distributed systems where centralization is a huge bottleneck. Likewise it also can help with some threaded engine models (though not the correct models, in my experience) since it removes contention on a central object.
Simplifies decisions of when and where to put logic vs data, and ensures that your design is clean and understandable. Traditional systems allow an equal amount of cleanliness but requires discipline to avoid piling on hacks and inappropriate object coupling.
More obvious on how to implement in a language that does not natively support traditional OOP, such as JavaScript, while still being easy to implement in native OO languages like Java, C#, or C++. Fits better with a purely functional design as well as with purely procedural designs.
Enforces a data-driven approach to object design, which is generally a very good thing.
Obviously I'm a bit biased towards component-based design rather than entity systems, but that bias comes from actually having built fairly complex component-based games and engines and having solved problems that I can't solve as easily or naturally in a pure entity system.
tl;dr -- Entity systems take many existing good practices from component-based design and throw away everything else (good or bad) and replace them with a purely data-driven approach to game object design. Understand the problems entity systems try to solve and apply those lessons to traditional OO component-based design. Real-life hybrid systems that apply different solutions to different problems are usually superior to academic systems that attempt to use a single design pattern to solve every problem.
