The whole process involved in what you want to do is too broad for the context of a StackExchange question, so I'll cover broadly what you're looking for.
As it's been said in the comments, collision detection is not done on the GPU. The GPU is not aware of the concept of physics, it cares only about the graphics (¬), so you'll want to process the physics on the CPU and then send the results to the GPU (through a matrix) so that it shows correctly the world state. Alternatively, the physics engine does not care about the GPU or what the users sees (and how they see it).
To do that, you need a part of your code to only care about physics. Let's say you have two primitives:
You need a way to define those items; for a cube it's its length, and for a sphere, it's the radius. You can assume that those are centered at the origin, so the vertices of the cube, in local frame are situated at ±l/2 where l is the length of the cube.
Then you'll need to know how those are positioned in the world, so you add to each of those a transform matrix used to transform the object from local frame to world frame; it is essentially containing the position and the orientation of the object. This is what you'll end up sending to the GPU each frame.
The primitive length/radius and the transform matrix give you all the information you need to know where exactly is the object.
You'll use this information to determine if objects collide. You'll need some functions to tell you if two objects collide (I suggest you look up those using your favourite search engine, it's been implemented many times):
Each frame, you check each pairs of objects using those functions, and if they collide, you then handle the collision in the way you see fit (sometimes, those function will tell you the point(s) of contact and how much they penetrate, along with the axis of penetration, so you can push them out if you want them to only "touch").
¤
That's it. Once you have handled the collisions, the objects are placed at their final location this frame using their transform matrix, so the next physics step is ready to go. It's now time to make your user aware of it: you find a way to send the transform matrix (that's your "model" matrix) to the GPU (AFAIK, you need to set a uniform before drawing your primitive's VAO).
¬ I'm pretty sure some physics can be done on the GPU in some context, but generally speaking in game development, you'll want some parts of your game logic aware of the results of the physics steps (e.g. a bullet has hit an enemy, so you'll want to remove some hit points on the enemy entity), and going from the GPU back to the CPU (where your game logic executes) is not trivial/optimal.
¤ Often, physics engines will have "two phases" when checking for collisions
- the broad phase, where they roughly check if the two object may be colliding (e.g. using AABB)
- the narrow phase, if the broad phase concluded that they may be colliding, the narrow phase will actually check more precisely if the two objects collide, and how they collide.
