What you are attempting is non trivial and using the internal Physics system is going to be a ton less problematic.
That said, if you really want to eek it out by hand then you need to first detect intersections between the relevant balls, then you need to resolve these out, then you need to update the velocities.
- To detect intersections between the balls
This is simply a sphere / sphere intersection test. Iterate through all pairs and check the distance. Distance = (sphere1.transform.position - sphere2.transform.position).Length()
If Length < sphere1.radius + sphere2.radius then you have an intersection and you should resolve them
- Resolution of intersection
You first calculate the normal of collision. This is simply
NormalOfCollision = sphere1.transform.position - sphere2.transform.position;
NormalOfCollision.Normalise();
You calculated the sphere distance in (1) so the resolutionDistance is totalRadius - sphereDistance. You then apply this to the bodies.
sphere1.transform.position += NormalOfCollision * resolutionDistance
sphere2.transform.position -= NormalOfCollision * resolutionDistance
If you are going to have spheres of different masses colliding then you should scale the impulse so that the smaller mass is moved more than the larger mass. I.e.
sphere1Impulse = sphere2Mass / (Sphere1Mass + Sphere2Mass);
sphere2Impulse = sphere1Mass / (sphere1Mass + Sphere2Mass);
sphere1.transform.position += NormalOfCollision * resolutionDistance * sphere1Impulse
sphere2.transform.position -= NormalOfCollision * resolutionDistance * sphere2Impulse
- Calculation of velocity after impact
You need to use the conservation of linear momentumm law. The wikipedia page gives a worked example.
NB
- The above is pseudo code, I've not run it, so I may have some of the +ve / -ve signs around the wrong way - butit's the right direction
- This just handles the linear effects NOT angular (so won't simulate spin shots etc)
- If you have larger time steps, or multiple balls interpenertating you should resolve until you have no more contacts (ObjectA may push ObjectB into ObjectC so you need to resolveagain)
- There is no optimisation in this (islanding / broadphase collision etc)
- There is no restitution in this with different objects being more bouncy than others
- Using PhysX is going to be a ton easier than doing everything yourself