There is no technical solution to this. It is a matter of the dialog recording director and editing, and it is hard work. If you've ridden modern trains in Europe, you will find that they do this (with varying levels of success) for the announcements at stations and in trains.
The short answer is: You have to get the voice actor to read all numbers at the same pitch, ideally during the same recording session, and at the right pitch to fit into the flow of the sentence.
You then have to edit all recordings so they have the same amount of silence at the start and end, so that when plugged into the sentence there are no unnatural pauses around the insertion.
Given the people tend to bind words together ( e.g. You'd likely say "You hafive hundred points" and not actually fully enunciate the end of "have" and then pause), it would likely still sound a bit unnatural.
If you also take into account how much more work it is for a voice actor to repeatedly hit the same pitch (and energy level) and how much more likely they will have to re-do the line, and how much time it takes to edit the recorded bits together so they sound good, plus develop engine support for loading several phrase fragments and playing them back gaplessly, and taking into account how cheap disk space and bandwidth are these days, and how well voice audio compresses, it usually makes more sense to record the entire sentence repeatedly than try to cut it together from multiple parts.
This also means the voice actor can play with the voice lines a bit, e.g. Do special versions for higher numbers, and it will sound more human if the sentence melody, speed etc. vary a bit as the numbers rise.
In fact, many AAA games include several versions of the same phrase, or even slightly different phrasing to provide more variety.