It depends of the event
Feeling that, say, an explosion you see and hear is a single event will have the tolerances described in other answers - no more than ~50ms; some people may be more sensitive (e.g., amateur musicians), so I'd suggest to aim at 30ms or no more than 2 frames at 60fps.
I believe that the perceived distance should affect those tolerances. People expect far sounds to be slightly delayed, since in real life sound lags sight by approximately 1ms per each foot of distance. So an explosion on an zoomed out RTS game 'map' might have a larger tolerance for sound lag than the player firing their own gun in an FPS.
Specialized cases, such as having a proper feel for a music/rythm game may require much tighter tolerances, 15-20ms or even lower - for example, if the player hears both the "input action" such as singing into a mic or banging a plastic instrument, and also a sound generated by your system for the same event, then a 50ms lag will cause the "original" and "played" sounds to mix weirdly.
In addition, keep in mind the lag between start of the audio file and the "event" inside that audio file - in many audio clips, the "event" won't be right on the edge, you may have a sound of a lightning strike where the 'strike' happens 200ms after the beginning, which would be obvious to everyone, and pretty much all sound files, even a drum-hit, will have some delay there.
Don't measure averages - look at worst case
Sight&hearing are deeply connected in human perception, and if one of them stutters relatively to other, then it will be perceivable. It's not okay if most of the time it's very fast but occasionally there's a 0.2 second delay while something is loading - people will notice such situations. This is why audio is often kept running on a separate thread, isolated from the other activities and just getting rapid notifications on what preloaded clips should be played.