200ms of delay (between input and response) can be considered to be the point at which things get annoying. Sensitivity to the issue varies from person-to-person, of course, so it's not really possible to give a concrete number.
A delay of 50ms is well under that, and is probably not going to be that noticeable. It may be a problem if your event dispatch is tied to frame rate and you have sustained dips below 60, or if anything else occurs to change the delay from your theoretical worst-case noted in your question.
In general, user testing can give you a more complete picture of whether or not the effect of the latency matters. I suspect you'll be fine, however, unless you're making a game for which reaction time is very important (fighting games, rhythm games).