I'm assuming this is a game balance question and not a coding question, and there are a lot of ways to handle this. From your existing algorithm, I suspect you are making it more complicated than it has to be (three attributes AND a random number make for an extremely muddy system from the player's perspective). My first instinct is to simplify!
Favor attributes that do multiple things, rather than results that are determined by multiple attributes. DEX works just fine as the primary determiner of to-hit chance; the addition of everything else just serves to make it harder to balance. Think of it this way: a player's expected damage on each hit is their to-hit % multiplied by their average damage when they do hit. How do you plan to calculate an "expected" or "typical" to-hit % with so many variables?
I see another issue, which is that your formula doesn't scale, and this is probably what you're experiencing. Assuming DEX, LUCK and SPEED tend to increase over the course of the game, differences between player and monster in those stats will also tend to increase. Just as an example, if a player's early-game DEX is in the 4 to 6 range and so are monsters, that's a 10-20% to-hit modifier. If the range is more like 10 to 20 in later stages of the game, that's up to a 100% modifier.
One way to fix that is to keep the stats fixed. Player and monster ranges are the same throughout the game, and getting something like +1 DEX is a huge deal, extremely rare, and treated as such (if it exists at all). Then your formula will work fine for the entire game, and you'll see a player with about the same to-hit chance throughout.
Another way is to change that hardcoded "10" to scale with level, so maybe it's "10 + monster level" or "10 + dungeon level" or something, which causes a slight increase in difficulty as the game progresses unless the player levels up their stats accordingly.
A third way, as others have said, is to set hard caps on the highest and lowest acceptable to-hit %, and simply say "if the calculated value is less than X, use X instead".
For the sake of completeness, I'll point out that there's no absolute law that you MUST include a to-hit roll in the first place. Taking an action in a game that's ineffective is not particularly fun, and doesn't really add to the experience except in rare cases. An alternative is to let players ALWAYS hit, and just mess with the damage range so that sometimes they'll do a lot less damage than others. Remember that complex systems are more fun for the designer than the player.