one game being more fun/engaging than another.
More fun/engaging, for which players? It's tricky, because different people rank game-fun differently. And how do you contact them? How do you know how much fun it was to them? This is all too hard! So instead, we consider just one player, who we have great access to: you
(Alternative: some writers, to have a target audience in mind, think of a single individual, and what would be entertaining to them - several children's books have been written with some specific child in mind.)
Now, how domwe tell which game you find the most fun/engaging, when it doesn't exist?
The first thing is to ask yourself: which do you think you'd find the most fun/engaging?
The next approach is to perforrm an experiment to help you tell. The purpose of an experiment is to not be a result in itself, but to give you information. The lowest-cost experiment, that gives you the best information, is the way to go. We can consider what is really easy to try (low cost), and the start again, considering what will answer your question (high information).
This is the MVP concept others have mentioned, but more extreme. If you can try out part of an idea, without coding at all, that's a better experiment. For example, can it be done with pen and paper? A board game? A card game to represent the choices? A map drawn on paper? You can combine your imagination with the prop, as part of your prototype. There's no difficulty with getting the answer out of someone, or if they are imagini g it wrong, because it's you. Plus, imagination is even lower-cost than pen and paper!
Some non-game examples: a guy wanted to test the idea of selling cars online. So he made a webpage that emailed the form to him, and did everything else manually. He wasn't trying to build a business, but get informatiin aboit whether people would actually buy cars online.
The Wright Brothers realized control of an aircraft was the tricky bit, but it took 6 months to build a test aircraft (and risk life and limb to try it). So they built a wind-tunnel (perhaps the first?), and could try out ideas on models, sometimes several a day.
Note that the task here is not to make a prototype of your game, but to see if a game idea is fun, how much fun and which one is the most fun. It's about information, not to build your game. It should feel easy to throw out a failed experiment.
- A problem with this approach is that a game idea might not be that much fun in the first version. This is a problem for you question only if it might change the fun-ranking: if one game idea is the least fun... but with some tweaks, becomes the most fun. You might need to change the idea, maybe a lot, to get the fun out of it. The experiment/prototyping aproach here can also be adapted to that - it's easier to tweak an idea in your mind, or on paper, than in code.
But if you consider all these variations, there are too many game ideas to compare. So, you shouldn't do this until you've choosen an idea (or, if you have a brilliant idez to make it better, and you can't help yourself but explore it).
Finally... through this process, as yo learn more about your game ideas, you may get a sense that this is the one to do! This would be awesome fun! That game is the one to do.