This is technically a few questions in one (that do go together, and organically flow from one another), and I will break it apart to answer them. I will be paraphrasing:
Is it reasonable to have a player pay a "fee" to perform actions in my game?
This is the whole concept behind micro-transactions (I will use real money here for something more tangible, but can be extended to imaginary currency/points/rep), and companies have made millions in terms of real money from it. there is still some discussion of what should be the limits of this, and sometimes the question of ethics comes up, but that aside. Yes you can require a fee for some actions in your game, but the following question needs to be observed.
how do I balance a monetary system in my game?
This can be done in terms of pure numbers, or in terms of psychology. pretty much it all comes down to what is the value of the currency with respect to effort put into the game, or physical currency being spent (I do not condone/like the latter, so I will not discuss it to much). If there is a item (that boost your characters stats to an extraordinary amount) in the game that can only be obtained by purchasing it from a specific vendor for lets say 2,000,000 (currency), and it requires the player approximately 1 minutes of actively trying to earn 50 (currency) then it will take said player 40,000 minutes to acquire enough currency to buy said item. putting that in perspective that is 27.8 days (without breaks). if the player does the same math they will do some risk/reward analysis, and might come to the conclusion that that amount of time is not worth said item.
the big take away is figuring out some kind of calculation of this risk/reward analysis the easier it is to earn currency the higher prices can be, and visa-versa, but some numbers no matter how easy the currency is to obtain the player might just think are to high even if it is obtainable.
if the player is going to be gambling in my game how should I approach setting the wager amounts?
this is actually the meat of your question (I know you didn't say gambling, but in essence you are "put down amountX for the chance to win amountY" is gambling)
this falls back in a circular fashion to the previous section where if the currency is easy to obtain then players will be more likely to wager more. though at the same time players will also see it like they would putting money down on a game of High card draw or Black-Jack. The higher the anti the higher the likely-hood of better players, and the statement "to rich for my blood" comes to mind.
for this reason I would advocate for a tiered wager system(tier1 costs x, tier2 cost Y, tier3 costs z... || x < y < z < ...) this will allow for your in game economy to feel more open, and also give players a sense that "if I go to a low wager match then I will see weaker opponents, and a high tier match will have strong opponents" this also gives the player the chance to try the system out by possibly entering at the lowest tier to get a feel for how the system works. then the "better" players will gravitate toward the higher tiers, though your more casual players will gravitate toward your lower tiers, and the middle tiers will only ever be used for the high tier player to recoup losses, and get back into the high tiers, or the low tier players to seek a challenge, and improve their skills.
In conclusion:
you can't really set hard and fast numbers for what is to much, and what is just right its the same as balancing how much damage a weapon does with respect to how much damage a shield can take, or how many points eliminating a given enemy type is worth. Though when you throw in perception it does make things interesting (low to one person can be high to another).
It will come down to how easy the player perceives earning currency in your system is. if they see that after every "story event" they earn 1,000,000 currency, but for most other "events" they earn 20 currency then you can put higher prices on things, but if there are only say 10 of those "story events" in the game, and the player hasn't seen one in a while they might tighten up the purse strings a little.
In short the key word is balancing.
aside:
If your going to have a wager system on a section of the game have a defined rule set that you state up front (winning/losing, scoring, and objectives) even separately (or mandatory the first time they enter the section, but still reachable later) otherwise a new player might feel cheated if they lose the first time, and if it is at the beginning of each match an experienced player might get bored having to sit through it every time.