I will go to college very soon and I want to choose math major. I want to know if being graduate with a strong mathematical background will help me success in game design.
For example if I want to be a numerical planner to decide the value of attack/defense/critical hits, does it need a strong mathematical background, the ability to use a really complicated mathematical formula?
There's more than one way to be a game designer. So the best answer to your question is tautological: if you want to be the kind of designer who solves problems with math, then knowing some math is helpful.
I'm a fairly math-y game designer myself. I'm in the Analyst category, using the taxonomy of designers Richard Carrillo presents in the link above. When I encounter a design question, my first impulse is "how can I model that with an equation or spreadsheet, to understand the parameter space and make numerical predictions?" Ian Schreiber might be another good example of this kind of designer.
This approach has led to me doing a lot of economy and progression balancing work, because that's an area of game design where those math and spreadsheet tactics are especially fruitful. I'll also use my knowledge of geometry and math functions to help solve 3D problems, like making objects move through space the way I want when designing character controllers or AI behaviours, or sculpting procedurally generated content or shader visuals. (Though those last few activities are less commonly the responsibility of a game designer, it can still help when collaborating on a small team, or when making solo projects)
But even for all that, I don't use that much math, compared to a full major. Scan through my answers here and you probably won't find much that involves more than first-year calculus and linear algebra, stats and probability formulas you can pull off of Wikipedia, or the occasional differential equation that I just ask Wolfram Alpha to solve for me. 😉 This is all stuff you could learn on your own, without a full degree in math. Knowing that there's a math-y way to solve something, and having a hunch for the right keywords to search for it, is enough, even if you don't have the full theory memorized and at the tip of your tongue at all times.
Meanwhile, many game designers do their work without complicated math formulas. The designer I worked with who laid the foundations of the economy and progression systems in Splinter Cell Blacklist was definitely not a math-y person, and went into games straight out of high school, and continues to do excellent game design work to this day with nary a formula in sight. I'd say less than a fifth of the game designers I've worked with so far lean heavily on math in their process. Because there are lots of other ways to do good game design:
Develop experience: by playing and making lots of games, you'll develop a mental repository of designs that worked, and you can draw from that rich background to remix and combine solutions to novel problems.
Use iteration and playtesting: guess and check your work by feel or by observing and experimenting with other players. This approach can help capture a lot of the complexities of the experience that a formula might simplify out, though it can of course be time consuming!
Understand player psychology: knowledge of how human minds work, what different types of players seek, and empathy to see the player perspective can guide you in making decisions that are right by them.
Listen to instinct: with enough experience, you develop a "gut feel" for what choices are likely to work, and that itself is a skill you can train.
Leverage experts: when you encounter a problem that needs a skill like math that's outside your expertise, find someone who can help and work with them to find a solution. That could be working within your team, or finding and engaging in online resources, as you're doing by asking here. Good communication and search skills are a kind of skeleton key for knowledge - use them right and any expertise on the planet can be yours to wield!
There are also limitations to the mathematical approach. It can help you optimize your formulas and design parameters for a given goal, but it won't tell you what goal you should be seeking in the first place. This is a common problem we see with balance questions on this site: folks looking for a formula that gives the "right" numbers to make their game fun, but the formula isn't the hard part, it's figuring out what's "fun" - and unfortunately there's no mathematical expression for that. Questions involving human behaviour and perception are also notoriously difficult to formalize. We can use a spreadsheet model to show what the optimizing / min-maxing player would do, but not what players who haven't done the math / read the strategy guide would be drawn to, or how they'd feel about the experience at each step. We need other design skills there.
But, once you do identify the right goals via other means, math can be a useful accelerator in finding the right systems and parameters to meet those goals, and for making evidence-based arguments that they're the right choices, which can be useful if you need to win over stakeholders in a larger studio.
So, we can't tell you what to study. Just about anything you learn can be helpful to the design of some games. What kind of game creator you want to be and how you'll apply your knowledge, experience, and skills to craft your games is something you'll have to figure out for yourself.
