Are your test player having trouble discovering the controls or using them?
If they have trouble discovering them, you can add a prompt that explains them. To make it unobtrusive for players that don't need it, you can have it appear if the player stays there doing nothing.
Which, brings me to the next thing: probably all your game needs is a practice/creative/free mode, where players feel free to experiment with the controls without any pressure. That will get them used to the controls in the case that the trouble is using them and not discovering them.
While a game can do well without instructions or tutorials, they could help you reach more people (because not everybody is keen on experimenting without an example or guidance).
Usually I would say that you should let the player configure the controls. However, I recognize that that does not make sense for every game and input device. With that said, consider if you need sensitivity options.
There is also an argument for allowing for a simpler control scheme which will allow players who struggle to beat the game (a kind of assist mode, you can let the player know that it is not the intended way to play, yet it can allow them to enjoy the game anyway if they are having trouble). And this can be implemented without a lose of depth if the full controls are required to get high-scores.
If you are going for a tutorial, the player does not need to notice it. Examples such as NES Mario (which Philipp explains, you might also be interested in Super Mario 3D World's design philosophy) and Megaman X (which is f■■king genius) are great case studies. A more recent example is Portal, which is tutorial for the most part.
See How to Design a Good Game Tutorial.
On the other hand, if you are thinking about tweaking the difficulty, consider that "Hard" does not have to mean (although it often means):
- Punishing (you fail, you lose a lot of things)
- Tedious (it takes a lot of time to get things)
- Inaccessible (hard to understand or control)
Instead, "Hard" can mean challenging. That way, Hard can be fun. Separate challenging from difficult, and you can increase challenge by adding rewards that require more skill, without making completing the game harder.
In fact, who says difficulty is a single dimension concept?
Thus, if you are increasing accessibility, you are not necessarily making an "easy mode". The game can remain hard (challenging). I want you to consider the concept of an assist mode.
To be honest, it is a bit of a branding issue. You see, some games lie to you (have hidden mechanics) as a mean to tweak difficulty. Usually to make it less punishing, however sometimes to make the content last longer (more tedious).
Difficulty in games has lowered in some parts of the industry, and accessibility is blamed for it (even though it is not the only factor). Also, some people want challenge※ and some people want to relax※※... there is no perfect difficulty.
※: Some games keep you engaged the whole way, they often throw unexpected twists to the player, have your adrenaline rush, your heart pumping, and huge dopamine rewards. Yeah, too easy might not be good.
※※: Some games allow you to get in the zone, archive a semi-meditative state as your mind is freed while you body just does the motions - rhythm games are good at this, also crowd-combat fighting games (no, repetition does not have to ruin the game)... Other are fun because they act more like sandboxes, they are more toys than games, where the fun comes from finding creative and amusing ways to do things. Yeah, too hard does not help either of that.
And then there are games that mock the player for picking an easy mode, failing or quitting.