In general, as a matter of practicality, you might want to find a difficulty level that suits your and your testers' skills and balance around that to begin with. What name you end up assigning to that difficulty in the end is secondary and may very well change as designs evolve and emergent gameplay becomes more apparent.
However, where to start and whether it's easier to scale up or down also depends on the kind of game and how you implement difficulty levels.
Many shooters and RPGs simply assign modifiers to certain numbers (e.g. health points, damage). This is easy to implement, and easy to change later on, but often affects the volatility/randomness of challenges (e.g. random critical hits insta-killing the player on "hard") as well as their duration. With this method, I'd advise to find a good middle ground first (smooth, interesting gameplay) and go from there, in both directions. Start on "medium".
Another approach would be to keep the numbers the same, but add or remove obstacles and assistance depending on difficulty. For example, levels for a puzzle game might be easier to design around a fairly simple premise, with complications added in harder difficulties. Start on "easy".
A game focused around timing (e.g. racing game) or resource conservation (e.g. "get from A to B without running out of fuel") might be calculated around the worst case (highest difficulty) scenario where a player playing perfectly might just about make it, with additional pickups or more lenient parameters added in easier difficulties. Designing around the maximum possible score is especially useful in "score attack" games. Start on "hard".