If you make the graphics for the average sized target device, the bigger screens will clearly notice the stretching, distortion, aliasing, etc. People will know the game wasn't made for their device unless its running pixel art, which in many cases still feels good in any resolution.
Then, when stretching down the assets, you usually get less issues and less quality loss as when increasing, so this should be the optimal way to go.
Since you want only one set of graphics (to avoid huge .APK files) here's my recommendation. Use texture atlases for everything you can, and fit as much as possible in the same atlas to save space. The graphics you pack into those atlases, make sure they look good in the bigger screensizes, even target 720p or 1080p if you so desire (remember tablets). With all graphics made for big screens and packed tightly as possible, find the format that best suits size with lossless compression. I usually use .PNG. Try with different image editors and find the one that will write the most optimized PNG file for optimal size.
In android, a good size for texture atlases is 2048x2048, according to my research. It can fit images that cover a full HD screen(1080p), its a power of two size, it can fit many ui elements and graphics together in the same texture (this also improves rendering performance a lot). If you don't need this much size, just use something below it. If you think about using textures bigger than this size, please don't. Many devices won't support it and you will waste people's time installing your application. Even 2048x2048 is not fully safe to use on older hardware, but many low-end android devices support it these days.
Now that you ensured the best quality/filesize ratio that covers all screens, there is the aspect ratio issues. Try to avoid black bars if possible, as many people don't like those, especially when their screen is already small and they are covering it a bit with the touches. Whats the most adequate is usually dynamic positioned UI that fits a given pattern in the screen, working for all resolutions more or less appropriately. As for the gameplay camera, what I usually do is to choose a fixed size for either width or height, then set the other dimension to something relative to ensure aspect ratio on all screens (prevents graphics stretching non-uniformly completely).
Having a game with a nice aspect ratio, placement of UI controls and assets oriented for all screens, if you still aren't satisfied with how the assets look stretched down to lower screens, you could activate smoothing on all textures, also known as texture filtering, syntax may vary in different API's and I never used libgdx. If you still aren't satisfied from the relatively poor results with the previous solution, then you should look into some image-based antialiasing approaches or perhaps some blur to hide detail and therefore hide stretching problems, if it applies to your asset style.
Hope it helps. Sorry for the lengthy answer.