First you should have at minimum a rough idea of what your game will be and what it will need. There are the usual questions like whether or not your game will need physics. Then there are the interrelated questions, which are mostly about the environment your game will be developed and run on and how commercial your game is or isn't.
- What platforms will you support? What platform are you developing your game on?
- What programming language(s) do you intend to implement your game in?
- This is influenced by the platforms you intend to support.
- What kind of graphics, audio, and input will your game have?
- Is your game freeware or open source, or is it commercial?
- If open source, will you want to go commercial later, either with your game or with another game that uses your first game's engine?
If you're going to be Windows-only then DirectX and C#-based libraries are strong candidates. If you want to be multiplatform then you'd want to look at libraries based on OpenGL and C/C++, or Flash if your game is 2D and you can afford Adobe's tools. Like your platform, your programming language will influence your available libraries. A program written in C++ will have a hard time calling a Java library.
If your game is commercial then you can consider buying an engine like Unity. If your game is freeware or open source you will want to focus on libaries that are also open source, or at least free for non-commercial projects. Open source libraries are useful to commercial projects too of course. When looking at open source libraries make sure to check their license. Some licences require you to make parts of your own software open source depending on how you use a library. Restricting yourself to open source libraries will obviously place another limit on what libraries you can use.
When looking at a library, make sure to check how active its development and community is. I'd trust a library that is well-known and actively maintained more than one that is hosted on someone's university web page that hasn't been updated since 1999.
One final question, especially if this isn't your first game project, is whether or not there aspects your game will need that you know you'll struggle with if you try to do it yourself. Because these aspects in particular are candidates for finding a library. If your game needs collision detection and you know you can't implement collision detection yourself (like I can't), consider picking up a physics engine and leverage the collision detection features in it.