I think the wikipedia article on casual games answers the question pretty well as to what casual games are.
It should be noted that people who describe themselves as "casual gamers" don't necessarily play casual games. "Casual" can describe their dedication to games or the amount of time spent playing them. The games they play could easily be the same games someone who describes themselves as a "hardcore" gamer plays. In this sense, they use the term "casual" to tell you it's not something they're really serious about.
However, it's likely that statistics relating to gamers may have a category of "casual gamers", that would identify a group of people who play casual games.
Just be aware of the context these terms are used in and they should be easy enough to distinguish.