To supplement the excellent answers this question has already gotten, I thought it would be helpful to offer one of the most straightforward ways of comparing hands once the basic classification technique in place. First of all, you'll want to tag hands with their class, as numerous answers have suggested - most of your comparisons of 'is hand X better than hand Y?' can then be done just by comparing the two hands' classes and seeing which class is better. For the rest, you'll actually need to compare on a card-by-card basis, and it turns out that a little bit more work in classification will make this easier.
As the baseline case, consider the situation where both hands are 'high card' hands; in this case, you'd compare the two highest cards first, then (if they matched) the next two cards, etc. If you assume that each input hand is sorted from highest to lowest card, this approach leads to code that looks like this:
int CompareHandsOfSameClass(Hand h1, Hand h2) {
for ( int i = 0; i < 5; i++ ) {
if ( h1[i].rank > h2[i].rank ) {
return -1;
} else if ( h1[i].rank < h2[i].rank ) {
return 1;
}
}
return 0;
}
Now, the good news: it turns out that this lexicographical ordering, suitably tweaked, works for comparing two hands in any of the classes, as long as their class is the same. For instance, since the way of comparing pairs is to compare the pairs first, then the other three cards, you can sort your hand to put the pair first (or even one card of the pair first!) and run this same comparison. (So, for instance, a hand like A9772 would be stored as either 77A92 or, better yet, 7A927; the hand A9972 would be stored as 9A729, and comparing with the above code you'd start by pitting 7 against 9 and find that A9972 won). A hand of two pair would be stored with the higher of the two pairs first, then the lower, then the 'kicker' (so, e.g., A9977 would store as 97A97); three of a kind would be stored with one card of the three first, then the kickers, then the other cards (e.g., A7772 would be 7A277); a full house would be stored with one of its three and then one of its two (e.g., 99777 would be stored as 79779); and straights and flushes can both be stored in 'direct lexicographical' order since they're both compared just like high-card hands are. This leads to a straightforward outer comparator function that works for all classes of hands with the already-given function:
// Compare two hands, returning -1/0/+1 as hand 1 is less than, equal to,
// or greater than hand 2. Note that this function assumes the hands have
// already been classified and sorted!
int CompareHands(Hand h1, Hand h2) {
if ( h1.handClass > h2.handClass ) {
return -1;
} else if ( h1.handClass < h2.handClass ) {
return 1;
} else {
return CompareHandsOfSameClass(h1, h2);
}
}
Hopefully this will be of some help!