I'm trying to think of sounds that will help convey the time of day and weather condition. I'm not even sure of all the weather conditions I would consider, and some are obvious. Like if it's raining, the sound of rain. But then I'm thinking, what about for a calm day? If it's morning time, I could do birds chirping or something. Night time could be an owl or something. What are some good combinations of sounds/weather/time to have a good effect?
The sound of rain, provided it is not too loud, is a good idea, as a rain visual without a rain sound is disconcerting. Occasional playing of the sound of thunder can really help, but you would probably need to support some kind of lightning effect for that to work.
Is your game set in the country or the city? This drastically alters the sound of a 'calm' day and a usual night. Considering what you mentioned, I'm going to assume country. The sound of an occasional wind gust or a wolf howl for such a situation is not out of the question. Whatever enhances the overall feeling of being in the environment. That is quite possibly the biggest problem with music and sound effects - they are at their best when you don't even notice them for what they are; their presence is just part of the scene and they don't stand out.
Whatever your choice, make sure to sufficiently space the sound effect between plays and provide a pretty decent number of sounds that are played. Hearing the same hooting owl every three seconds at night could get really annoying.
How about using background music instead of sound effects?
Animal Crossing is a real-time game (meaning the game keeps track of the time), and it had a different background tune for each hour of the day; the tune in the middle of the day is upbeat, "friendly" sounding, and the night one is very quiet and slow with some extra percussion sounds thrown in. I don't recall any bird chirping sounds, though there is the sound of water running when you walk near a stream and the sound of your own footsteps when you walk around (specific to the material being walked on), and there definitely was the pitter-patter of rain when it was raining.