Endianness matters when it comes to game consoles. The Wii, the PS3, and the XBox 360 all run big-endian, while all major desktop computers (as of the date I'm writing this answer) run little-endian. If there's a chance you'll want to compile your code for one of those game consoles someday, or if someone releases another popular big-endian desktop machine someday -- it wasn't so many years ago that all Macintoshes were big-endian, after all -- then you need to think about endianness. And if you don't think about it now, then from personal experience I can tell you that it'll be extremely painful when you have to add big-endian support to a whole completed codebase which was written without it.
Similarly, endianness can matter when you're communicating with internet servers. If you want cross-platform server software, then all your network code should really be transmitting/receiving using the standard network byte ordering: big-endian.
A related issue which you don't mention is that of struct padding. A single struct or class can have different sizes on different platforms or even just between different compilers, and its fields can exist at different offsets within the struct. Put simply: Even if you decide not to care about endianness, you still can't simply treat a struct or class as a binary buffer and naively write it to a file or a network packet. In order to transmit a struct of data from one machine to another, you need to individually read or write each field, rather than treating the struct as a single blob of memory. And honestly, if you're doing that (because you have to in order to support cross-platform), then you might as well convert the values to/from network byte ordering at the same time.