There have been some really interesting breakthroughs in mainstream UI toolkits over the past few years. Game UIs have been evolving too, but at a somewhat slower pace. This is likely because developing a full-featured UI subsystem is a significant undertaking in itself, and it's difficult to justify the resource investment.
My personal favorite UI system is the Windows Presentation Foundation (WPF). It really takes the potential for UI differentiation to the next level. Here are some of its more interesting features:
Controls are "look-less" by default. A control's logic and visual appearance are generally decoupled. Every control ships with a default style and template, which describes its default visual appearance. For instance, a button may be represented as a rounded rectangle with an outer stroke, a solid color or gradient background, and a content presenter (to present the caption). Developers (or designers) can create custom styles templates which change the appearance and (to an extent) the behavior of a control. Styles can be used to override simple properties of a control, like foreground/background color, template, etc.; templates change the actual visual structure.
Styles and templates may include "Triggers". Triggers can listen for certain events or property values (or combinations of values) and conditionally alter aspects of a style or template. For instance, a button template would generally have a trigger on the "IsMouseOver" property for the purposes of changing the background color when the mouse hovers over the button.
There are a few different "levels" of UI elements ranging from lighter weight elements with minimal functionality to full-blown heavy-weight controls. This gives you some flexibility in how you structure your UI. The simplest of the UI element classes,
UIElement, has no style or template and only supports the simplest of input events. For simple elements like status indicators, this functionality might be all you need. If you require more extensive input support, you can derive from
FrameworkElement (which extends
UIElement). Traditional widgets generally derive from
Control, which extends
FrameworkElement and adds custom style and template support (among other things).
WPF uses a retained, scalable, resolution-independent vector graphics model. On Windows, WPF applications are rendered and composed using Direct3D.
The introduction of WPF included a new declarative programming language called Xaml. Xaml, based on Xml, is the preferred language for declaring UI "scenes". It's actually pretty slick, and though it may appear similar at first glance, it's fundamentally different from existing languages like XUL (and much more powerful).
WPF has a very advanced text subsystem. It supports most if not all OpenType font features, bidirectional ClearType antialiasing, subpixel positioning, etc.
"Ok, great, now how is this relevant to game development?"
Back when WPF was still in development (and known as "Avalon"), I was taking a video game design course at Georgia Tech. My final project was a turn-based strategy game, and I wanted a very capable UI. WPF/Avalon seemed like a good route to go because it had a complete set of full-featured controls and it game me the capability to completely change the look and feel of those controls. The result was a game with a beautiful and crisp UI with the level of functionality that you normally only see in full-fledged applications.
Now, the problem with using WPF for games is that it's fairly heavy-weight and renders in its own "sandbox". By that, I mean there is no supported way of hosting WPF within a Direct3D or OpenGL game environment. It is possible to host Direct3D content within a WPF application, but you will be limited by the framerate of the WPF application, which is generally lower than you would want. For some types of games, like 2D strategy games (screenshot of my current WPF game project), it could still work great. For anything else, not so much. But you could still apply some of WPF's more compelling architectural concepts in developing your own UI framework.
There is also an open-source, unmanaged C++/Direct3D implementation of WPF called "WPF/G (WPF for Games)"](http://wpfg.codeplex.com/) that is specifically designed for use in games on both Win32 and Windows Mobile. Active development appears to have ceased, but the last time I checked it out, it was a rather complete implementation. The text subsystem was the one area that was really lacking compared to Microsoft's WPF implementation, but that's less of an issue for games. I would imagine that integrating WPF/G with a Direct3D-based game engine would be relatively straightforward. Going this route could provide you with an extremely capable, resolution-independent UI framework with extensive support for UI customization.
Just to give you an idea of what a WPF-based game UI might look like, here's a screenshot from my pet project:
Screenshot from my ongoing WPF-based game project http://www.startreksupremacy.com/supremacy_wpf.png