Immediate GUI - yae or nay? [closed]

Question

I've been working on application development with a lot of "retained" GUI systems (below more about what I mean by that) like MFC, QT, Forms, SWING and several web-GUI frameworks some years ago. I always found the concepts of most GUI systems overly complicated and clumsy. The amount of callback events, listeners, data copies, something to string to something - conversions (and so on) were always a source of mistakes and headaches compared to other parts in the application. (Even with "proper" use of Data Bindings/Models).

Now I am writing computer games :). I worked with one GUI so far: Miyagi (not well-known, but basically the same Idea as all the other systems.)

It was horrible.

For real-time rendering environments like Games, I get the feeling that "retained" GUI systems are even more obsolete. User interfaces usually don't need to be auto-layouted or have resizable windows on-the-fly. Instead, they need to interact very efficiently with always-changing data (like 3d-positions of models in the world)

A couple of years ago, I stumbled upon "IMGUI" which is basically like an Immediate Graphics mode, but for user interfaces. I didn't give too much attention, since I was still in application development and the IMGUI scene itself seemed to be not really broad nor successfull. Still the approach they take seem to be so utterly sexy and elegant, that it made me want to write something for the next project using this way of UI (I failed to convince anyone at work :(...)

let me summarize what I mean by "retained" and "immediate":

Retained GUI: In a separate initialization phase, you create "GUI controls" like Labels, Buttons, TextBoxes etc. and use some descriptive (or programmatical) way of placing them on screen - all before anything is rendered. Controls hold most of their own state in memory like X,Y location, size, borders, child controls, label text, images and so on. You can add callbacks and listeners to get informed of events and to update data in the GUI control.

Immediate GUI: The GUI library consists of one-shot "RenderButton", "RenderLabel", "RenderTextBox"... functions (edit: don't get confused by the Render prefix. These functions also do the logic behind the controls like polling user input, inserting characters, handle character-repeat-speed when user holds down a key and so on...) that you can call to "immediately" render a control (doesn't have to be immediately written to the GPU. Usually its remembered for the current frame and sorted into appropiate batches later). The library does not hold any "state" for these. If you want to hide a button... just don't call the RenderButton function. All RenderXXX functions that have user interaction like a buttons or checkbox have return values that indicate whether e.g. the user clicked into the button. So your "RenderGUI" function looks like a big if/else function where you call or not call your RenderXXX functions depending on your game state and all the data update logic (when a button is pressed) is intermangled into the flow. All data storage is "outside" the gui and passed on-demand to the Render functions. (Of course, you would split up the big functions into several ones or use some class abstractions for grouping parts of the gui. We don't write code like in 1980 anymore, do we? ;))

Now I found that Unity3D actually uses the very same basic approach to their built-in GUI systems. There are probably a couple of GUI's with this approach out there as well?

Still.. when looking around, there seem to be a strong bias towards retained GUI systems? At least I haven't found this approach except in Unity3D and the original IMGUI community seems to be rather .... .. quiet.

So anyone worked with both ideas and have some strong opinion?

Edit: I am most interested in opinions that stem from real-world experience. I think there is a lot of heated discussions in the IMGUI-forum about any "theoretical weakness" of the immediate GUI approach, but I always find it more enlightening to know about real-world weaknesses.

Answer 1

Nay. I've done paid gamedev work on an awful 'retained mode' GUI and on an awful 'immediate mode' GUI and although both made me want to tear my eyes out, the retained mode approach is still clearly the better one.

The downsides of immediate mode are many:

• they don't make it easy for artists and designers to configure the layout;
• they make you mix logic with presentation;
• they make it harder to have a single consistent input model;
• they discourage complex controls of dynamic data (eg. list views);
• layering becomes very awkward (eg. if I call RenderComboBox, the drop-down box can't possibly render yet because it needs to go above the rest of the window - same for tool-tips)l
• configuring rendering style ends up being done with globals (ick) or extra function arguments (ick);
• performance can be poor because calling all these functions to query values that may or may not have changed tends to create many small objects, stressing the memory manager;
• ...and I can't even imagine how painful it would be to allow someone to drag bits the GUI around and re-order them. You'd have to maintain a data structure telling you where to render everything - which is much like rewriting a retained mode system yourself.

Immediate mode GUIs are tempting for lone programmers who want a quick HUD system, and for that purpose they are great. For anything else... just say no.

Answer 2

As someone with five or six years of actual experience with IMGUI on the desktop, I feel obligated to defend it. All of the responses (and the question itself) are based on a very restricted definition of IMGUI, and there seem to be a lot of questionable claims being made.

A lot of it originates from the assumption that an IMGUI library can't retain any data under-the-hood. This isn't true at all. A sophisticated IMGUI library will in fact retain about as much data as an equivalent RMGUI library. The difference is that an IMGUI library mostly caches results, whereas an RMGUI library maintains authoritative state. In IMGUI, the application provides a function that takes the current model state and produces a GUI for that state. This is the authoritative definition of the GUI. The library is responsible for making sure that the on-screen GUI matches the result of that function. This doesn't mean that it needs to fully evaluate that function each frame and fully rebuild the GUI. That's the point of caching.

With this view of IMGUI, you can in fact do advanced layout, and the performance is quite reasonable. You can also retain actual authoritative state if it makes the interface easier. The point of IMGUI is not to make the application retain everything. The application probably doesn't want to have to store the position of every scrollbar and the expanded/collapsed state of every tree node. The point is to be able to procedurally specify the contents of those tree nodes and their very existence.

I would go through and respond to all the IMGUI criticisms point by point, but I don't really understand a lot of them. Most of them seem to stem from some fundamental belief that IMGUI is hackish and RMGUI is well-structured, which I guess stems from the above misunderstandings. There's no inherent reason that IMGUI libraries should have trouble with complexity or have to be littered with globals or mix logic with presentation. I will say that the advantages of IMGUI become less important the more artist-driven your content is, but I'm not sure why it would actually be worse for artist-driven content. (I don't personally use artist-driven content, aside from style sheets for things like colors, font/border sizes, etc., but I don't see why it would be harder to implement than for RMGUI.)

In my mind, IMGUI is unquestionably a good thing. It gives you a much better model for reasoning about your GUI. It becomes much more functional and reactive, and you minimize the amount of mutable state that you have to reason about. On the other hand, implementing a sophisticated IMGUI library is quite a challenge, and definitely more difficult than implementing an equivalent RMGUI library. It's a tradeoff between library complexity and application complexity. If you're planning on building complex apps (or even a lot of simple apps), the tradeoff is very good. If you're doing this for one single app, and it's pretty simple, you'll probably achieve your goal faster with RMGUI.

In any case, good luck!

Answer 3

The GUI library consists of one-shot "RenderButton", "RenderLabel", "RenderTextBox"... functions that you can call to "immediately" render a control (doesn't have to be immediately written to the GPU. Usually its remembered for the current frame and sorted into appropiate batches later).

I would go so far as to say that this doesn't constitute a GUI library. It's just a bunch of rendering functions.

Rendering the GUI is the easiest part of making a GUI. Take a text box for example. I'm going to assume the easiest possible case: a simple single-line entry text box.

RenderTextBox will just draw a text box. It will do some simple text measurement and draw the glyphs on the screen. It will not

• Detect that the user has clicked the mouse in the control and move the cursor to the proper location in the string.
• Detect that the user double-clicked the mouse in the control and select a block of text.
• Detect that the user pressed the "Home" key and move the cursor to the front of the text.
• Detect that the user pressed a valid key and modify the string accordingly. If text was selected, the selected portion will be replaced with the inserted text.

I can keep going, but I think my point is clear. If you want to use RenderTextBox to draw a text box, all you will get is a static, non-functional text box. Any actual functionality must be implemented by you.

Doing text measurement and drawing glyphs? That's the easy part of GUI work. GUI stands for "Graphical User Interface". The last two words, where you take input from the user and do something with it, are the hard part.

Game players expect GUI controls to work reasonably. In a PC-style environment (ie: mouse and keyboard), if players see a text box, they expect it to work like a regular text box. They expect to be able to move around in it with arrow keys, jump to the front and end with home and delete, select letters in it, etc.

That means you're going to have to implement all of this UI code. You have to test what control was clicked. You then have to do something based on which control was clicked. You have to have the concept of an "active" control, the one that gets keyboard input. Different controls must respond differently to different types of user interaction.

Basically, you're going to need a TextBoxWindow class.

Let's say you're implementing a game options screen. So you have your different groups of options: gameplay, graphics, sound, controls, etc. So you have a list of different groups on the left side of the screen. Each group brings up a series of controls that the user can manipulate. What happens when the user presses the Tab key?

Well, it depends on what control is active. If one of the group controls is active (was most recently clicked), then it moves to the next group. If instead one of the controls within the group is active, then it moves to the next control in that group. That's how the system is intended to work.

So not only must an active control process keyboard input, it must now farm any unprocessed input to the control that owns it. Either that, or the controls must be in some kind of linked list, so that it can go back and forth. That means there needs to be default behavior in each control.

This sounds more and more like a retained GUI, doesn't it? The only difference is that... you are the one doing the hard work. Using someone elses GUI is supposed to be a way to do less work. But the effort it takes to make a GUI respond as the user expects is non-trivial.

Remember: the UI isn't for you, it's for your users. It's for the player. It should behave as they expect it to. And if it doesn't, then you have failed.

User interfaces usually don't need to be auto-layouted or have resizable windows on-the-fly.

That is a limited and limiting thought.

I could give the spiel about different games having different UI needs, that some games do need resizeable windows and so forth. But there's a much bigger problem.

Your kind of thinking leads to the Gratuitous Space Battles problem.

If you've never played that game, it's a cheap, indie, and very GUI-heavy game. The actual gameplay is all done in the GUI (it's a "setup the units and watch them fight" kind of game). The problem?

THE GUI CANNOT BE RESCALED!

Oh, this is fine if you're using a normal monitor or have normal eyesight. But if you're me, for example, who runs a ridiculously huge monitor (one so big that you have Windows scale up the size of all text so you can actually read it), this is terrible. The text is microscopic. There's no way to change the font size.

Granted, GSB is a GUI-based game, so it's absolutely inexcusable for them. A GUI-lite game can probably get away with it.

Never assume that your user is looking at their game exactly as you are. To do so is folly.

Having a GUI system that can rescale itself to the user's resolution, a truly resolution-independent GUI, requires layout to be part of the GUI system itself. If you're not going to provide that, then your text is going to appear tiny on someone's giant monitor. This is not a good thing.

So I would say that your thinking on this subject is shortsighted. You do need that stuff. You need what retained GUIs provide. Oh, you may not need everything any particular GUI system provides. But you need some of it.

The boilerplate work you need to do to get data to the system is a small price to pay next to having reasonable assurance that the user can interact with controls correctly and that it will resize itself to fit the player's needs.

You might be able to get away with an immediate-mode GUI if you don't take much user input from the player. But that'd be about it.

Answer 4

A while ago IMGUIs also caught my interest, as a test I wrote a couple of tutorials to familiarize myself with the techniques (start here if you're interested, but it's not much more than C#/XNA + the original 'tutorial' here).

I really liked IMGUIs for a short while because the info they display is always up-to-date and correct (since there is no state you always feed to actual data). But in the end I lost interest so normally now I use retained GUIs again.

In the end I thought that the immediateness of the GUI made it harder to separate the GUI from the data and at times I tried to decouple things more by introducing some state, breaking the elegance of IMGUIs. I also don't think that writing code for retained GUIs is more work than writing code for IMGUIs and I like that retained GUIs can be fired and forgot about, and I really like events :).

However, every time someone mentions IMGUIs something inside me wants to try again, and try harder to get it right, because there really is some elegance in there, I'm just not 100% sure if it will always make your job easier. I'd say try it out, maybe try as I did and write some tutorials (I find the best way to learn new techniques is by explaining them to others).

Answer 5

I worked a lot with Unity3D GUI system, which works just as you described. And I must say, I hate it with a passion.

"Immediate" approach works really good in some cases. First, when you only need a couple of buttons and labels - think shooter HUD, for example. Creating them with "retained" approach is not very hard, but its super-easy with UnityGUI. Second case is when you need to stuff lots of controls on screen, but do not really care about layout. Especially when exact controls are only known at runtime. This is not actually useful in any game, but really helpful when making tools running inside Unity editor.

However, any half-complex GUI - for an (MMO)RPG or a strategy game, for example - inevitably turns into a horrible mess of indecipherable code, full of special cases and breaking in tons of ways. When building such a GUI, you usually have pretty specific layout that must work for any resolution and be able to show whatever data you send its way. You need things like, for example, moving some buttons lower to make way for longer text. With "retained" GUI, you can have a control that does that automatically. With "immediate" mode, there are no controls, so you have to do everything explicitly.

Another problem with UnityGUI (not sure if it happens to all immediate mode GUIs) is that, for example, a Button() call works without considering any other GUI calls. So, if one button is on top of another, a click will press both buttons at the same time. This is definitely not what user expects, so this adds another special case to handle.

To live with all this, I've always written my own wrapper around UnityGUI that turns it into a "retained" mode system: controls that store their own state and just call required GUI functions every frame for me.

I can't say that "immediate" mode is definitely worse than "retained", but I certainly feel much better working in "retained" mode.

Answer 6

I am going to defend IMGUI here because some of the problems that were listed previously can be solved if you are willing to write the code for it. Additionally if you do write the code for it you then have a robust library that is re-usable and only very rarely requires modification.

• loading GUI from schema

perhaps at first it seems that artists do not have a lot of flexablity with IMGUI, this is solved or improved with loading GUI layout from an xml or JSON schema.

• menu layer draw calls

this problem is solved with sorting, you should create UI Manager class that sorts the order of draw, I solved this problem in C# XNA with moveable, click-able panels that draw on top of each other. I can post pseudo code if asked.

• list boxes

can you draw strings from an array? can you define rectangle areas from the height and width of the fonts of those strings? can you create a ratio of the size of your scroll button by the height of all those rectangles added together? Then you are halfway to creating a listbox with a scrollable button to move up or down. I thought it was going to be hard, but it turned out to be much more simple than I thought. no buffers, no states, no callbacks involved.

• separating data from GUI

don't let the individual panels, buttons, boxes hold the data. I know that at my first attempt of IMGUI was immediate but only in the graphic sense. I made the mistake of retaining data on the UI. It was almost a philosophical shift to think differently on where to place and modify data via the UI changes.

• GUI from SDKs

this is the limit and functionality of the SDK's code, not yours. I find that SDK UI (Hammer, Unity, UDK) is almost a new language to learn anyway. In fact they are similar to me as Windows.Forms, both are set up for the broadest possibility and therefor have some extra weight in complexity.

and lastly watch this > http://mollyrocket.com/861

Answer 7

User interfaces usually don't need to be auto-layouted or have resizable windows on-the-fly. Instead, they need to interact very efficiently with always-changing data (like 3d-positions of models in the world)

It depends on the genre and game in question. A good inventory management system is vital for most any RPG. You going to want something that handle layouts for you, supports scroll bars, drag n' drop, tooltips, and other features much easier to implement using Retained GUI.

I can't think of a way to drag n' drop with an immediate GUI that doesn't take a lot juggling or user state saving such as changing the Z buffer temporarily and storing time info to rule out a click that happened to move a bit.

But from a design perspective I have trouble imagining a world without my GUI events. Also an Immediate GUI is not compatible with OOP forcing you to resort to procedural programming.

The simplicity afforded by immediate GUIs comes at the cost of additional complexity in your code that grows rapidly as the complexity required by you UI increases; where as a good retained GUI is more complex originally but scales better. So below a certain point of complexity a immediate GUI is adequate but for most situations I'd rather a retained GUI.