How to preserve EGL context when pausing Android ndk app

Question

I am trying to improve openframeworks so that the GL context is preserved when pausing and resuming my game. So that I don't have to reload all textures after each pause. I don't expect you to know openframeworks code, I'll explain my problem independent of it.

I have java activity which loads a main_layout.xml file with a RelativeLayout inside another RelativeLayout. Then I instantiate a GLSurfaceView and add it to the inner RelativeLayout using glContainer.addView(glView, new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));. In onPause function, I call glView.onPause(); on GLSurfaceView and in onResume, I call glView.onResume(); on GLSurfaceView.

This code works but it destroys the gl surface in onPause (GLSurfaceView.surfaceDestroyed gets called on my GLSurfaceView, which is extended to OFGLSurfaceView, so that I can intercept this call).

To fix this, I added call to GLSurfaceView.setPreserveEGLContextOnPause with param 'true' to initialization of my GLSurfaceView. Now I see only black screen after pausing and resuming the game. I found out, that I can fix this by removing the GLSurfaceView from parent RelativeLayout and adding it back right away in onResume function: glContainer.removeView( glView ); glContainer.addView( glView, new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT) );. This however destroys the gl surface again and forces me to reload textures.

There are no reinstantiations of either one of the RelativeLayouts or the GLSurfaceView (I checked it by looking at the code and then checking identities of those instances with System.identityHashCode).

main_layout.xml:

xml
<?xml version="1.0" encoding="utf-8"?>
   <RelativeLayout android:id="@+id/relativeLayout1" android:layout_width="fill_parent" android:layout_height="fill_parent" xmlns:android="http://schemas.android.com/apk/res/android">
        <RelativeLayout android:id="@+id/of_gl_surface_container" android:layout_width="fill_parent" android:layout_height="fill_parent"/>
        <!-- add here other views' layouts -->
    </RelativeLayout>

Code in OFAndroidLifeCycle.java which handles events received from Activity:

java
public static void glCreateSurface()
{
    if(mGLView == null)
    {
        mGLView = new OFGLSurfaceView(m_activity);

        OFGLSurfaceView glView = getGLView();

        glView.setPreserveEGLContextOnPause( true );

        ViewGroup parent = (ViewGroup)glView.getParent();
        if( parent == null )
        {
            ViewGroup glContainer = getActivity().getSurfaceContainer();
            glContainer.addView(glView, new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
        }
    }
}

public static void glPause()
{
    OFGLSurfaceView glView = getGLView();
    if( glView != null )
        glView.onPause();   
}

public static void glResume(ViewGroup glContainer)
{
    OFGLSurfaceView glView = getGLView();

    if( glView != null )
    {
        glView.onResume();

        glContainer.removeView( glView );
        glContainer.addView( glView, new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT) );
    }
}

After working on it a bit, I improved it so that the GLSurfaceView is being paused and resumed from Activity.onStop and Activity.onStart respectively.

Problem remains and I was able to better locate the source of the problem. The surface is destroyed between onPause and onStop events, or better to say right before onStop event. Here is the stack trace of the call that reports that the surface was destroyed.

java.lang.Exception
    at cc.openframeworks.OFGLSurfaceView.surfaceDestroyed(OFAndroidWindow.java:251)
    at android.view.SurfaceView.updateSurface(SurfaceView.java:639)
    at android.view.SurfaceView.windowStopped(SurfaceView.java:222)
    at android.view.ViewRootImpl.setWindowStopped(ViewRootImpl.java:1348)
    at android.view.WindowManagerGlobal.setStoppedState(WindowManagerGlobal.java:623)
    at android.app.Activity.performStop(Activity.java:7151)
    at android.app.ActivityThread.performStopActivityInner(ActivityThread.java:4158)
    at android.app.ActivityThread.handleStopActivity(ActivityThread.java:4210)
    at android.app.ActivityThread.-wrap24(Unknown Source:0)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1646)
    at android.os.Handler.dispatchMessage(Handler.java:105)
    at android.os.Looper.loop(Looper.java:164)
    at android.app.ActivityThread.main(ActivityThread.java:6710)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:770)

This corresponds to android-26 sdk sources.

I am still not sure where the problem is. I consider three alternatives:

1. problem with android platform version (I could use some advises here)
2. the way openframeworks interfaces with the GLSurfaceView might be somehow invalid (overriding some functionality)
3. other problem in propagation of that preserveEGLContextOnPause flag, I will need to read through the android code to better understand how that works

I found it! Summary:

• The point is that GL objects do not get destroyed when GLSurfaceView.surfaceDestroyed is called. I kept the setPreserveEGLContextOnPause on true and disconnected any signals to native code from GLSurfaceView.surfaceDestroyed and now the textures are good after minimizing and restoring the window.

So I guess that the surface and probably also EGL context can be destroyed and textures will still remain in GL context or something.

So now I just need to do some cleanup work and find the real indicator that GL objects were destroyed. And maybe try to verify that this approach really is the correct one, but I am confident I am on the right track. I might post more info once I do some more work on it.

Answer 1

Introduction to terminology and description state transitions are in the Android Understand the Activity Lifecycle guide. There are basicaly three states:

First state is CREATED, which is reported through the onCreate and onDestroy callbacks in Activity class. This state is entered at the beginning of the lifetime and exited at the end of the lifetime. Android app should be able to save its state during onDestroy and restore its state (if available) in onCreate so that Android can quickly and cleanly reload the app when for example screen configuration changes. These quick restarts do not really apply to games, the expected behavior from games is that they restart as rarely as possible.

If the Activity is in the CREATED state, it can enter second state - STARTED, that is reported through onStart and onStop callbacks of Activity class. The app can enter STARTED state multiple times during its lifetime. STARTED means that some part of Activity window is visible to user. Activity is not STARTED when we minimize the Activity or switch to another Activity. This is when EGL context is lost in default configuration and when Activity should be running on low resources without animation and rendering updates.

Third state is RESUMED, which can be entered from CREATED state and is reported through onResume and onPause callbacks of Activity class. This is not that useful, it basicaly means that the Activity has input focus. So for example in multiwindow mode, where two Activities are visible, both of them are CREATED but only the one that received input last is in RESUMED state. For games this basicaly means that when we exit RESUMED state (onPause is called), we should pause the gameplay. When the game enters the RESUMED state, we should do nothing but wait for user to manually unpause the game when he is ready to continue playing.

Now to the technicalities. When we are making game in Android, we can use either NativeActivity that I am not familiar with, or we can use standard java Activity class and request updates and renders from native code using NDK JNI calls. In that case, we will be usualy using GLSurfaceView. That is basically a UI element that calls our custom callback GLSurfaceView.Renderer.onDrawFrame in which an OpenGL context is available and we can make OpenGL calls and whatever we draw during this callback will be visible in the GLSurfaceView. We need to call GLSurfaceView.onPause when Activity exits STARTED state (Activity.onStop is called) and call to GLSurfaceView.onResume when Activity enters STARTED state (Activity.onStart is called).

Historically, there could be only one OpenGL surface existing at a moment, so when apps were switched, only the app in foreground had valid opengl context and apps in background had to restore the state of opengl context (reload textures, etc.) when they came to foreground. Since api level android_11, more that one opengl context can exist (it needs to be requeste by the app), but we still need to be able to handle the case when system decides to kill either the opengl context or the whole app due to system resources constraints. I am not sure if devices with api android_11 and higher may even kill only the opengl context or if they kill the whole app every time they run out of system resources. It might be useful for you to read more about this if you would like to avoid the necessity to reload resources on demand in your project.

Now we need a way to tell that opengl context was destroyed so we can reload all the textures and other resources. To do that, we can use callback GLSurfaceView.Renderer.onSurfaceCreated on renderer that is assigned to our GLSurface view. The renderer does not receive any callback when surface is destroyed, only when it is created. But that is not a problem. While the surface exists, the onDrawFrame callback is being repeatedly called. When the app goes to background, the onDrawFrame will not be called anymore. When the app enters foreground, either onSurfaceCreated will be called first, in which case we know that we need to reload everything before next draw or it will not be called and onDrawFrame is again repeatedly called in which case we are good to draw with old opengl resources.

Openframeworks was using GLSurfaceView.surfaceDestroyed callback to determine that opengl context was lost. But according to my testing, this is not how it should be used. That callback is called somewhere around onPause and onStop when app is going to background. And this happens even when the opengl context is not in fact destroyed. So don't use that. Second problem I had was that various callbacks in the code were a bit confused, so the rendering might sometimes get paused by custom bool variable, which caused the black screen in some test cases.

To tell Android to try to preserve the opengl context even when minimizing the app, we need to use method GLSurfaceView.setPreserveEGLContextOnPause - call it with parameter 'true' after the GLSurfaceView was instantiated. This will cause that GLSurfaceView.Renderer.onSurfaceCreated will not be usually called after minimizing and restoring the app.

I am glad I finaly solved this riddle, so back to gameplay programming for me :-).

Answer 2

That your context gets discarded when the user switches to a different app is a feature to free up memory for the different app.

However pause can also mean pausing the game and seeing the pause menu from witch you can edit options etc. This is the meaning that you want I believe.

If you want to do that then you don't need to call onPause but instead swap over to a different "game layer" which is a simple menu just like the main menu.

The most straightforward way to do so in any framework would be to add a switch(layer) in update() and draw() that dictates what gets done in each layer. So when the player hits the pause button you do layer = PAUSE; and the next frame will be the pause menu which you can add a nice animation to to make it swipe in.

A quick gander through the doc of openframeworks shows a way to switch the current main window. But I'm not sure that is fully what you want.