2
\$\begingroup\$

Description

I'm working on a system to seamlessly reconstruct and play numerous segments of a continuous video stream by playing one segment on the active Video Player while loading the next on the hidden, waiting Video Player—and swapping between them when the active segment is done playing.

Although it's not necessary to see the code for this, I included it below just in case:

public class VideoController : MonoBehaviour
{
    [SerializeField] private VideoPlayer activePlayer;
    [SerializeField] private VideoPlayer waitingPlayer;
    // The waiting player starts off invisible.

    private Queue<string> _segmentUrls;
    private bool _videoIsPlaying;

    private bool NextSegmentExists => _segmentUrls.Count >= 1;

    private void Awake()
    {
        // Initialize fields.
        _segmentUrls = new Queue<string>();

        // Subscribe to events.
        // VideoPlayer.Prepare() doesn't seem to work in WebGL. The workaround is to call
        // Play() instead and set the players to pause once the track is finished preparing.
        activePlayer.prepareCompleted += source =>
        {
            source.Pause();
        };
        waitingPlayer.prepareCompleted += source =>
        {
            source.Pause();
        };
        
        activePlayer.loopPointReached += SwitchVideoPlayer;
        waitingPlayer.loopPointReached += SwitchVideoPlayer;
    }

    public void EnqueueSegment(string segmentUrl)
    {
        _segmentUrls.Enqueue(segmentUrl);

        if (!_videoIsPlaying)
            StartCoroutine(StartPlayback());
        else if (!waitingPlayer.isPrepared)
            PrepareNextSegmentOnWaitingPlayer();
    }

    private IEnumerator StartPlayback()
    {
        PrepareNextSegmentOnWaitingPlayer();
        
        // Give the Video Player a second to finish preparing the segment for playback.
        yield return new WaitForSeconds(1f);

        SwitchVideoPlayer(lastPlayer: activePlayer);
        _videoIsPlaying = true;
    }

    private void PrepareNextSegmentOnWaitingPlayer()
    {
        var segmentUrl = _segmentUrls.Dequeue();
        waitingPlayer.url = segmentUrl;
        waitingPlayer.Play();
    }

    private void SwitchVideoPlayer(VideoPlayer lastPlayer)
    {
        activePlayer = waitingPlayer;
        TransparencyController.ShowGameObject(activePlayer.gameObject);

        waitingPlayer = lastPlayer;
        TransparencyController.HideGameObject(waitingPlayer.gameObject);
        waitingPlayer.Stop();
        
        if (activePlayer.isPrepared)
            activePlayer.Play();
        else
        {
            activePlayer.Stop();
            _videoIsPlaying = false;
            return;
        }

        if (NextSegmentExists)
            PrepareNextSegmentOnWaitingPlayer();
    }
}

Problem

When a Video Player begins playing, the first frame or two is just a black screen. As a result, when a segment finishes playing and it's time to swap the active and waiting players, there's a momentary black "blink" on the screen.

Demonstration

Below is a GIF of the issue:

Blinking Demo

Each blink corresponds to swapping Video Players.

(Not to be confused, the video player is showing a recording of a Unity project.)

Attempted Fix #1

When a Video Player is done preparing the new segment for playback, I have it advance two frames, so it picks off right where the old segment ended and transitions perfectly smoothly.

This is done by changing the callback for VideoPlayer.prepareCompleted as follows:

activePlayer.prepareCompleted += source =>
{
    source.StepForward();
    source.StepForward();
    source.Pause();
};
waitingPlayer.prepareCompleted += source =>
{
    source.StepForward();
    source.StepForward();
    source.Pause();
};
// Experimentation found that two calls to StepForward() are necessary.

Why It Doesn't Work

It works on Windows, but testing it on the WebGL build—my primary build target—throws an exception telling me VideoPlayer.StepForward() is not supported on WebGL.

Attempted Fix #2

I tried the equivalent of calling StepForward() by setting the current time of the Video Player to a value equal to 2 / framerate, where the "2" stands for two frames.

Why It Doesn't Work

It works on Windows, and it seems adjusting the time value works on WebGL, too—at least for values equal to or less than 0.5 seconds. However, on WebGL, the black blink persists despite the time-skip.

Attempted Fix #3

I tried the same approach as in #2, but by directly adjusting the current frame of the Video Player when preparing the segment for the waiting player.

Why It Doesn't Work

For the same exact reason as in #2. 😫

Question

Why does the Video Player start off with black frames on WebGL no matter what, and how can I avoid that?

\$\endgroup\$

1 Answer 1

1
\$\begingroup\$

I settled on the following solution: start playback on the waiting player a few frames before the active player finishes its segment.

Since I still don't understand what's causing the black screen on Video Players, I gave up trying to fix it and decided to work around it. Every frame, I check to see if the active player is near the end, and if so, start the waiting player—while it's still hidden.

private void Update()
{
    // The below code starts the playback of the waiting player early; when the active player is almost finished with its segment.
    // This gives the waiting player time to "play out" the first few "black" frames it starts out with and get to the actual video.
    // The waiting player will become visible (and active) when the current, active player completely finishes playing.
    if (activePlayer.isPlaying && waitingPlayer.isPrepared)
    {
        var frameCount = 2;
        
        #if UNITY_EDITOR
        frameCount = 3;
        #endif
        
        var framesInSeconds = frameCount / activePlayer.frameRate;
        var currentSegmentIsEnding = activePlayer.time >= activePlayer.length - framesInSeconds;

        if (currentSegmentIsEnding)
            waitingPlayer.Play();
    }
}

It's still not perfect. I can't precisely determine how many (video) frames the black screen will last, and therefore, there's a slight freeze/stutter when the Video Players switch. As you can see from the code, I start the waiting player 3 frames before the active's end in the Unity Editor, where I use 60 FPS clips, and 2 frames in my WebGL build, where it's 24 FPS. I'm not even sure I should calculate the advance time by (video) frames—it seems to vary each swap.

Until then, I'm eyeballing it. I'm going to test this on several devices to narrow down a satisfactory number, but if anyone knows a better way—or the cause for the black frames—please, chime in.


The full, unedited code can be viewed here:

public class VideoController : MonoBehaviour
{
    [SerializeField] private VideoPlayer activePlayer;
    [SerializeField] private VideoPlayer waitingPlayer;

    private Queue<string> _segmentUrls;
    private bool _videoIsPlaying, _waitingPlayerIsCurrentlyPreparing;

    private bool NextSegmentExists => _segmentUrls.Count >= 1;

    private void Awake()
    {
        // Initialize fields.
        _segmentUrls = new Queue<string>();
        
        // Hard-configure required settings.
        activePlayer.source = VideoSource.Url;
        waitingPlayer.source = VideoSource.Url;
        activePlayer.isLooping = false;
        waitingPlayer.isLooping = false;

        // Subscribe to events.
        activePlayer.prepareCompleted += source => source.Pause();
        waitingPlayer.prepareCompleted += source => source.Pause();
        activePlayer.loopPointReached += source => StartCoroutine(OnSegmentEnd());
        waitingPlayer.loopPointReached += source => StartCoroutine(OnSegmentEnd());
    }

    private void Start()
    {
        #if UNITY_EDITOR
        StartCoroutine(EnqueueSegmentsForTesting(3));
        #endif
    }

    private void Update()
    {
        // The below code starts the playback of the waiting player early; when the active player is almost finished with its segment.
        // This gives the waiting player time to "play out" the first few "black" frames it starts out with and get to the actual video.
        // The waiting player will become visible (and active) when the current, active player completely finishes playing.
        if (activePlayer.isPlaying && waitingPlayer.isPrepared)
        {
            var frameCount = 2;
            
            #if UNITY_EDITOR
            frameCount = 3;
            #endif
            
            var framesInSeconds = frameCount / activePlayer.frameRate;
            var currentSegmentIsEnding = activePlayer.time >= activePlayer.length - framesInSeconds;

            if (currentSegmentIsEnding)
                waitingPlayer.Play();
        }
    }

    private IEnumerator EnqueueSegmentsForTesting(int segmentCount)
    {
        for (int i = 0; i < segmentCount; i++)
        {
            EnqueueSegment($"file://D:/Projects/ffmpeg-wasm-test/Assets/Videos/{i}.mp4");
            yield return new WaitForSeconds(2.5f);
        }
    }

    public void EnqueueSegment(string segmentUrl)
    {
        _segmentUrls.Enqueue(segmentUrl);

        if (!_videoIsPlaying)
            StartCoroutine(StartPlayback());
        else if (!waitingPlayer.isPrepared && !_waitingPlayerIsCurrentlyPreparing)
            StartCoroutine(PrepareWaitingPlayer());
    }

    private IEnumerator StartPlayback()
    {
        // Check the _videoIsPlaying flag at the very beginning to prevent StartPlayback() from being 
        // started multiple times when EnqueueSegment() is called in rapid succession.
        _videoIsPlaying = true;
        PrepareNextSegmentOnVideoPlayer(activePlayer);

        while (!activePlayer.isPrepared)
            yield return null;
        
        activePlayer.Play();
    }

    private IEnumerator PrepareWaitingPlayer()
    {
        _waitingPlayerIsCurrentlyPreparing = true;
        PrepareNextSegmentOnVideoPlayer(waitingPlayer);

        while (!waitingPlayer.isPrepared)
            yield return null;

        _waitingPlayerIsCurrentlyPreparing = false;
    }

    private void PrepareNextSegmentOnVideoPlayer(VideoPlayer videoPlayer)
    {
        var nextSegmentUrl = _segmentUrls.Dequeue();
        videoPlayer.url = nextSegmentUrl;
        
        // VideoPlayer.Prepare() doesn't seem to work in WebGL. The workaround is to call
        // Play() instead and set the players to pause once the track is finished preparing. 
        // (The pausing happens in a callback to the VideoPlayer.prepareCompleted event.)
        videoPlayer.Play();
    }

    private IEnumerator OnSegmentEnd()
    {
        activePlayer.Stop();

        // If the waiting player isn't already loaded with the next segment...
        if (!waitingPlayer.isPrepared && !_waitingPlayerIsCurrentlyPreparing)
        {
            // ...then the end of the segment buffer has been reached.
            waitingPlayer.Stop();
            _videoIsPlaying = false;
            yield break;
        }

        // The exception is when it's currently preparing.
        if (_waitingPlayerIsCurrentlyPreparing)
        {
            // In that case, wait for it to finish, play it, and continue as normal.
            while (!waitingPlayer.isPrepared)
                yield return null;
            
            waitingPlayer.Play();
        }

        // Otherwise, assume the waiting player has already begun playback.
        SwapVideoPlayers();

        if (NextSegmentExists)
            PrepareNextSegmentOnVideoPlayer(waitingPlayer);
    }

    private void SwapVideoPlayers()
    {
        var lastPlayer = activePlayer;
        activePlayer = waitingPlayer;
        waitingPlayer = lastPlayer;

        TransparencyController.ShowGameObject(activePlayer.gameObject);
        TransparencyController.HideGameObject(waitingPlayer.gameObject);
    }
}
\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .