How can you sync video files with animations within Unity?

Question

I'm wondering if anyone might be able to offer some advice regarding an issue I am currently facing in trying to sync video files with animations within Unity.

Ideally my aim is to have a video file play whilst an animated camera (exported from blender) moves in sync with the rendered video file. I've nearly got this working, however there is a very slight delay between the two in which at certain points (quick rotations of the camera) the animation lags behind the video.

I have attempted to solve this issue by creating a script in which for every frame of the video, the animation component is accessed and the animation file frame set to its equivalent time with regards to the video player position. To do so I made use of 'animationState.normalizedTime' (mapping the video frame from its current position to a value of 0 - 1, so as to fit within the requirements of the normalized time function).

I'm also trying to make use of 'videoPlayer.frameReady' in order to set the animation with each new frame, with the idea being to try to only set the animator once a new frame is displayed.

The result I'm currently getting is that the animation is linked to the playback of the video, and will match up regardless of speed of playback, however there are parts in which the animation does not match up to the video. These are particularly noticeable when the camera makes a sudden movement, or when it is rotating. My current thoughts are that perhaps this could have something to do with camera easing settings (from blender to unity) or with the animation component settings. I've tried to remove animation compression and keyframe reduction, however am still having these issues unfortunately.

Does anyone have any ideas as to how this might be solved?

I've included my code below :

public class animationSync1 : MonoBehaviour
{

    // Animation component 
    private Animation animator;

    // Video player (add via inspector)
    public VideoPlayer vp; 

    // total frame count of video 
    private float totalFrame; 

    // current video frame
    private float currentFrame;

    // animation frame count
    private float animationFrame;

    // Start is called before the first frame update
    void Start()
    {

        // Animation component
        animator = GetComponent<Animation>();

        // get total number of frames from video file
        totalFrame = vp.frameCount;

        // start video playback
        vp.Play();

        // begin playback of animation
        animator.Play("Camera_entrance|Camera_entranceAction");
        // set animation speed to 0 in order to set the animation frame point manually in fixed update
        animator["Camera_entrance|Camera_entranceAction"].speed = 0f;
        
    }

    // Use of fixed update to call 30 times a second (to try to match up to video and animation playback)
    void FixedUpdate()
    {

        // get current video frame
        currentFrame = vp.frame;

        // get frame count between 0 - 1 from current video frame 
        animationFrame = currentFrame / totalFrame; 

        // set the animation position to that of the current video frame
        animator["Camera_entrance|Camera_entranceAction"].normalizedTime = animationFrame;
        
    }
}

Answer 1

I think checking the frameReady event is the way to go. I managed to solve my stuttering issues by updating the camera animation every time the video frame had changed. In my case I was updating every frame anyway, so i didn't bother subscribing to the frameReady event.

I just checked the previous frame against the current one and updated the camera position then redrew my backplate quad that was drawing the video.

I'll test to see if there is better performance from using the frame ready event as well. But currently this seems to work fine for me as the video frame rate is well below my update frame loop.

The following code snippet works for me ;

    void Update()
    {
        float current_video_time = (float)videoplayerObj.time;
        if (current_video_time != previous_video_time)
        {
            Mesh mesh = planeObj.GetComponent<MeshFilter>().mesh;
            //enamble camera animation
            cameraAnimObj.Play();
            cameraAnimObj["Take 001"].time = myTimeoffset + (float)videoplayerObj.time;
            previous_video_time = current_video_time;
            //force camera position update
            cameraAnimObj.Sample();
            //stop camera animation
            cameraAnimObj.Stop();
        }
    }