How to batch coroutines (IEnumerators) in parallel

Question

We are working in a context where multithreading is not an option, so we're using coroutines to handle our loads. Often there are multiple things that need to be downloaded, leading to a pattern like this:

// set up multiple downloads
dataToDownload = 3;
StartCoroutine(ReadDataCoroutine1(OnDataRead));
StartCoroutine(ReadDataCoroutine2(OnDataRead));
StartCoroutine(ReadDataCoroutine3(OnDataRead));

// handle each of the loads completed
private void OnDataRead()
{
    // nasty and error prone way to check completion of all data
    dataToDownload--;
    if (dataToDownload > 0) return;

    DoStuffNowThatDataIsAllLoaded();
}

So I thought I would build a helper class to batch the downloads:

public BatchDownloader(
    IEnumerable<Func<IEnumerator>> downloadFuncs,
    Action<int, int> onDownloadComplete = null,
    Action<int> onAllDownloadsComplete = null)
{
    this.downloadFuncs = downloadFuncs;
    this.onDownloadComplete = onDownloadComplete;
    this.onAllDownloadsComplete = onAllDownloadsComplete;
    totalDownloads = downloadFuncs.Count();
}
public IEnumerator DownloadBatch()
{
    completedDownloads = 0;
    foreach (var downloadFuncs in downloadFuncs)
        yield return ProcessDownload(downloadFuncs);
    onAllDownloadsComplete?.Invoke(totalDownloads);
}
private IEnumerator ProcessDownload(Func<IEnumerator> downloadFunc)
{
    yield return downloadFunc();
    onDownloadComplete?.Invoke(totalDownloads, ++completedDownloads);
}

Used something like this:

public IEnumerator LoadAllTheThings()
{
    Func<IEnumerator>[] downloadFuncs = new Func<IEnumerator>[]
    {
        ReadDataCoroutine1,
        ReadDataCoroutine2,
        ReadDataCoroutine3
    };

    BatchDownloader batchDownloader = new BatchDownloader(downloadFuncs);

    yield return batchDownloader.DownloadBatch();

    DoStuffNowThatDataIsAllLoaded();
}

Now that is much cleaner, more flexible, and it all works a treat, except that no load starts until the previous finishes. This obviously has a huge effect on load time. So in an attempt to have the coroutines run simultaneously, I'm trying the following:

public IEnumerator DownloadBatch()
{
    completedDownloadsCount = 0;

    var activeDownloads = new List<IEnumerator>();
    foreach (var downloadFunc in downloadFuncs)
    {
        activeDownloads.Add(downloadFunc());
    }

    while (activeDownloads.Count > 0)
    {
        yield return null;
        for (int i = activeDownloads.Count - 1; i >= 0; --i)
        {
            var download = activeDownloads[i];
            if (!download.MoveNext())
            {
                onDownloadComplete?.Invoke(totalDownloadsCount, ++completedDownloadsCount);
                activeDownloads.RemoveAt(i);
            }
        }
    }

    onAllDownloadsComplete?.Invoke(totalDownloadsCount);
}

Now while this correctly kicks off the coroutines, and runs them "in parallel", any code after the last yield in a coroutine is not executed. So in a method like the following:

private IEnumerator ReadDataCoroutine1()
{
   yield return WaitForStuffToDownload();

   DoPostDownloadStuff();
}

DoPostDownloadStuff() is never called.

What am I misunderstanding here about stepping through these IEnumerators? Is there a way to do what I'm trying to, without introducing StartCoroutine into the batch downloader? (that is my solution for now, but idealy I'd like to not have to)

Answer 1

For completeness sake, here is the version that uses a passed in MonoBehaviour to run the downloads.

[Edit]: I've settled for this solution, I haven't had any success with a version that does not use StartCoroutine. [/Edit]

Note also the fix in ProcessDownload, for the case where no onDownloadComplete delegate was provided (whoops).

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;

namespace MB.Other
{
    public class BatchDownloader
    {
        private readonly IEnumerable<Func<IEnumerator>> downloadFuncs;
        private readonly Action<int, int> onDownloadComplete;
        private readonly Action<int> onAllDownloadsComplete;
        private readonly int totalDownloadsCount;

        private int completedDownloadsCount;

        public BatchDownloader(
            IEnumerable<Func<IEnumerator>> downloadFuncs,
            Action<int, int> onDownloadComplete = null,
            Action<int> onAllDownloadsComplete = null)
        {
            this.downloadFuncs = downloadFuncs;
            this.onDownloadComplete = onDownloadComplete;
            this.onAllDownloadsComplete = onAllDownloadsComplete;
            totalDownloadsCount = downloadFuncs.Count();
        }

        public IEnumerator DownloadBatch(MonoBehaviour coroutineRunner)
        {
            completedDownloadsCount = 0;

            foreach (var downloadFuncs in downloadFuncs)
                coroutineRunner.StartCoroutine(ProcessDownload(downloadFuncs));

            while (totalDownloadsCount != completedDownloadsCount)
                yield return null;

            onAllDownloadsComplete?.Invoke(totalDownloadsCount);
        }

        private IEnumerator ProcessDownload(Func<IEnumerator> downloadFunc)
        {
            yield return downloadFunc();

            completedDownloadsCount++;

            onDownloadComplete?.Invoke(totalDownloadsCount, completedDownloadsCount);
        }
    }
}