Unity3D multi-threaded data importer?

Question

Goal

I would like to be able to separate all my data file readers from the main thread and balance the load over multiple frames. The idea being that while the data is being read/parsed/filtered I can continue playing the game at a decent FPS and show a loading progress bar while the results of the file importing pop in one at a time when they are ready.

Information

The data files contain things like transform information for objects, points to be used in runtime mesh generation, and terrain data. They all have to be parsed, filtered, and/or modified to be ready to be used in the scene.

I already have import code for the project I just need to transplant it from blocking my main thread and being done in one frame to a multi-threaded and multi-frame approach.

Thoughts

I have little experience in multi-threading so this is also a learning project for me. I would appreciate all your comments, input, code samples, and pointers.

From my perspective I need to create a queue of tasks. Each task being a file to load, read, parse, and filter. Then if a CPU is open a task is taken from the queue and is given a thread to be executed on that CPU. The task completes and gives the resulting data back to Unity3D for it to handle.

Questions

1. Does a heavy load on other CPUs freeze the game like it would if it were on the main thread, and thus require some sort of system where you yield for one frame if execution time is taking too long?
2. How do I receive progress updates from the tasks to be used in a interface progress bar via some sort of co-routine?
3. If a user only has one CPU how would this system handle the situation? Is that even a realistic scenario?
4. What sort of C# classes (usable in Mono's subset of the .NET runtime) already exist for multi-threading and task queuing that can save me time?

Answer 1

Here is some example code to expand on my comment above. A true multi threaded solution would be faster than this, but susceptible to many more problems and edge cases. Unity's model for concurrency is coroutines, and thus using them aligns your code with what is expected in the unity space. Unless you have a real need for ultra fast file processing, this solution should work.

using UnityEngine;
using System.IO;
using System.Collections;
using System.Collections.Generic;

public class FileImporter : MonoBehaviour
{
    public int LinesToReadPerFramePerFile = 100;

    private int _linesProcessed = 0;
    private Dictionary<string, Coroutine> _currentCoroutines = new Dictionary<string, Coroutine>();

    public int LinesProcessed {  get { return _linesProcessed; } }

    public void ProcessFile(string pathToFile)
    {
        if (_currentCoroutines.ContainsKey(pathToFile))
        {
            throw new System.Exception("Already reading file: " + pathToFile);
        }
        else
        {
            _currentCoroutines.Add(pathToFile, StartCoroutine(readFileLines(pathToFile)));
        }
    }

    private IEnumerator readFileLines(string pathToFile)
    {
        using (var data = new StreamReader(pathToFile))
        {
            int count = 0;
            var line = data.ReadLine();
            while (line != null)
            {
                count++;
                _linesProcessed++;

                /// PROCESS THE LINE HERE HELD IN "line"

                if (count == LinesToReadPerFramePerFile)
                {
                    yield return null;
                    count = 0;
                }

                line = data.ReadLine();
            }

            data.Close();
        }

        _currentCoroutines.Remove(pathToFile);
    }
}