4
\$\begingroup\$

Goal

I would like to be able to separate all my data file readers from the main thread and balance the load over multiple frames. The idea being that while the data is being read/parsed/filtered I can continue playing the game at a decent FPS and show a loading progress bar while the results of the file importing pop in one at a time when they are ready.

Information

The data files contain things like transform information for objects, points to be used in runtime mesh generation, and terrain data. They all have to be parsed, filtered, and/or modified to be ready to be used in the scene.

I already have import code for the project I just need to transplant it from blocking my main thread and being done in one frame to a multi-threaded and multi-frame approach.

Thoughts

I have little experience in multi-threading so this is also a learning project for me. I would appreciate all your comments, input, code samples, and pointers.

From my perspective I need to create a queue of tasks. Each task being a file to load, read, parse, and filter. Then if a CPU is open a task is taken from the queue and is given a thread to be executed on that CPU. The task completes and gives the resulting data back to Unity3D for it to handle.

Questions

  1. Does a heavy load on other CPUs freeze the game like it would if it were on the main thread, and thus require some sort of system where you yield for one frame if execution time is taking too long?
  2. How do I receive progress updates from the tasks to be used in a interface progress bar via some sort of co-routine?
  3. If a user only has one CPU how would this system handle the situation? Is that even a realistic scenario?
  4. What sort of C# classes (usable in Mono's subset of the .NET runtime) already exist for multi-threading and task queuing that can save me time?
\$\endgroup\$
  • \$\begingroup\$ How big are the files? How many are there? What format are they in? Is there some hierarchy to the data, such as JSON or XML, or are they flat, line-by-line files. \$\endgroup\$ – PatrickSharbaugh Nov 6 '15 at 19:03
  • \$\begingroup\$ I already have the file IO section of the program complete. Point files used for mesh generations are multiple kilobyte binary files that just list points line by line and there could potentially be up to a hundred. Transform data files are a couple kilobytes of plain text line by line and usually only 1 is loaded at a time. Terrain Data files could be far larger, one at a time, and I'm still looking into optimal formats. \$\endgroup\$ – Saevax Nov 6 '15 at 19:20
  • \$\begingroup\$ If it is just line by line data, I recommend processing a certain number of lines each frame by using a coroutine. \$\endgroup\$ – PatrickSharbaugh Nov 6 '15 at 19:44
1
\$\begingroup\$

Here is some example code to expand on my comment above. A true multi threaded solution would be faster than this, but susceptible to many more problems and edge cases. Unity's model for concurrency is coroutines, and thus using them aligns your code with what is expected in the unity space. Unless you have a real need for ultra fast file processing, this solution should work.

using UnityEngine;
using System.IO;
using System.Collections;
using System.Collections.Generic;

public class FileImporter : MonoBehaviour
{
    public int LinesToReadPerFramePerFile = 100;

    private int _linesProcessed = 0;
    private Dictionary<string, Coroutine> _currentCoroutines = new Dictionary<string, Coroutine>();

    public int LinesProcessed {  get { return _linesProcessed; } }

    public void ProcessFile(string pathToFile)
    {
        if (_currentCoroutines.ContainsKey(pathToFile))
        {
            throw new System.Exception("Already reading file: " + pathToFile);
        }
        else
        {
            _currentCoroutines.Add(pathToFile, StartCoroutine(readFileLines(pathToFile)));
        }
    }

    private IEnumerator readFileLines(string pathToFile)
    {
        using (var data = new StreamReader(pathToFile))
        {
            int count = 0;
            var line = data.ReadLine();
            while (line != null)
            {
                count++;
                _linesProcessed++;

                /// PROCESS THE LINE HERE HELD IN "line"

                if (count == LinesToReadPerFramePerFile)
                {
                    yield return null;
                    count = 0;
                }

                line = data.ReadLine();
            }

            data.Close();
        }

        _currentCoroutines.Remove(pathToFile);
    }
}
\$\endgroup\$
  • \$\begingroup\$ Thank you for taking the time to write up a solution but it's precisely those problems and edge cases from multi threading I want to learn. \$\endgroup\$ – Saevax Nov 7 '15 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.