1
\$\begingroup\$

I'm building an Android app using Unity3D where users can visualize data immersively.

The data is immutable and stored using a MySQL database hosted on cPanel.

When the user clicks a button in the environment, it starts the creation of a chart. First, contacts a .php file present in the server responsible to query the database, fetching the data and make it available to the Unity controller which then adds the points to the chart. This works fine for small quantities of data.

To increase performance, thought of exporting from MySQL as CSV and transform to one JSON, instead of querying a database or try to make MySQL transform to JSON, but I'm afraid this won't be enough.

Could also extract as CSV and use a 3rd party for reading it.

How would you load the data in such case to have the best performance?

\$\endgroup\$
  • 1
    \$\begingroup\$ Why the convert from CSV to JSON? They're both just data formats, converting from one to the other just wastes CPU cycles. \$\endgroup\$ – Draco18s Oct 3 at 14:06
  • 1
    \$\begingroup\$ Have you considered pre-digesting the data into a more compact binary format that doesn't require the parsing load of CSV/JSON? As a bonus, this can also save you bandwidth. \$\endgroup\$ – DMGregory Oct 3 at 16:34
  • \$\begingroup\$ Not until you said so. I don't really need the DB and query it, just using it to join many .csv files together and remove the header. So, could extract the data as one big .csv and then forget the DB (unless there's many files, in that case store in the DB the filepath) and convert from .csv to .bytes. Then, because the data is immutable, store it in the Project Folder and read directly from there and add to the chart. \$\endgroup\$ – Tiago supports GoFundMonica Oct 3 at 17:46
  • \$\begingroup\$ How much data are you talking about here? Kilobytes? Megabytes? Gigabytes? Terabytes? \$\endgroup\$ – Philipp Oct 4 at 14:45
  • \$\begingroup\$ @Philipp Max 5 GBs \$\endgroup\$ – Tiago supports GoFundMonica Oct 4 at 16:48
2
\$\begingroup\$

TL;DR

Use metrics to find what's a performance problem, don't just guess what it is.

TL

As you're accessing it via a web interface, the datastore at the back end is irrelevant. The performance considerations for you concern the transport of the data to your client and then the parsing of the data into workable data structures (i.e. C# classes).

If you have control of the format you receive the data in, and it's tabular, then CSV will smaller than JSON. If not, then you have to work with what you're given.

You said you can parse from CSV into JSON, which means you already have a working CSV parser and so have no need for the JSON.

The performance hits now are to do with processing power (to read the CSV) and memory (is it big enough to hold the dataset you want to work with?

If those are fine then you need to consider the number crunching and searching. To find the data (in memory) that you're interested in you need to use a nicely indexed data structure. You might find that the System.Data.DataTable is best, it's old and can cause allot of GC collections but it may be good enough for your MVP. If that doesn't work then you need to write a bespoke structure.

You may also was to calculate some aggregations and store those so that they're not calculated on the fly (you might even have these delivered to you by the call to your PHP file, the server & DB are far more capable of doing this that a mobile device), as well as store aggregations the user has requested ready for when they ask for them again.

Long answer, but you question isn't a simple one to answer. Whatever you decide to do, you must gather metrics in order to find out what is really the performance problem rather than what you think is the performance problem.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.