Design question: best way to model and visualize a long grid of data [closed]

Question

Perhaps some noob questions for a scientific application that you can assist with: I am trying to render a large, evenly spaced grid of data in 2D. The full data set dimensions could be anywhere from 48x250,000 to 1000x15,000,000 points/pixels.

The data is evenly spaced, but I need to change the aspect ratio, and then render only a section of that data on the screen at a time. I need the users to be able to smoothly scroll forward or backward along the long dimension with user input, or press the "play" button and have it move forward or backward at certain speed.

I am looking for advice on how to handle this via DirecX or OpenGL (probably using SharpDX or OpenTK).

The basic approach I am considering is to use textures that the application generates on the fly from the data and then map them to a very simple model. I would keep the graphics card supplied with a large enough scene so that as the user scrolls or moves around that they don't have to wait for the application. Questions I have are:

1) does this approach make sense? i am presuming this is way more efficient and faster than creating a bunch of primitives with discrete colors. true?

2) is there a particular texture format that is good to use and easy to create (or do i create it directly)? I have the data in application memory as large arrays and it loads/unloads from disk as needed. I would map the scalar values to a color as I create the image/texture blocks.

3) would mipmaps be useful? i could have an aspect ratio (or small enough window) that there are fewer screen pixels than data in the viewable range. Or, would it just automatically subsample?

4) if mipmaps are used, most of the scaling is along the long dimension, so could i uses non-square subsamples?

4) does creating compressed images make sense? or, would the compression overhead yield much benefit (or how might i determine this)

5) how would i go about finding an optimal texture size?

Any other general advice?

Hopefully this is a straightforward problem for the cool stuff y'all do in games!

Answer 1

1. Yes, this is a fine approach - it will be orders of magnitude faster than creating a primitive per data point.

2. Assuming single-channel data, you should use whatever DXGI_FORMAT (or equivalent in OGL) maps to your source data, and convert in the pixel shader. There are native types for 8, 16, and 32-bit integers, as well as 32-bit floats. You should also only create a few textures up front and then reuse them - don't create them on the fly. Instead, keep track of which ones are off-screen and overwrite the data of e.g. the least-recently-used one.

3. If you allow scaling out significantly, MIP maps will be useful. Be sure to enable 16x anisotropic filtering though, since you'll have a huge uv-delta mismatch basically all the time. 4(first): Anisotropic filtering does precisely this.

4. I wouldn't bother with GPU compressed formats initially. Your bottleneck will almost certainly be disk IO, so if anything I would use something like LZMA to compress the chunks of data on disk.

5. A good size would be to use a height of whatever your data height is (48 - 1000), and use 1024 or 2048 for your width. This should offer a good balance of texture count versus total memory used (and the ability to swap out LRU texture data).

Other tips:

1. You might intially think to create a geometry and camera that you conceptually scan accross it. A much simpler approach is to create a full-screen quad, and just update the texture coordinates on the fly using a cbuffer with scale/offset parameters.

2. Do all your visualization math (color gradients, etc.) in the pixel shader - don't convert on upload unless it's required to fit into one of the hardware native formats (e.g. if you're using a custom number format for some reason). Sub-point: if your data is double-precision, use the R32G32 format, then in the shader use the asdouble(color.r, color.g) instruction to get the original value back. Note that double-precision support is an optional feature though - not all hardware supports it.