What are the pros / cons of the three?
coderanger is right about HLSL targeting DirectX, GLSL targeting OpenGL and CG being available with both interfaces.
However there are other things to consider (learned on the OGRE forum) :
- CG will not allow you to use the latest features of GLSL (I'm not sure about HLSL). It's a middle ground so you'll not be able to fully exploit the GLSL features, only the common one until Shader Model 3.0 (AFAI understood).
- CG is made by NVidia that optimize it for it's card but don't seem to do lot of work for ATI cards... some people report that there might be problems with ATI cards using CG.
- OpenGL seem to be a bit slower than DirectX on Windows platforms. That's really relative to what you're doing and the reason for this find it's source in the drivers implementations but it's good to know before you choose the API and the associated shader language. However, OpenGL is the only one interface available on all (win/mac/unix/some consoles) platforms. So knowing that, using exclusively GLSL might be the better choice for a non-Microsft-centric-cross-platform game.
- NVidia drivers on Linux seem to be more stable than ATI drivers, making CG more interesting if you need to be sure it works nicely on linux (that's all reported informations, you'll have to check for details)
So, if you're not using the lastest shader features, CG seem a good choice. GLSL seem a beter one if you're going full OpenGL. HLSL if you're going exclusively on Microsoft platforms.
Now first developping in HLSL for windows to use DirectX and then convert to GLSL for linux and mac could be the better solution to be sure of performance and have the larger set of shader features available. It might however be a lot of work (didn't do it myself so I cannot tell). OGRE graphics engine (and other engines) allow to use any API (DirectX or OpenGL or others) so it helps, but there is still shader code to convert if you go this way.
That's all the information I gathered while choosing my shader language (I've not made my decision yet).
Update: Valve did a conversion of one of their game to OpenGL and found no way to make the DirectX version faster than the OGL one. So keep in mind that the state of driver implementation, API quality, etc., all that change too much each year for you to totally rely on raw performance as an argument to choose one or the other. With this in mind, choose OpenGL/GLSL to simplify your life when working (or having plans or hopes to work) with other platforms than Windows, use DirectX/HLSL if you really want to use only Microsoft platforms and focus and maybe have some good API faster than OpenGL (this is reversing now, so don't count on it); use CG if you want to provide both possibilities to the user, but if you have the work force (and tools) to do it, using both GLSL and HLSL might be a viable solution too.
Update (2012): It is important to note that CG has been discontinued, and is no longer supported or actively worked on by Nvidia. Nvidia recommends that all users switch to a combination of GLSL and HLSL, or a newer library such as nvFX (on github). This is because it was too difficult to maintain feature-compatibility between GLSL and HLSL.
I can only talk about CG vs HLSL because those are the 2 I have used so far.
Cg is not the same as HLSL.
In Cg, NVIDIA did an excellent job in creating a very clean shader syntax. Its very similar to HLSL.
But, tie-together with D3D9/D3D11 (init code, shader compilation code) is much cleaner on HLSL than Cg. -1 Cg. Cg has a nasty bit of start up code that you don't even need to have for HLSL over D3D.
In Cg, you have to "cgGetNamedParameter" for every
uniform shader variable you want to set/modify. And you have to maintain a
CGparameter reference in your code for that variable
// C++ code to interact with Cg shader variable (shader language independent) CGparameter mvp = cgGetNamedParameter( vs, "modelViewProj" ); CG::getLastError("Getting modelViewProj parameter"); // check for errors cgSetMatrixParameterdr( mvp, &modelViewProj._11 ) ; // setting the matrix values
In HLSL, this ends up being much cleaner - just one line, and you don't have to maintain that
// D3D9 C++ code to interact with HLSL shader variable DX_CHECK( id3dxEffect->SetMatrix("modelViewProj", &mvp._11 ), "Set matrix" ) ;
In the above,
DX_CHECK is just a simple function that checks the HRESULT that gets returned from the
SetMatrix call. The above code is d3d9. D3D10 and 11, are of course, a lot more painful (since there isn't an ID3DX11Effect object).
Before I started using HLSL, I used to look at this code and actually get jealous.
Although NVIDIA did their best to make a common interface for Cg between OpenGL/D3D, practically speaking its not that way, and you have
cgD3D11 function groups to contend with. So that whole works for OpenGL and D3D!! claim only goes so far. You still have to wrap everything up into OpenGL/D3D
#ifdef type groups to make it work on different platforms. -2 Cg.
Further I have recently had a bad experience with Cg/ATI cards, which I'm pretty sure isn't my bad. (Someone else try it out?). I think it may be true that NVIDIA doesn't completely test ATI cards, as Klaim claims. Or that ATI doesn't test on Cg. One way or the other, there's a mismatch there and some sort of conflict of interests. -3 Cg.
All in all I preferred Cg. Its shader code syntax and naming is clean, sweet, and tidy. Its too bad its got these other problems.
Another crucial difference between HLSL and GLSL (I don't know CG so I can't speak for it) is that with HLSL Microsoft provide the shader compiler as part of the D3D runtime whereas with GLSL your hardware vendor provides it as part of their driver.
This has advantages and disadvantages on both sides.
With the GLSL method the vendor can tune the compiler to the capabilities of their hardware. They know their own hardware best, they know what to do and what not to do. On the other hand the disadvantage is that - in a world where there are multiple hardware vendors - you have a situationwhere there can be inconsistencies between shader compilers, and the vendor also has complete free reign to screw up.
With the HLSL method Microsoft control the compiler. Everyone is on a consistent tech base, and if a shader compiles successfully in one place then it can be reasonably supposed to compile everywhere. The same shader will produce the same compiled output irrespective of vendor (all other things being equal of course). On the other hand the HLSL compiler has to be a "works consistently on everything" thing, so it's not able to pull any special vendor-specific tricks to get the last few drops of juice out of the tank.
If this comes across as though I have a preference for the HLSL view of the world it's because I do. I've been bitten badly before by wildly inconsistent behaviour on different platforms, and in one case even ended up having to roll a load of GLSL back to ARB ASM just to get a baseline that worked. NVIDIA's GLSL compiler can be viewed as particularly notorious here - it will even accept HLSL syntax and keywords, meaning that if you're not careful you can end up producing shaders that will only work on NVIDIA and nothing else. It is a jungle out there.
You might want to look into the RTSS (Run Time Shader System) that comes with Ogre as well. It is fairly new, but you basically write shaders in code rather than external files. I have not implemented it yet, but definitely plan to use this when the time comes.
Here is a huge series of tutorials on the Ogre wiki as well for writing shaders. http://www.ogre3d.org/tikiwiki/JaJDoo+Shader+Guide
As for your original question, like Praetor said, its not really a matter of pros/cons, it is a matter of what rendering system you want are going to use. Since using DX/OpenGL with Ogre is just a matter of loading the plugin, you will most likely want to use .cg format.