I have encountered an issue I would like your input on. While attempting to make the RTSS hardware skinning (HS) feature work for both cg and hlsl I have encountered a problem. The HS sub-render state requires 3x4 world matrices. cg and glsl use a row major representation for their matrices while hlsl uses a column major representation.
As far as I can see this can be solved in 3 ways:
- During the construction of the HS sub-render state, check if the current target language is hlsl. If so construct the shader with 4x4 matrices instead of 3x4 matrices - This is currently how I implemented it.
- When using glsl set the shader in the RTSS to always use row major representation - I don't think this is a good Idea. I don't think I can blindingly set all glsl in the RTSS to use row major representation. Some shader may rely on the exact opposite representation. Plus this also has a small effect on the framerate.
- Create a new option in RTSS parameters. When this option will be turned on a parameter receiving an X by Y matrix will declare himself to receive an Y by X matrix. All that is need to do then is to change the multiplication operation in the shaders. Instead of multiplying [matrix 3 by 4] * [vector 4] we will multiply [vector 4] * [matrix 4 by 3].