Motivation
Before I start, let me give you a little background info. I'm a hobby game developer and I find it particularly exciting to write shaders. More specifically, I've been using Ogre's material system for writing shaders for games since about a year.
Now, Ogre has a nice material system and I'm not criticizing it at all. For what it can do, it does a great job. However, I've experienced that the more complex your shaders and materials get, Ogre's material system alone simply doesn't fit your needs - while it gives you nice rendersystem-independent access to features, I found that if you use a complex setup of shaders, it's just not high-level enough. In this section, I'm going to explain some of the reasons behind this, and share the experiences that I've made coding shaders for Ogre.
Poor Choice of High-Level Languages
This section is mostly based on my own observations and as us such might be slightly biased. For further reading, you might be interested in reading this StackOverflow discussion.
The most popular languages for writing shaders are HLSL, GLSL and CG. They have basically the same features, but the syntax is slightly different. So which one do you choose?
HLSL is only available for DirectX rendersystems, and GLSL is only available for OpenGL rendersystems.
CG claims to be "cross-platform" because it supports both DirectX and OpenGL, but if you have worked with CG for a while, you will see that it's not really portable in reality:
- CG is poorly supported on ATI cards, especially in OpenGL mode (This is not really surprising, since CG is provided by NVIDIA and no one can blame them for "advertising" their own cards). To my knowledge, there's not even an ATI/OpenGL CG profile that supports more than 32 constant registers.
- CG Runtime is very slow. In my profiling tests, compiling a shader with CG took about 20 times longer than the same shader in GLSL.
- CG Runtime is closed source, and NVIDIA doesn't provide a version for FreeBSD. So it's not a good choice for open source games.
If you want a decent amount of portability for your shaders, your best choice seems to be using both HLSL and GLSL by writing the same shader source code twice and maintaining 2 different code paths. Yes, there really are real-world libraries that actually do such a thing. Look at the SkyX / Hydrax sources for example. If that fact doesn't indicate a poor choice of languages, then I don't know.
Permutations and non-predictability
Thomas Grip from Frictional Games has written a very nice article about this problem, i highly recommend reading it. Here's a small extract:
In OGRE, you can use defines ( #ifdef MATERIAL_HAS_SPECULAR and so on ) to use the same shader source code for a set of shaders. However, you still have to add a "vertex_program" / "fragment_program" set for each combination there is (or you can simply create the actual vertex/fragment programs in c++ code, but then you lose the ability of setting auto constants through material scripts and many other neat features).In HPL1 there I used a bunch of shader files, one for each specific effect I wanted to do. No specular - new shader, new light type - new shader and so on. Even though HPL1 does not have that many shader combinations I ended up with 10 or so shaders that did about the same thing with various smaller changes. This was not so hard at the start, but when I wanted to optimize it was very frustrating to have to update 10 files all the time and some optimization became a bit hackish or simply left out. There where also times when I wanted to add even more shaders to optimize for specific hardware but I just didn't have the energy to manage even more shaders, so I skipped it.
For this reason, as soon as lots of shader permutations come into play, many OGRE users/developers are simply assembling their shaders completely in C++ code (this is done in Hydrax and Ogre::Terrain for example). I have done so as well, but found it really annoying to code shaders this way for a couple of (obvious) reasons.
- After every shader code change, no matter how small it is, I have to recompile my application. This is especially annoying because shaders can be very hard to debug, and for me this has resulted in a huge loss of productivity.
- The shader as a whole becomes obfuscated and hard to look at.
- There is no syntax highlighting.
Another point to consider is that if your game has a system for user-created mods, your users won't be able to create any custom shaders since you have essentially hard-coded your shaders.
Okay, so we can agree at this point that assembling shaders in code is bad. So what are the alternatives? At this point, there don't seem to be any decent ones, and that's why lots of people are still using this approach, unfortunately.
Managing user settings
This is one another example where Ogre's material system alone simply doesn't fulfill my needs. If you look at most games, there's a (from my experience, quite high) probability that all they offer the user to choose from in terms of graphics settings are presets like Low, Medium, Ultra. But from a user's standpoint, there is no reason why it shouldn't be possible to enable or disable each shader feature individually. So why do many game developers choose this approach? My guess is that they are probably using a material system similiar to Ogre's. They probably add 3 techniques for all their materials, one for low, medium and high, respectively. This is a really bad choice because it creates lots of duplication and redundancy in your material scripts, since the 3 techniques will usually do the same thing with minor differences.
The main point here is that a Material simply isn't the way to go as an interface for content creators. It's desirable to abstract away all the things that an artist (or material designer) should not have to worry about at all - in this case, that is, managing all different combination of graphics settings that exist for the game. I'm proposing to add a more high-level class than a material. The only thing that this class should have and know about, is a simple set of properties that allow a designated Factory class to automatically create a material with several techniques, passes and texture units, because that's nothing that a content creator should worry about.
User settings are not even the only reason for this. Even if your game doesn't have configuration options at all, it can still be necessary to create several techniques that do the same thing with minor differences - for example for using a simpler version of the shaders when rendering a top-down ingame minimap. Unless that minimap is fully dynamic, you wouldn't want specular lighting, etc, etc.
What now?
I sat down for quite a while and thought about ways to make this process easier, both for content creators and shader developers. To be able to solve the problems that were pointed out, I figured I'd need a utility library that acts as a layer on top of Ogre's material system, with at least the following features:
- The ability to create even very complex shaders (such as the Ogre::Terrain shader, which has an arbitrary number of layer calculations) outside of the application itself, either through a scripting language or by extending regular shader files with a set of macros (such as a foreach macro).
- An abstracted interface (both C++-interface and serialization interface, i.e. material scripts) for content creators, in order to automate the unintuitive and repetitive process of creating materials.
- This interface should not be referring to specific shaders with a deterministic source, but instead to a "base shader", and the source/behaviour of this shader can be altered by per-material properties as well as user settings.
- Permutation management for base shaders, automatically compiling new/altered versions of the shader as needed and sharing the same shader between several materials when possible.
- A reliable way of setting auto constants and uniforms outside of the application itself, either through a scripting language or by using macros parsed by the shader factory.
- Being able to bind abstract material properties to uniforms in the shader.
- Automate the conversion between shader languages, or provide a meta-language on top of GLSL/HLSL/CG. This should be optional, because some people might still prefer writing only in one language.
Of course we have RTShaderSystem, but it doesn't even fulfill the first requirement - if you want to add custom shaders to it, you will have to code them in C++.
So, in the past weeks, I've worked on a library that covers all this and much more:
The result
I believe that I've reached a point where it matured enough that it can be useful for others. Here are the current features:
- High-level layer on top of OGRE's material system. It allows you to generate multiple techniques for all your materials from a set of high-level per-material properties.
- Several available Macros in shader source files. Just a few examples of the possibilities: binding OGRE auto constants, binding uniforms to material properties, foreach loops (repeat shader source a given number of times), retrieving per-material properties in an #if condition, automatic packing for vertex to fragment passthroughs. These macros allow you to generate even very complex shaders (for example the Ogre::Terrain shader) without assembling them in C++ code.
- Integrated preprocessor (no, I didn't reinvent the wheel, I used boost::wave which turned out to be an excellent choice) that allows me to blend out macros that shouldn't be in use because e.g. the shader permutation doesn't need this specific feature.
- User settings integration. They can be set by a C++ interface and retrieved through a macro in shader files.
- Automatic handling of shader permutations, i.e. shaders are shared between materials in a smart way.
- An optional "meta-language" (well, actually it's just a small header with some conditional defines) that you may use to compile the same shader source for different target languages. If you don't like it, you can still code in GLSL / CG etc separately. You can also switch between the languages at runtime.
- On-demand material and shader creation. It uses Ogre's material listener to compile the shaders as soon as they are needed for rendering, and not earlier.
- Shader changes are fully dynamic and real-time. Changing a user setting will recompile all shaders affected by this setting when they are next needed.
- Serialization system that extends Ogre's material script system, it uses Ogre's script parser, but also adds some additional properties that are not available in Ogre's material system.
- A concept called "Configuration" allowing you to create a different set of your shaders, doing the same thing except for some minor differences: the properties that are overridden by the active configuration. Possible uses for this are using simpler shaders (no shadows, no fog etc) when rendering for example realtime reflections or a minimap. You can easily switch between configurations by changing the active Ogre material scheme (for example on a viewport level).
- Fixed function support. You can globally enable or disable shaders at any time, and for texture units you can specify if they're only needed for the shader-based path (e.g. normal maps) or if they should also be created in the fixed function path.
The current source can be found here: https://github.com/OGRECave/shiny
The doxygen-generated documentation along with a hand-written manual can be found here: https://ogrecave.github.io/shiny/
I might add some more in-depth usage examples at a later point. For now, taking a look here should definitely get you started.
Future work
- Geometry shader support (shouldn't be hard to add, I never used them though)
- More documentation.
- Add samples, or a demo.
- Material editor, maybe?
- Loading time optimizations (for example caching the preprocessed files - some early profiling tests have shown that preprocessing sometimes takes longer than compiling the actual shaders) DONE
- Saving generated shaders / materials as regular .material / .cg / .glsl files
Evaluation & showcase
Okay, you got me, this is also a showcase thread so I need screenshots
"Meh, this looks just like Ogre's standard terrain shader / material. What's special about it?"
Yes, this shader has the same features as Ogre's standard terrain shader (TerrainMaterialGeneratorA). But I have rewritten it to use shiny, and I'm quite happy with how it turned out. In short, it's now more efficient, portable, flexible, and maintainable:
- The shader is an external file and can be changed without recompiling the application.
- Efficiency. the standard TerrainMaterialGeneratorA implementation takes more than 2000 lines of c++ code. My implementation takes only 270 lines of c++ code coupled with a 370 lines shader file. Oops.. that's 300% smaller
Okay, I cheated a bit because it doesn't support composite or normal maps. However, it would still be significantly smaller. Oh, besides it supports multiple lights while TerrainMaterialGeneratorA doesn't.- The shader compiles in both CG and GLSL, depending on the target platform or user preference. TerrainMaterialGeneratorA only supports CG.
- Generating additional techniques with smaller changes works with ease. As you can see on the screenshot, the terrain on the minimap does not have shadows or directional lighting. The only thing required to achieve that is registering a Configuration as follows and then changing the viewport material scheme to minimap:
Code: Select all
configuration minimap
{
fog false
mrt_output false
lighting false
shadows false
}Here's a screenshot of a more advanced shader, proving that writing in a meta-language does not really limit you in any way as opposed to writing in GLSL/CG/...
), I will remove it when it appears.