Stencil Shadows

[dark_sylinc]

So, while I'm still working on getting instancing back online; I'm wondering about shadows next.

Right now Ogre supports three kinds of shadow mapping (forgetting about the modulative-additive variations):

• Stencil shadows (aka. shadow volumes)
• Shadow maps through Fixed Function
• Shadow maps through shaders

Of the above three, the last one is the predominant method in the industry. It's the easiest to maintain, fastest and the most scalable.
Shadow Maps through FF are just crap. And Stencil shadows... well they're a major pita to maintain.

I'm bringing this up because I'm cleaning the code with my GSoC 2013 and I'm afraid stencil shadows may not be working in it's current state.
Furthermore:

• They take a lot of work
• To my knowledge, this technique is ancient and rarely used.
• My knowledge about stencil shadows is rusty. I wasn't very familiar with its inner working to begin with

Everyone's life would be easier if we could just get rid of them, and it could improve performance too, as there is some nasty code in the render queue devoted to the different paths needed for each shadows technique (except for shader-based stencil map which naturally fit on the default rendering sequence); because each technique requires object to be renderent different rendering order.

Before rushing, let's take at the features:
Stencil Shadows
+ When using FF, for the user turning stencil shadows "just works" (turn them on -> it's already working)
+ They're mathematically stable (very few artifacts, no aliasing)
+ Shadow quality is fixed
- Only hard edges. No soft shadows
- Give a lot of work to maintain engine side; they don't fit flawlessly with normal rendering sequences or advanced compositing work
- Mesh preprocessing is required (though Ogre does a good job on this side and doesn't require maintenance)
- They're slower (on most cases)

Shadow maps (shader based)
+ They're faster (on most cases)
+ Soft shadows, customizable filters.
+ Naturally fit with GPU's normal rendering sequences and compositors.
- Shadow quality depends on light/receiver/camera direction & position
- Artifacts: Aliasing, Shadow Acne, Peter panning.
- Requires the user to have some knowledge of shaders (can be alleviated if stuff like RTSS proves to be successful. Furthermore all APIs are moving to shader-only rendering)

So, long story short, I want to code knowing that if I have to, stencil shadows support can be axed. They're old, hard to support, look fake because of their hard edge, are slower, and rarely used. And they don't fit in an engine using shaders.
Chances are, if you really benefit from Stencil Shadows, you're targeting older machine specs where Ogre 1.x performs fine (not to mention, it could run faster since Ogre 2.x makes assumptions about the hardware, for instance mandatory SSE2 support for x86 builds; also benchmark and tuning is done using latest hw. Cache line sizes & branch predictors changes, single core chips, all can affect the performance between 1.x & 2.x; Ogre 1.x was designed with Pentium III HW in mind)

If a breakthrough causes stencil shadows to be all the rage again, we'll see how to port the 1.x code into 2.x again. But for the foreseeable future, it looks like stencil shadows are dead (there is some stuff I will leave there, for example quite a few users have reported Mesh' edge lists are very useful for their own algorithms, unrelated to shadows; so they're staying)

I would like to know what you think.

[Xavyiy]

This is something I've been thinking since a couple of years, why still supporting stencil shadows if them add a big code overhead and -almost- nobody uses it? (at least in big projects)

In my opinion, if there is a good moment to stop supporting stencil shadows, it's right now.

[masterfalcon]

I would think it would be ok to remove them. They don't work on mobile either unless we emulate it with a texture. Someone wrote an article about using that old technique in Ogre on Gamasutra once.

I also think that we need to provide better built in shadow techniques. Just see how cumbersome is it to actually get depthmapped shadows working right now.

[Zonder]

Drop them

I am with masterfalcon ogre should provide more inbuilt modern techniques.

I also think it's fair to say OGRE 1.x is old hardware based 2.x is new hardware based.

This does mean patching and providing 1.9 as an official old hardware build imo (the maintenance shouldn't be high anyway once stable).

[phobossion]

Agree. As discussed elsewhere, Ogre should move towards modern techniques like shader-based rendering, deferred pipeline etc, dropping some legacy techniques on the way is a necessity IMO.

[Wolfmanfx]

Last time checked the code I asked my self if its worth all the branches inside the scenemanger - so i am also for dropping it because the scenemanager code will get simpler.

[holocronweaver]

patching and providing 1.9 as an official old hardware build

I agree with Zonder. By providing maintenance patches to 1.9, anyone targeting older hardware should be satisfied.

[TheSHEEEP]

Yes, getting rid of them is a good idea.
We need to lose some weight

[Grognard]

I would think it would be ok to remove them. They don't work on mobile either unless we emulate it with a texture. Someone wrote an article about using that old technique in Ogre on Gamasutra once.

I also think that we need to provide better built in shadow techniques. Just see how cumbersome is it to actually get depthmapped shadows working right now.

Yeah, I don't see why you'd need to do any extra work just to make shadows, kind of weird.

[mkultra333]

I've always wondered why Ogre supports any shadows at all. I mean, it's just RTT and shaders. I use VSM shadows and I don't touch any inbuilt Ogre shadow code at all. I always found Ogre's shadowing system confusing and hard to customize. It was much easier and more versatile to learn RTT and shader programming.

As for stencil shadows, I don't use them or need them. Poor little things though, I remember back in 2004 when they were the new hottness.

[Zonder]

I've always wondered why Ogre supports any shadows at all. I mean, it's just RTT and shaders. I use VSM shadows and I don't touch any inbuilt Ogre shadow code at all. I always found Ogre's shadowing system confusing and hard to customize. It was much easier and more versatile to learn RTT and shader programming.

I disagree shadows are such a standard thing I would expect a rendering engine to provide them.

[Xavyiy]

I agree with Zonder. It's true that it's possible to implement the same shadows provided by Ogre but it's not a trivial task, specially for unexperienced people (the maths behind Focused, Plane optimal, LiSPSM, etc, are not easy and it's expected to be provided by the render engine).

Some time ago I implemented Cascaded Shadows in my ogre-based engine (so, basically add a ShadowCameraSetup and write custom shaders), and while the final code isn't that complex the implementation process wasn't that easy.

Also, some ogre shadow-related params are very useful, like shadow_caster_material / shadow_receiver_material in material techniques, MovableObject::setCastShadows, etc... Yes, it's true that you can do all these checks by hand using regular RTTs and custom code, but it's a pain in the ass.

[mkultra333]

I didn't really expect to people to agree, so I'm not really arguing for Ogre to be changed in this way. It's just how it looks to me.

I wasn't saying Ogre shouldn't provide shadow sample code. When I first started using Ogre (version 1.6), the only sample code was the shadows demo where you could switch between different shadows on the Athena model. I could not understand it at all, and it was all the worse for being every type of shadow Ogre could make all crammed together into one confusing application. What would really have helped a beginner like me would have been simple, basic examples of each type in their own unique projects. I very nearly gave up.

Fortunately I found Nullsquareds softshadow VSM sample, which was exactly that. It did, however, still use some inbuilt Ogre shadow stuff. At the time, this meant that for every shadow casting spotlight you needed to create another square texture, instead of being able to reuse the one texture. That, plus some other customizations I wanted to add, meant it made more sense to redo it all with RTT so I could just reuse the same texture for mulitple spotlights. I've never looked back since, and always been extremely wary of Ogres in-built shadows ever since.

So yes, I would fully expect Ogre to provide working sample projects for all the different types of shadows, LiSPSM, VSM, PSSM, etc. But personally I'd prefer the workings were simply laid out in terms of RTT, shaders, and rendering orders, instead of first having to decipher Ogre's take on how to set up shadows and then hope I didn't need to do any customization that simply wouldn't work.

I feel similarly about the compositor system. In the time it takes to learn the scripting needed, and get it working within a setup, I can just as easily learn RTT and how to render a fullscreen quad. (Well, actually a little longer... it took weeks of whining on the forums before someone finally gave me any info on how to set up Multiple Render Targets using code instead of compositors.)

To me, there are the complex things I expect Ogre to do (handling resources, setting up draw surfaces, compiling shaders, updating animations and bones, mediating between my program and data on the graphics card, and maybe stuff like the terrain manager, as well as providing excellent math functions), and then there's the things which are really just RTT and shaders (Shadows and compositors). I love the former, but I've never been a fan of the later and ended up not using it.

Maybe I'll get some hate for saying that, but hey, just my two cents.

[PhilipLB]

Would it be possible to refactor the Stencil Shadows into a separate component? Or is this a complete nightmare because it's so tightly integrated?

[drwbns]

I think Ogre heading towards optional components is the future anyways. But, as Phillip said, is this possible to do?

[phobossion]

Not a core Ogre developer myself, but generally speaking anything related to low-level rendering techniques is very hard to do in a modular fashion and bears along a hit in performance (the branching wolfmanfx mentioned). The Gamainstitute's Carbon engine does this pretty well actually, by abstracting rendering pipeline into a script API so that they can later script deferred pipeline, forward pipeline, add shadow passes etc. But this is a pretty advanced solution and it depends on the general direction Ogre wants to take:

- if you guys want to keep is as a "general purpose rendering tool" (for whatever reason) then maybe an approach similar to the scripted pipeline would work best because it is nearly infinitely flexible
- if you want to create more of a plug'n'play renderer for games & visualization, then maybe more fixed and less configurable solutions are preferable (forward & deferred pipelines with out-of-the-box shadow support for example, with access to used shaders)

This is a whole another discussion, but look at the direction the industry is taking nowadays (scientific applications using Unity, advertised as a game engine) for a hint.

[dark_sylinc]

Oh!! I'm very happy a lot of people is agreeing on removing them.

To mkultra333 (btw. I read your name as "Mortal Kombat, Nintendo 64 Three Three Three" your profile pic goes along with that!):
Like Xavy said, it's not just rtt and shaders. If you're a beginner trying to understand how the h@!# shadow mapping works, then yes, the answer is "it's just rtt and shaders" and everything snaps. But behind the doors there's a lot of math (particularly LiPSM calculating the intersection body B, skewing the projection matrixes, etc) just to fight artifacts. For spot lights is very easy to think as rtt and shaders because by nature spot lights have a defined position, direction, and a fov, just like cameras (that's why horror games with lamps as main light source in dark rooms look so damn good). However directional lights... they're supposed to be placed at an infinitely far position. And their "projection" is supposedly orthogonal. Where and how do we place the camera for directional lights? There's a lot of math involved to calculate the "optimal" position. And ShadowCameraSetup::getShadowCamera tries to achieve exactly that.

I share your frustration with the default shadow sample. It's a smashup of shadow techniques and the user is supposed to know which one is best. While a veteran would tell you just go for integrated shadow maps, the others suck. That's partially why I'm cleaning this up.

I also share some of the frustration of setting up shadow mapping. It's hard, requires advanced shader knowledge, doesn't always work on the first try (though once you get it working, it keeps working) and isn't very flexible (shadow map atlases aren't supported, reflective shadow maps aren't either, nor native depth buffer textures)

I'm thinking of a solution to put an end to most of this problems, but it will still require advanced shader knowledge. However we have to change the demos so most of them are shader based and use depth shadow maps as default. Hopefully with not just an example of how to use them, but rather actually using them in every other demo, there is enough info on how to get them right.

Unfortunately the only way to fix that roadblock is by providing a material system that allows modular shader creation (like RTSS, but somehow, rtss doesn't feel "right") like UDK's node editor or Unity materials; so that the user puts together the stuff he wants to use, and the system compiles a custom shader for you (also called ubershaders).

Another way I can see to mitigate the problem is to offer several next gen material implementations for drag 'n drop (blinn, blinn phong, cook torrance, oren nayer, etc) so beginners can use use them directly (assign a texture and run it) and advanced users can copy paste the shader code and customize the rest.
Btw. our current depth shadow mapping code is an example of how not to do things. It contains a double nested loop for PCF tapping for "flexibility" in case you want to change the number of taps (from 2x2, to 3x3, 4x4, 3x4, etc). If you benchmark it (or see the assembly) the compiler is terribly bad at optimizing that loop.
The code we provide should be optimized (unless it's strictly for learning purposes, but should be tagged as such!).

Would it be possible to refactor the Stencil Shadows into a separate component? Or is this a complete nightmare because it's so tightly integrated?

May be, but it's really a nightmare. Shadow volumes have to be inserted between opaque objects and transparent, in case of additive stencil shadow, each light has to be done in each passes; ambient lighting must be performed in the first pass. And I'm sure I'm forgetting something. It's messy really. Would be quite a challenge to make modular (I don't believe it's imposible though) and very few people would be interested in seeing it working.

[TheSHEEEP]

I don't agree that a better solution would be an RTSS like system.

Things like RTSS heavily fragment the whole pipeline so you have no idea where what is defined. Trying to figure out where somethign in a specific effect comes from is a nightmare in any RTSS system, and it gets much worse when the parts are defined in the C++ code.

It makes learning shaders much harder than it needs to be. And Ogre also lives from being easy to use, so I'd always see RTSS as purely optional. Or we'd need tools like mental mill or the UDK node tool. But that will never happen

[sparkprime]

Relevant: http://www.ogre3d.org/forums/viewtopic. ... 86&start=0

edit: in particular, the dark sylinc of 2010's comments on the subject of stencil shadows obsolescence

[dark_sylinc]

Nobody resists an archive, eh?

And I wondered myself what was my opinion a few years ago Thanks for the refresher!
I had to admit I thought stencil shadows were supported in phones back then while shadow maps weren't, but I was wrong. As time passes by, even less and less people use stencil shadows. Furthermore I didn't realize how pain in the neck they were to refactor.

I still stand on the software skinning though. Software skinning can be a win when very gpu bound (or the cpu & gpu share the same resources, such as SoC or crappy integrated gpus paired with ultra powerful cpus); plus some very advanced users want more than 4 weights per vertex, and this hard to achieve in the gpu.

Regarding your topic on render queue, I believe we still need some sorting, but we still need to fix a lot of redundant api state changes, and the RQ is screaming for a refactor.

[drwbns]

I share your frustration with the default shadow sample. It's a smashup of shadow techniques and the user is supposed to know which one is best. While a veteran would tell you just go for integrated shadow maps, the others suck. That's partially why I'm cleaning this up.

I also share some of the frustration of setting up shadow mapping. It's hard, requires advanced shader knowledge, doesn't always work on the first try (though once you get it working, it keeps working) and isn't very flexible (shadow map atlases aren't supported, reflective shadow maps aren't either, nor native depth buffer textures)

I'm thinking of a solution to put an end to most of this problems, but it will still require advanced shader knowledge. However we have to change the demos so most of them are shader based and use depth shadow maps as default. Hopefully with not just an example of how to use them, but rather actually using them in every other demo, there is enough info on how to get them right.

Unfortunately the only way to fix that roadblock is by providing a material system that allows modular shader creation (like RTSS, but somehow, rtss doesn't feel "right") like UDK's node editor or Unity materials; so that the user puts together the stuff he wants to use, and the system compiles a custom shader for you (also called ubershaders).

Another way I can see to mitigate the problem is to offer several next gen material implementations for drag 'n drop (blinn, blinn phong, cook torrance, oren nayer, etc) so beginners can use use them directly (assign a texture and run it) and advanced users can copy paste the shader code and customize the rest.

A node-based editor would be awesome. Would probably make a great GSOC

[sparkprime]

Forget node editors, they are absolutely absurd. What you want is a high level shader language that allows specification of the same behaviour as a node-based language but in a few lines instead of a tangled screen full of boxes.

[dark_sylinc]

Forget node editors, they are absolutely absurd. What you want is a high level shader language that allows specification of the same behaviour as a node-based language but in a few lines instead of a tangled screen full of boxes.

Actually what's on my mind right now is a shader language like you say that can be surrounded into nodes. So something like this (pixel shader example, very rough not thoroughly thought, let's forget about dynamic/static typing etc.):

Code: Select all

float4 main_ps() : COLOR0
{
    inNormal = DO_NORMALMAPPING( inNormal, texNormal )
    float4 diffuse, float4 specular = DO_PHONG( inNormal );

     return diffuse + specular;
}

Which can be warped into a node system in case you're a visual guy:
NormalMapping node has 2 inputs (a Vector3 & a texture) connect the vertex normals & texture, use it's output as input for the Phong node.
Phong's node two output can be mixed with an addition node.
Done.

Something like that (a language that can be expressed as nodes, or nodes that can produce code) could get best of both worlds (though it could be a pain in the neck to engineer, particularly sorting out the details where node simplicity and code syntax like to differ and choices have to be made).
The system would generate native glsl or hlsl code (probably based on snippets, eg. have external files do_phong.hlsl & do_phong.glsl that implement DO_PHONG; and people could write & contribute their own custom nodes/macros).

A node is just an abstract construct that has inputs, does some blackbox processing, and returns an output (and for sake of shaders, has no state). We just need a language that follows the same pattern (functional programming, anyone?)
I prefer high level languages (wiring them together tends to get messy), but having an editor being able to watch how it looks after tweaking each node in real time is really nice.

But right now thinking about it is not it's not on my priority list.

[sparkprime]

The idea of converting a thousand lines of shader code, including loops, lots of function abstraction, arrays, etc, into a dataflow graph is somewhat terrifying

[mkultra333]

I'm not a fan of these node or higher level language ideas, mainly because I don't want the stuff I learn in Ogre to be redundant if I move onto other projects that don't use Ogre. At the moment it's fine, I program in Cg, HLSL or GLSL and know I can transplant those exact skills, and even those same shaders, to other environments if the time comes. And as TheSheep said, it adds another layer of complexity to working out bugs.

Regarding shadows not just being "RTT and shaders," fair enough, I'm obviously simplifiying. But by the same token, I doubt the math involved is anywhere need as large a code base as what's required to set up draw surfaces, load resources, update animations, etc. How many lines of code is it to set up orthagonal projections? Relatively speaking, I bet not much. So maybe it's really "RTT, shaders, and a couple of math functions." I still think that should be treated as a user problem rather than an Ogre problem BUT (and this is the point I think people keep missing) Ogre should provide examples of these solved problems.

Basically, when it comes to anything that involves shaders, I want to be a close to the metal and to be using the real thing, not some Ogre unique invention. The reason is two fold: So that I can customize it better, and so that the skills I aquire aren't useless outside an Ogre context.

BTW didn't know the N64 was called an "Ultra," learn something new every day.