Lights with shadows - dynamic implementation instead of inline

[DimA]

Hi all!

Why not to have alternative dynamic implementation of support for lights with shadows? I mean implement them in the same way as Forward3D lights in the hlms shaders, using dynamic branching instead of shader code inlining for each point and spot light.

In large or/and dynamic scenes with many lights of both types(spot and point) we can get very unstable nearest point and spot lights combinations.
For example, at one position in the scene we can have 10 point-lights and 6 spot-lights, and at another point we can get 4 point-lights and 10 spot-lights, etc.

Current inlining technique, in theory, may generate (N+1)(N+2)/2 shader variants, where N is total lights(point+spot) number. Actually, we'll get even more shader variants if count all material differences. Imagine N==32, then we can get up to 561 shaders!!!

Our app, Live Home 3D https://livehome3d.com , is affected. Indeed, our app is an editor where users may design any scene and place many light sources. So, we have to take in account only nearest lights and preserve their shadow maps if possible to minimize shadow maps updates. When the user walks through the scene in 3D view, we have to generate new shaders each time we find a new lights combination and this takes time.

What do you think about this, guys?

[dark_sylinc]

Hi!

Eugene tells me you're already using HlmsPbs::setUseLightBuffers.

I've been remembering and also checking the code again.

Our current implementation has hlms_static_branch_lights and setMaxNonCasterDirectionalLights; but it is (currently) only working for directional non-caster lights and uses static lighting to pass an arbitrary number of lights:

Code: Select all

@property( !hlms_static_branch_lights )
	@foreach( hlms_lights_directional_non_caster, n, hlms_lights_directional )
		@insertpiece( ObjLightMaskCmp )
			finalColour += BRDF( light0Buf.lights[@n].position.xyz, light0Buf.lights[@n].diffuse.xyz, light0Buf.lights[@n].specular, pixelData );@end
@else
	for( int n=0; n<floatBitsToInt( light0Buf.numNonCasterDirectionalLights ); ++n )
	{
		int i = @value( hlms_lights_directional ) + n;
		@insertpiece( ObjLightMaskCmpNonCasterLoop )
			finalColour += BRDF( light0Buf.lights[i].position.xyz, light0Buf.lights[i].diffuse.xyz, light0Buf.lights[i].specular, pixelData );
	}
	/// Increase fineMaskLightIdx to keep it working with spot/point lights
	@insertpiece( ObjLightMaskCmpNonCasterLoopEnd )
@end

Note: just to clarify; static branching are branches (and loops) where the values are sent via uniforms and is therefore equal for all pixels within the draw call; and this is enough to get the feature working. "Dynamic" branching means the branch/loop can vary per thread or pixel, which is overkill.

We'd need to do the same for shadow mapped lights, with a few caveats or things to watch out:

• Each shadow can arbitrary live in any atlas. The solution to this is that
  1. Lights that share the same atlas are grouped together (i.e. one static loop per atlas type and inside the loop we check whether it's spot or point and execute its code), or....
  2. Textures are statically indexed i.e. shadowmapAtlas[ i ]. This may limit HW supported (I think this is FL 11? I'm can't remember if FL10 can do this). The main issues to encounter is that our GL3+ RS unnecessarily always wants to set the textures units from C++ (this is a bit of a PITA) and Metal SL 1.0 did not support arrays of textures (it does now, in more modern versions of Metal SL; back then compatibility would've been a huge problem)
• The UV borders from shadow atlas (i.e. hlms_shadowmap0_uv_param) would probably have to be sent through uniforms rather than hardcoded

Conceptually the problem is not hard, but the details of getting everything right (lights sorted by atlas, or getting the arrays of textures working as intended in all platforms) can be overwhelming.

Alternatively you can take on compromises and make the solution much easier:

The main problem right now is that if light types switch (i.e. you had 10 spot + 10 points; and now you have 11 spot + 9 points) you will need a new shader. If you instead limit it to 10 spots + 10 points separately (i.e. "this shadow map only accepts spots", "this shadow map only accepts points"; which is something already possible in the Compositor as you must tell the compositor which shadowmaps accept which light types) you can do the following:

1. Modify Hlms to optionally allow always generating 10 spots + 10 points; and wrap each light in a static branch that toggles it on / off.
2. As a workaround you can test this idea right now using CompositorShadowNode::setLightFixedToShadowMap to manually assign lights to their shadow maps, and you can ensure your max point/spot lights are never exceeded, and when you have fewer lights than necessary (e.g. you only have 4 spots) you assign a black light to that shadow map. This will burn CPU & GPU but it will show you a proof of concept

This alternative sounds much more feasible and easy; but the compromise is that with the old system you could have 11 spot + 9 points; whereas with the new you can only have either 10 spot + 9 points (1 disabled) or 10 spot + 10 points (1 of them is far away)

[dark_sylinc]

I just realized there is one more alternative:

If you want to support 20 lights (any type), we could write code for 20 spot lights + 20 point lights (40 lights in total).

If we have the 10 spot + 10 points in the scene then:

• 10 spot branches will be true
• 10 spot branches will be false
• 10 points branches will be false
• 10 points branches will be true.

This doesn't scale well with number of lights compared to a for loop (because a for loop early outs; whereas we always must evaluate 40 if statatements), but it gets the job done.

Theoretically we can mitigate this scaling problem in two ways:

1. Generate multiple shaders for multiple increments. For example if only 10 lights are active (any type) we use a shader that supports 10 spot + 10 point lights (20 in total). If you add 1 more point light; we jump to a version that supports 10 spot + 20 point (30 in total). There will be shader hiccups, but it won't be the combinatorial explosion we have today.
2. Async generation. Basically the ubershaders that Dolphin uses. Use the generic solution with 40 lights while more specialized Hlms shaders are compiled in the background (this solution is not mutually exclusive with the previous one)

[DimA]

Hi Matias!

Thank you for quick response!
You are right, we can implement that using static branching - all necessary variables (point lights number and spot lights number as well as UV borders for shadow atlas) can be passed via uniforms.
All we need is to get one or few(depending on material differences) shaders for set of N lights-shadow casters instead of (N+1)(N+2)/2.
In our app we keep lights presorted and grouped by type using prepareShadowsForWorkspace() callback and using setLightFixedToShadowMap() to preserve shadow maps for already known lights(with alive shadows from previous frames).

Next week I'll try to implement this using static branching as you suggested.
I'm not sure if it'll be possible for all RS, currently we use Metal and DX FL11. I believe we'll use Vulcan as well for Android port, but right now I don't have any testing Android device.
Anyway I'll probably create a new property to turn this technique and disable other properties to avoid shaders duplications. I'm not sure if I can use hlms_static_branch_lights property for this, may be it's better to create another one.

Thank you for explanations!
I hope static branching for shadow mapped lights will be useful for many Ogre users.

[dark_sylinc]

Yes, when you get some progress post the github link so I can take a look. This is a requested feature, particularly by applications similar to yours (user-edited scenes with lots of lights)

[DimA]

Sure, I’ll create a fork on GitHub.

[jwwalker]

This is a requested feature, particularly by applications similar to yours (user-edited scenes with lots of lights)

I don't know enough at this point to contribute to a solution. But I just wanted to say that I too have user-edited scene with (potentially) lots of lights, and I wish that it were easier to get started with shadows. I know that there are various shadow algorithms and parameters that can be tweaked, but it would be great if there were some default mode that one could just turn on and get half-decent shadows from any lights designated as shadow-casting.

[DimA]

Hi all!

It's taken a while. I was busy with my project.
Nevertheless I've started implementation and almost done it.
I started with Mac platform and thanks to cross-platform nature of HLMS shaders it looks almost done for Metal, DirectX and Vulkan but I have to test it on Windows platform to be sure.
It'll take time to setup Ogre on my new Windows PC, but I'm sure the final implementation will be soon.

I work in my Ogre fork here:
https://github.com/dyunchik/ogre-next/t ... _branching

So, everyone can have look at the code. Any suggestions are welcome

[dark_sylinc]

Awesome!

I'm currently full on fixing warnings and C++11-izing the codebase on roadmap-2.4 branch (some of it has already been merged with master).

I want to try this, so probably after I'm done I'll merge your stuff into its own branch for me to test.

Btw if you test Vulkan you may find to some inexplicable error because you're now using arrays of textures for the shadow maps, and we need to set that in the Root Layout.

This is fixed by adding to HlmsPbs::setupRootLayout:

Code: Select all

rootLayout.addArrayBinding( DescBindingTypes::Texture, RootLayout::ArrayDesc( shadowmapTexSlot, numElementsInArray ) );

The details are covered in the manual. I'm telling you so you don't waste precious time figuring out wtf is going on.

Cheers

[DimA]

Hm, I started on 2.3.
"master" has moved further since my branch "sm_lights_branching" start.

Btw if you test Vulkan you may find to some inexplicable error because you're now using arrays of textures for the shadow maps, and we need to set that in the Root Layout.

I use atlases instead of arrays of textures for the shadow maps.
I use a hack to access textures in loop by index:

Code: Select all

@insertpiece( TEXTURE2DSHADOW ) shadowmap[@value(hlms_num_shadow_map_lights)] =
			@property( syntax == glsl || syntax == glsles || syntax == glslvk )
				@insertpiece( TEXTURE2DSHADOW )[@value(hlms_num_shadow_map_lights)](
			@else
			{
            @end
            @foreach( hlms_num_shadow_map_lights, n ) hlms_shadowmap@n ,
            @end
            @property( syntax == glsl || syntax == glsles || syntax == glslvk )
			);
			@else
			};
			@end

That produces following shader code:

Code: Select all

depth2d<float> shadowmap[7] =
			
			{
            
             hlms_shadowmap0 ,
             hlms_shadowmap1 ,
             hlms_shadowmap2 ,
             hlms_shadowmap3 ,
             hlms_shadowmap4 ,
             hlms_shadowmap5 ,
             hlms_shadowmap6 ,
            
            
			};

The main requirements: each shadow map should be universal and support both light types: point and spot.
So, the shadowmap setup should be like this:

Code: Select all

shadowParam.addLightType( Ogre::Light::LT_POINT );
shadowParam.addLightType( Ogre::Light::LT_SPOTLIGHT );

I've modified ShadowMapFromCode sample for debug and testing purposes. Now it uses USE_STATIC_BRANCHING_FOR_SHADOWMAP_LIGHTS macro to turn on SM lights branching.

[DimA]

Oops!
Looks like my hack with sm indexing doesn't work on glsl/glslvk
I've already managed it for hlsl, but stuck with glsl/glslvk - syntax doesn't allow local actual arrays with texture2D objects

I have to find a solution.

[DimA]

Ok, I figured out how to overcome the problem.

And now I understand better what Matias talked about:

Btw if you test Vulkan you may find to some inexplicable error because you're now using arrays of textures for the shadow maps, and we need to set that in the Root Layout.

I’m going change shadow maps uniforms to an array.

[DimA]

I've fixed sm lights branching for glslvk.
So, now this works for Metal, DirectX and Vulkan, old inline technique is used as fallback for OpenGL.
Now I plan to do some optimizations and make sure unnecessary hlms properties are removed, that finally decreases possible shaders variants from (N+1)(N+2)/2 to N.

[DimA]

Finally I've done this sm lights branching technique!

Brief description:

SM lights branching technique decreases possible shader variants for point and spot shadow-casting lights. The technique is very suitable when you have many regular (point or spot) shadow-casting lights in the scene and their combination may change in different scene locations. Default inline technique requires new shaders for every possible shadow-casting lights combination.
SM lights branching technique creates two separate loops in PS shader - one for point shadow-casting lights and one for spot lights. Make sure your lights are ordered and grouped correctly(point lights first, then - spots) if you have your own lights management.
Current implementation works for Metal, DirectX and Vulkan render systems. Default inline technique is used for OpenGL/OpenGLES.

To enable SM lights branching technique use Ogre::HlmsPbs::setMaxShadowMapLights() method before any PBS materials creation.
I've modified Sample_ShadowMapFromCode for demo and debug purposes. Use USE_STATIC_BRANCHING_FOR_SHADOWMAP_LIGHTS macro to turn on/off the technique.

My fork with implementation can be found here:
https://github.com/dyunchik/ogre-next/t ... _branching

I've created pull request:
https://github.com/OGRECave/ogre-next/pull/255