Shadow improvements
- areay
- Bugbear
- Posts: 819
- Joined: Wed May 05, 2010 4:59 am
- Location: Auckland, NZ
- x 69
Re: Shadow improvements
Thank you so much dark_sylinc. Going through the 1.10 -> 2.0 migration at the moment and can't wait to get the shadow stability improvements available in 2.1.
-
- Halfling
- Posts: 46
- Joined: Tue Feb 18, 2014 10:49 pm
Re: Shadow improvements
Where is the Shadows/ClearRegion material? i don't find it
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
?Feanor16 wrote:Where is the Shadows/ClearRegion material? i don't find it
I do not understand your question
-
- Halfling
- Posts: 64
- Joined: Mon Sep 12, 2011 7:35 pm
- x 13
Re: Shadow improvements
Integrated it to my project and so far as I tested everything is working just fine, specially point light shadows
You're the one dark_sylinc!
There was two things I needed to do to integrate (besides updating the hlms scripts):
1) Redefine my compositor to be like the one defined at ShadowMapDebugging.compositor
2) Define "Hlms/Pbs/Any" at the Library vector when creating the Ogre::HlmsPbs (as defined at Samples/2.0/Common/src/GraphicsSystem.cpp)
I've also copied the folder Samples/Media/2.0/scripts/materials/Common, based on this commit, but I'm not sure which are the necessary files or if any are necessary at all.
You're the one dark_sylinc!
There was two things I needed to do to integrate (besides updating the hlms scripts):
1) Redefine my compositor to be like the one defined at ShadowMapDebugging.compositor
2) Define "Hlms/Pbs/Any" at the Library vector when creating the Ogre::HlmsPbs (as defined at Samples/2.0/Common/src/GraphicsSystem.cpp)
I've also copied the folder Samples/Media/2.0/scripts/materials/Common, based on this commit, but I'm not sure which are the necessary files or if any are necessary at all.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
Niiice!farrer wrote:Integrated it to my project and so far as I tested everything is working just fine, specially point light shadows
Thanks!You're the one dark_sylinc!
Yes, we use DPSM.material and all contained in it + its dependencies (i.e. in short: CubeToDpsm_ps & Quad_vs).I've also copied the folder Samples/Media/2.0/scripts/materials/Common, based on this commit, but I'm not sure which are the necessary files or if any are necessary at all.
I added the whole folder because there's also other common functionality, for example Compute-based high quality mipmaps require the compute shader defined there. So it became more convenient to just include the whole folder (as long as "Common" doesn't become a huge bloated folder in the future).
Btw IMPORTANT: A user has reported Ogre is generating invalid shaders in certain cases. These cases are triggered if a spotlight is closer to the camera than a point light; since Ogre will try to put the spotlight first and then the point light afterwards, but the shader expects all point lights to come first, then all spotlights (it's an order issue).
I'll fix this bug soon.
-
- Halfling
- Posts: 46
- Joined: Tue Feb 18, 2014 10:49 pm
Re: Shadow improvements
Well, i tried your compositor_node_shadow thinking it will maybe solve shadow bugs i have. But when applying yours, I have no shadows displaying. so I thought i miss materials like "Shadows/ClearRegion" or "dpsm/converter". but i don't find them. There must be a file of these materials, right? ( Or maybe i miss something? )dark_sylinc wrote:?Feanor16 wrote:Where is the Shadows/ClearRegion material? i don't find it
I do not understand your question
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
Gotcha. The compositor script in the first post was a proposal. The final one is at Samples/Media/2.0/scripts/Compositors/ShadowMapDebugging.compositor in the v2-1-shadows-refactor branch.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
I just pushed a fix to the bug where invalid shaders would be generated if a spotlight was closer to the camera than a point light.
Additionally, I encountered a bug with static shadow maps which I've fixed (with the compositor script we provided, mixing spotlights & pointlights would cause spotlight shadows not to show).
Additionally, I encountered a bug with static shadow maps which I've fixed (with the compositor script we provided, mixing spotlights & pointlights would cause spotlight shadows not to show).
-
- Halfling
- Posts: 46
- Joined: Tue Feb 18, 2014 10:49 pm
Re: Shadow improvements
Oh ok! thank you for all
-
- Halfling
- Posts: 64
- Joined: Mon Sep 12, 2011 7:35 pm
- x 13
Re: Shadow improvements
Not sure if it's a bug or if it's me doing something wrong: On the compositors, I've used to define at the shadows render pass the render queues that should affect them, to avoid having GUI widgets interfering with the shadows, ie:
That's ok and working well. But when trying to do something similar to the render of point light's shadows, the first and last seems to be ignored and my widgets are affecting my shadows rendering (see the hand-made red circles of the affecting areas at the picture... moving the window will make easy to see those are generated from it):
(screenshot taken of OpenGL renderer on Linux)
Maybe I'm wrongly defining it for the point light shadows? This is my compositor (note that the widgets are rendered at the rq 240):
Thanks!
Code: Select all
shadow_map_target_type directional spot
{
shadow_map 3 4
{
pass render_scene
{
rq_first 0
rq_last 230
}
}
}
(screenshot taken of OpenGL renderer on Linux)
Maybe I'm wrongly defining it for the point light shadows? This is my compositor (note that the widgets are rendered at the rq 240):
Code: Select all
compositor_node DNTIndoorRenderingNode
{
in 0 rt_renderwindow
target rt_renderwindow
{
pass clear
{
colour_value 0.0 0.0 0.0 1
}
pass render_scene
{
shadows DNTIndoorShadowNode
}
}
}
abstract target cubemap_target_shadow
{
pass clear { colour_value 1 1 1 1 }
pass render_scene
{
camera_cubemap_reorient true
}
}
compositor_node_shadow DNTIndoorShadowNode
{
technique pssm
texture atlas 2048 7168 PF_D32_FLOAT no_fsaa
texture tmpCubemap 1024 1024 PF_FLOAT32_R cubemap no_fsaa
num_splits 3
pssm_lambda 0.95
shadow_map 0 atlas uv 0.0 0.000000000000000 1.0 0.285714285714286 light 0 split 0
shadow_map 1 atlas uv 0.0 0.285714285714286 0.5 0.142857142857143 light 0 split 1
shadow_map 2 atlas uv 0.5 0.285714285714286 0.5 0.142857142857143 light 0 split 2
technique focused
shadow_map 3 atlas uv 0.0 0.428571428571429 1.0 0.285714285714286 light 1
shadow_map 4 atlas uv 0.0 0.714285714285714 1.0 0.285714285714286 light 2
target atlas
{
pass clear
{
colour_value 1 1 1 1
}
}
shadow_map_target_type directional
{
shadow_map 0 1 2
{
pass render_scene
{
rq_first 0
rq_last 230
}
}
}
shadow_map_target_type directional spot
{
shadow_map 3 4
{
pass render_scene
{
rq_first 0
rq_last 230
}
}
}
shadow_map_target_type point
{
shadow_map_repeat 3 4
{
target tmpCubemap +X : cubemap_target_shadow {}
target tmpCubemap -X : cubemap_target_shadow {}
target tmpCubemap +Y : cubemap_target_shadow {}
target tmpCubemap -Y : cubemap_target_shadow {}
target tmpCubemap +Z : cubemap_target_shadow {}
target tmpCubemap -Z : cubemap_target_shadow {}
shadow_map
{
pass render_quad
{
material Ogre/DPSM/CubeToDpsm
input 0 tmpCubemap
rq_first 0
rq_last 230
}
}
}
}
}
workspace DNTIndoorWorkspace
{
connect_output DNTIndoorRenderingNode 0
}
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
Yes, you're incorrectly defining your compositor.
Your RenderQueue setting should be in cubemap_target_shadow (or inside each target tmpCubemap +-XYZ : cubemap_target_shadow {} declaration).
You've placed it instead in the render_quad which is the last stage and does nothing there.
Your RenderQueue setting should be in cubemap_target_shadow (or inside each target tmpCubemap +-XYZ : cubemap_target_shadow {} declaration).
You've placed it instead in the render_quad which is the last stage and does nothing there.
-
- Halfling
- Posts: 64
- Joined: Mon Sep 12, 2011 7:35 pm
- x 13
Re: Shadow improvements
Many thanks! It was that! (sometimes it's so hard to see something so trivial )dark_sylinc wrote:Yes, you're incorrectly defining your compositor.
Your RenderQueue setting should be in cubemap_target_shadow (or inside each target tmpCubemap +-XYZ : cubemap_target_shadow {} declaration).
You've placed it instead in the render_quad which is the last stage and does nothing there.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
This was just merged in:
Exponential Shadow Maps, which is basically soft shadows. The technique overall is much more expensive. However, what's great about this technique is that you can lower the resolution and it still looks great (it makes the shadows blurrier); while giving you more performance. I'm also using 2 PSSM splits instead of 3.
This is the same screenshot but with half the resolution of the above one (may shimmer a little more when the camera & objects) moves: Integration should be straightforward. Use the ESM's shadow node, make sure you include the resources from Samples/Media/2.0/scripts/materials/Common, add some C++ code to handle older GPUs (the ones who don't have a compute shader) you can copy from the demo, and tell HlmsPbs to use ESM. You can also tweak the radius and distribution of the gaussian filter.
There is yet a parameter that cannot be tweaked ('K') but soon will be.
The technique is great for indoor shadows, soft shadows, and has now shadow acne (hence no need for biasing at all). However there are a few edge cases I will talk about later (I am too tired right now) where the technique fails.
Exponential Shadow Maps, which is basically soft shadows. The technique overall is much more expensive. However, what's great about this technique is that you can lower the resolution and it still looks great (it makes the shadows blurrier); while giving you more performance. I'm also using 2 PSSM splits instead of 3.
This is the same screenshot but with half the resolution of the above one (may shimmer a little more when the camera & objects) moves: Integration should be straightforward. Use the ESM's shadow node, make sure you include the resources from Samples/Media/2.0/scripts/materials/Common, add some C++ code to handle older GPUs (the ones who don't have a compute shader) you can copy from the demo, and tell HlmsPbs to use ESM. You can also tweak the radius and distribution of the gaussian filter.
There is yet a parameter that cannot be tweaked ('K') but soon will be.
The technique is great for indoor shadows, soft shadows, and has now shadow acne (hence no need for biasing at all). However there are a few edge cases I will talk about later (I am too tired right now) where the technique fails.
- spookyboo
- Silver Sponsor
- Posts: 1141
- Joined: Tue Jul 06, 2004 5:57 am
- x 151
- Contact:
Re: Shadow improvements
Gui generator tool https://github.com/spookyboo/Magus ==> Windows binaries https://github.com/spookyboo/Magus_bin
HLMS editor https://github.com/spookyboo/HLMSEditor ==> Windows setup https://github.com/spookyboo/HLMSEditor ... e?raw=true
HLMS editor https://github.com/spookyboo/HLMSEditor ==> Windows setup https://github.com/spookyboo/HLMSEditor ... e?raw=true
-
- OGRE Expert User
- Posts: 1148
- Joined: Sat Jul 06, 2013 10:59 pm
- Location: Chile
- x 168
Re: Shadow improvements
woooo so pretty!! got to test it right now!
Edit: just built the sample and its looking reeaaally good! the only weird thing I noticed is that the shadow grow/dilate compared to the pcf ones, not a real problem really
Edit: just built the sample and its looking reeaaally good! the only weird thing I noticed is that the shadow grow/dilate compared to the pcf ones, not a real problem really
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
Pull again, I've fixed that. How much they dilate can be controlled now via setGaussianLogFilterParams's K parameter. Please read the docs to understand the pros & cons of doing that (most likely you'll end up tweaking that value until it goods the way you like it and will only touch it again if you run into an edge case where the shadow overall looks bad: not blurred at all, really aliased; or too blurry it's barely a shadow).
I've tweaked the default parameters so that it can adapt to most conditions (particularly extreme edge cases where it looked really bad), instead of failing miserably
I've tweaked the default parameters so that it can adapt to most conditions (particularly extreme edge cases where it looked really bad), instead of failing miserably
-
- OGRE Expert User
- Posts: 1227
- Joined: Thu Dec 11, 2008 7:56 pm
- Location: Bristol, UK
- x 157
Re: Shadow improvements
If not already noticed the HdrWorkspaceMsaa compositor is not working, see;
https://bitbucket.org/sinbad/ogre/commi ... nt-4575133
https://bitbucket.org/sinbad/ogre/commi ... nt-4575133
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
Thanks! Pushed a fix.
- 0xC0DEFACE
- OGRE Expert User
- Posts: 84
- Joined: Thu May 21, 2009 4:55 am
- x 7
Re: Shadow improvements
omg, its what ive always wanted. Ill have to update to the latest to have a play
Great work Matias!
Great work Matias!
-
- Goblin
- Posts: 253
- Joined: Sat Sep 28, 2013 3:46 pm
- x 10
Re: Shadow improvements
Hey I'm currently trying to integrate the new shadow system with HLMS_DS and I'm trying to figure out which part of the HLMS_PBS Shader handels shadows and more specifically where the shadow samplers are declared, so I can figure out how to make the new shaders work. Also Is there now a way to dynamically allocate and deallocate shadow cqasting lights? Because that might be very usefull for dynamically loading levels.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
In shader code it's in Samples/Media/Hlms/Pbs/Any/ShadowMapping_piece_ps.any line 111 (GLSL), 124 & 126 (HLSL) and 140 (Metal).white_waluigi wrote:Hey I'm currently trying to integrate the new shadow system with HLMS_DS and I'm trying to figure out which part of the HLMS_PBS Shader handels shadows and more specifically where the shadow samplers are declared, so I can figure out how to make the new shaders work.
The language specific shaders only have to do:
Code: Select all
@insertpiece( DeclShadowMapMacros )
@insertpiece( DeclShadowSamplers )
I'm not sure what you mean by that. What do you want to do? How do you expect it to work?white_waluigi wrote:Also Is there now a way to dynamically allocate and deallocate shadow cqasting lights? Because that might be very usefull for dynamically loading levels.
-
- Goblin
- Posts: 253
- Joined: Sat Sep 28, 2013 3:46 pm
- x 10
Re: Shadow improvements
Its okay I'll just look at your code, I'll figure it out
-
- Gremlin
- Posts: 173
- Joined: Wed Feb 03, 2016 2:24 am
- x 17
- Contact:
Re: Shadow improvements
Would it be easy to integrate the new awesome shadow into a custom HLMS implementation?
My HLMS is screaming for it.
My HLMS is screaming for it.
- dark_sylinc
- OGRE Team Member
- Posts: 5296
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1278
- Contact:
Re: Shadow improvements
I'd like to think it's easy to integrate. In fact the shadow code has been modularized by being moved from the main templates into its own ShadowMapping_piece_ps.any & ShadowMapping_piece_vs.any files.
The major changes:
The major changes:
- Add "Any" folder to the template library when initializing HlmsPbs, so the shadow mapping code gets parsed.
- texShadowMap[N]; is no longer an array. It's now several variables: texShadowMap0, texShadowMap1, ... texShadowMapN. This is for compatibility with Metal. This change mostly affects C++ code for GL3+ (since C++ sets the texture unit binding points of the variables)
- Definitions in shader code of samplers, getShadow have been replaced by insertpiece statements (actual locations may vary a little between HLSL, GLSL & Metal):
Code: Select all
@insertpiece( DeclShadowMapMacros ) @insertpiece( DeclShadowSamplers ) @insertpiece( DeclShadowSamplingFuncs )
- PSSM & directional shadow map code has been moved out of the template, so all you need to do is "@insertpiece( DoDirectionalShadowMaps )"
- Point lights had a "@insertpiece( DarkenWithShadow )" in their calculations. Now they need a "@insertpiece( DarkenWithShadowPoint )" since they have a different treatment
- struct ShadowReceiverData now defines a variable "vec4 texViewZRow;"
- Some languages had "passBuf" as variable for per-pass const buffers; others had "pass". It has been standardized to "passBuf" (since pass is a reserved keyword in HLSL)
- Vertex shader has been moved out of the main template. All you need to do is:
Code: Select all
@insertpiece( DoShadowReceiveVS ) @insertpiece( DoShadowCasterVS )
- CrossPlatformSettings_piece_all & RenderDepthOnly_piece_ps have changed slightly. Make sure your code adapt to these new changes.
- C++ side preparePassHash has changed to account for the new variables and for the fact that lights may now share the same shadow map. The changes are very straightforward
- SolarPortal
- OGRE Contributor
- Posts: 203
- Joined: Sat Jul 16, 2011 8:29 pm
- Location: UK
- x 51
- Contact:
Re: Shadow improvements
Hi, First off, great work on the recent changes to 2.1 branch! Point light shadows are amazing!
I have noticed a slight problem with atlased shadowmaps though and are wondering if you could help out.
What i am trying to achieve is a fill a 4096x8192 texture with as many shadows as possible ranging from 1024 to 2048 resolutions and including the pssm splits as you have in other compositor examples.
So 4096 will give me 2 columns of 2048 and 4 columns of 1024.
Here is an image of what we are trying to accomplish in the exact resolution of the texture (it is 4096x8192, so you may need to download and view it too get a good idea):
http://www.chi-ad.com/Skyline/downloads ... _Altas.png
but for previewing on the forum, here is a downscaled resolution: To show the problem we are having with a shadow texture that starts 0.5 across in the image(red square), here it is: Here is the compositor node for the shadow node: I left in the extra textures as the more you add, the wierder the results get with overlapping shadows, missing shadows and eventually the PSSM splits disappearing:
So the question is, are we missing anything or breaking the rules of how the atlases work on the new shadow system. For focused shadows, does the texture have to span 0.0 to 1.0 on the X axis.
Is there any advice or help you can offer @dark_sylinc
Thanks for the help and time
I have noticed a slight problem with atlased shadowmaps though and are wondering if you could help out.
What i am trying to achieve is a fill a 4096x8192 texture with as many shadows as possible ranging from 1024 to 2048 resolutions and including the pssm splits as you have in other compositor examples.
So 4096 will give me 2 columns of 2048 and 4 columns of 1024.
Here is an image of what we are trying to accomplish in the exact resolution of the texture (it is 4096x8192, so you may need to download and view it too get a good idea):
http://www.chi-ad.com/Skyline/downloads ... _Altas.png
but for previewing on the forum, here is a downscaled resolution: To show the problem we are having with a shadow texture that starts 0.5 across in the image(red square), here it is: Here is the compositor node for the shadow node: I left in the extra textures as the more you add, the wierder the results get with overlapping shadows, missing shadows and eventually the PSSM splits disappearing:
Code: Select all
abstract target cubemap_target_shadow
{
pass clear { colour_value 1 1 1 1 }
pass render_scene
{
camera_cubemap_reorient true
}
}
compositor_node_shadow ShadowMapNode_HIGH_NoStatic
{
technique pssm
texture atlas 4096 8192 PF_D32_FLOAT no_fsaa
texture tmpCubemap 1024 1024 PF_FLOAT32_R cubemap no_fsaa
num_splits 3
pssm_lambda 0.95
// Split1 = 2048 - Split 2&3 = 1024
shadow_map 0 atlas uv 0.0 0.0 0.5 0.25 light 0 split 0
shadow_map 1 atlas uv 0.0 0.25 0.25 0.125 light 0 split 1
shadow_map 2 atlas uv 0.25 0.25 0.25 0.125 light 0 split 2
technique focused
// 2048 shadows
shadow_map 3 atlas uv 0.0 0.375 0.5 0.25 light 1
shadow_map 4 atlas uv 0.0 0.625 0.5 0.25 light 2
// From here on shadows do not render correctly and eventually even the PSSM loses its splits.
shadow_map 5 atlas uv 0.5 0.0 1.0 0.25 light 3 // Only displays half a shadow as if the 0.5 is clipping
//shadow_map 6 atlas uv 0.5 0.375 1.0 0.25 light 4
//shadow_map 7 atlas uv 0.5 0.625 1.0 0.25 light 5
// 1024 shadows - Fill in the gaps to get more shadows.
//shadow_map 8 atlas uv 0.5 0.25 0.75 0.125 light 6
//shadow_map 9 atlas uv 0.75 0.25 1.0 0.125 light 7
//shadow_map 10 atlas uv 0.0 0.875 0.25 0.125 light 8
//shadow_map 11 atlas uv 0.25 0.875 0.5 0.125 light 9
//shadow_map 12 atlas uv 0.5 0.875 0.75 0.125 light 10
//shadow_map 13 atlas uv 0.75 0.875 1.0 0.125 light 11
target atlas
{
pass clear
{
colour_value 1 1 1 1
}
}
shadow_map_target_type directional
{
shadow_map 0 1 2
{
pass render_scene
{
}
}
}
shadow_map_target_type directional spot
{
shadow_map 3 4 5 //6 7 8 9 10 11 12 13
{
pass render_scene
{
}
}
}
shadow_map_target_type point
{
shadow_map_repeat 3 4 5 //6 7 8 9 10 11 12 13
{
target tmpCubemap +X : cubemap_target_shadow {}
target tmpCubemap -X : cubemap_target_shadow {}
target tmpCubemap +Y : cubemap_target_shadow {}
target tmpCubemap -Y : cubemap_target_shadow {}
target tmpCubemap +Z : cubemap_target_shadow {}
target tmpCubemap -Z : cubemap_target_shadow {}
shadow_map
{
pass render_quad
{
material Ogre/DPSM/CubeToDpsm
input 0 tmpCubemap
}
}
}
}
}
Is there any advice or help you can offer @dark_sylinc
Thanks for the help and time
Lead developer of the Skyline Game Engine: https://aurasoft-skyline.co.uk