visibility flags seems to break shadows

Problems building or running the engine, queries about how to use features etc.
Post Reply
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

visibility flags seems to break shadows

Post by suny »

Ogre Version: :?: 1.13
Render System: :?: dx9c

In my tool, I'm rendering some objects twice to create an outline pass which I then composite on top of my scene to show what is selected.
It was working well, but It doesn't anymore, since a few weeks.
I'm not sure if it's because of recent changes in Ogre or if it's because I compiled Ogre and my app in 64bits for the first time recently.

What happens is that when I change the mesh visibility flags, they are excluded from the shadow rendering (which is using LiSPSMShadowCameraSetup)

If an object is selected, I'm doing:

Code: Select all

if (editorState==NOTSELECTEDOVER || editorState==SELECTEDOVER)
	EntityMesh->setVisibilityFlags(1<<2);
else
	EntityMesh->setVisibilityFlags(1<<1);
Then my outline compositor is:

Code: Select all

compositor outline
{
    technique
    {
        texture outlinesSelected target_width target_height PF_FLOAT16_R no_fsaa
		texture outlinesHoover target_width target_height PF_FLOAT16_R no_fsaa
        texture scene target_width target_height PF_A8R8G8B8 
       
	    // the scene we want to modulate
        target scene
        {
            input previous
        }


        target outlinesSelected
        {
			input none
            material_scheme outlinesMaterial
			shadows off
            pass clear
            {
				colour_value 0 0 0 1
            }
			visibility_mask 4
            pass render_scene
            {
            }
        }
		
        target outlinesHoover
        {
			input none
            material_scheme outlinesMaterial
			shadows off
            pass clear
            {
				colour_value 0 0 0 1
            }
			visibility_mask 2
            pass render_scene
            {
            }
        }

        target_output
        {
            input none
			shadows off
			
            pass render_quad
            {
                material outline
                input 0 scene
                input 1 outlinesSelected
				input 2 outlinesHoover
            }
        }
    }
}
S.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1994
Joined: Sun Mar 30, 2014 2:51 pm
x 1074
Contact:

Re: visibility flags seems to break shadows

Post by paroj »

which version are you coming from? maybe this https://github.com/OGRECave/ogre/commit ... 79eb00a1d2
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

Re: visibility flags seems to break shadows

Post by suny »

I think it was working with 1.12 (I don't have access to my old versions right now).
I also add a shadowListener:

Code: Select all

struct shadowListener: public Ogre::ShadowTextureListener
{
    // this is a callback we'll be using to set up our shadow camera
    void shadowTextureCasterPreViewProj(Ogre::Light *light, Ogre::Camera *cam, size_t)
    {
        float range = light->getAttenuationRange();
        cam->setNearClipDistance(0.01);
        cam->setFarClipDistance(range);
    }

    void shadowTexturesUpdated(size_t) {}
    void shadowTextureReceiverPreViewProj(Ogre::Light*, Ogre::Frustum*) {}
    void preFindVisibleObjects(Ogre::SceneManager*, Ogre::SceneManager::IlluminationRenderStage, Ogre::Viewport*) {}
    void postFindVisibleObjects(Ogre::SceneManager*, Ogre::SceneManager::IlluminationRenderStage, Ogre::Viewport*) {}
} shadowCameraUpdaterPlay;
Is there a way to avoid excluding objects from the shadow pass with something else than 1 for their visibility flag?
S.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1994
Joined: Sun Mar 30, 2014 2:51 pm
x 1074
Contact:

Re: visibility flags seems to break shadows

Post by paroj »

just took another look at this and tried to reproduce it. I had the feeling that the visibility_mask might be not properly reset after rendering to a target.
However everything works fine here.

There are two things that must be given for your masks to work:
- your target viewport must use a mask like 0xF, such that both 2 and 4 objects are drawn
- all other objects must not have the 2 or 4 bit set - otherwise they get drawn again with "shadows off"

Ideally you would post the contents of your offscreen textures to aid debugging. This could be trivially done with renderdoc, but D3D9 is too old for that unfortunately.

note: your float16 targets are probably superfluous, as you could use multiple "target scene" sections without clearing to draw over previous contents.
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

Re: visibility flags seems to break shadows

Post by suny »

Still looking at my issue:
I found what part of my code is breaking the shadows, but I still don't understand why.

In my game, the actual game viewport can have several ratios, and I'm displaying a picture on the borders:

Image

What I'm doing is that I have 2 viewports:
-A full-screen one that only renders one overlay
-A viewport using the game screen ratio to only render the gameplay and clip it

Is there a more efficient way to do this?
Could I only use one viewport and clip the game and still render one overlay on top of it?

And it creates several issues:
Even if I only render only one overlay to the borders viewport, I noticed it seems to update all gameplay nodes twice and really slow-downs a lot the game.
So, I tried to use mviewportBG->setAutoUpdated(false); after one frame, and it worked OK.

But if I do this and render my outlines, it still messes up the shadows, as if the visibility masks from the post-processing are also changing the shadow's visibility.
And it only do this if I use mviewportBG->setAutoUpdated(false); once.
Even if I later do mviewportBG->setAutoUpdated(true);, the shadows are still broken...

paroj
OGRE Team Member
OGRE Team Member
Posts: 1994
Joined: Sun Mar 30, 2014 2:51 pm
x 1074
Contact:

Re: visibility flags seems to break shadows

Post by paroj »

suny wrote: Thu Mar 03, 2022 9:55 am


Is there a more efficient way to do this?
Could I only use one viewport and clip the game and still render one overlay on top of it?

if you dont update the lower viewport, it doesnt really matter. However, you could also use only one viewport and resize that after the first render. Only the viewport region will be cleared. (except on D3D11)

suny wrote: Thu Mar 03, 2022 9:55 am

Even if I only render only one overlay to the borders viewport, I noticed it seems to update all gameplay nodes twice and really slow-downs a lot the game.

the nodes are updated per camera. So two cameras, means iterating the scene twice.

suny wrote: Thu Mar 03, 2022 9:55 am

But if I do this and render my outlines, it still messes up the shadows, as if the visibility masks from the post-processing are also changing the shadow's visibility.

the visibility mask in the compositor applies to the target viewport. If shadows are enabled there, it will also update shadow textures with that mask. But actually, I have no idea what you are doing. You probably should create a diagram of what is rendered with which mask in which order.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: visibility flags seems to break shadows

Post by dark_sylinc »

paroj wrote: Thu Oct 28, 2021 8:01 pm

Ideally you would post the contents of your offscreen textures to aid debugging. This could be trivially done with renderdoc, but D3D9 is too old for that unfortunately.

DXVK can be used to run D3D9 via Vulkan, and RenderDoc will capture the Vulkan commands.

DXVK actually runs on Windows, so there is no need for WINE/Linux.

I've heard from devs having success using this method so it is definitely worth trying.

Post Reply