Should prepareShadowTextures() ignore visibility flags?

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin
x 65

Should prepareShadowTextures() ignore visibility flags?

Post by TheSHEEEP »

Hey,

I tried to reduce the number of objects that cast shadows by setting visibility flags.

This is what is done in a custom compositor pass:

Code: Select all

// Pause the rendering, switch to the shadow caster mask, render, then switch mask back
Ogre::SceneManager::RenderContext* context = p_sm->_pauseRendering();
            
unsigned int oldMask = _viewport->getVisibilityMask();
            
_viewport->setVisibilityMask(VISIBILITY_MASK_NONE);
p_sm->prepareShadowTextures(cam, _viewport, &ll);
_viewport->setVisibilityMask(oldMask);
            
p_sm->_resumeRendering(context);
VISIBILITY_MASK_NONE is 0, so in theory, nothing should cast a shadow.
However, everything casts a shadow, so that leads to the conclusion that visibility flags are ignored in the prepareShadowTextures() function.

I know I can use the dedicated function for that (setCastShadows), but IMO, visibility flags should also work here, as they allow some more fine grained control. Also, when an object is not visible in the actual scene due to visibility flags, it should not cast a shadow either. For that to work, you would currently have to setCastShadows to false in addition to certain visibility flags.

So, should this be changed or is it WAD (working as designed)?
My site! - Have a look :)
Also on Twitter - extra fluffy
scrawl
OGRE Expert User
OGRE Expert User
Posts: 1119
Joined: Sat Jan 01, 2011 7:57 pm
x 216

Re: Should prepareShadowTextures() ignore visibility flags?

Post by scrawl »

I'm doing something similar, and it works fine, so maybe it's a problem specific to compositor system?

Code: Select all

        TexturePtr shadowTexture = mSceneMgr->getShadowTexture(i);
        Viewport* vp = shadowTexture->getBuffer()->getRenderTarget()->getViewport(0);
        vp->setVisibilityMask(visibilityMask);
TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin
x 65

Re: Should prepareShadowTextures() ignore visibility flags?

Post by TheSHEEEP »

Strange. Do you explicitly call prepareShadowTextures?

Also, I somehow doubt that this is related to the compositor system. The prepareShadowTextures() function has nothing to do with it and I'm doing it with a paused context.
My site! - Have a look :)
Also on Twitter - extra fluffy
scrawl
OGRE Expert User
OGRE Expert User
Posts: 1119
Joined: Sat Jan 01, 2011 7:57 pm
x 216

Re: Should prepareShadowTextures() ignore visibility flags?

Post by scrawl »

No, I don't call prepareShadowTextures.
TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin
x 65

Re: Should prepareShadowTextures() ignore visibility flags?

Post by TheSHEEEP »

I just noticed that when I use visibility flags to hide an entity during normal rendering, it also does not cast shadows.

I think what happens is that all objects that were visible during scene are visible during shadow rendering when you call it the way we do.
So prepareShadowTextures likely does not check for visibility again and just renders all objects that are already visible (and have shadow casting enabled).
My site! - Have a look :)
Also on Twitter - extra fluffy
scrawl
OGRE Expert User
OGRE Expert User
Posts: 1119
Joined: Sat Jan 01, 2011 7:57 pm
x 216

Re: Should prepareShadowTextures() ignore visibility flags?

Post by scrawl »

I can disable shadow casting even for entities that are still visible during the normal render with my code.
TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin
x 65

Re: Should prepareShadowTextures() ignore visibility flags?

Post by TheSHEEEP »

Yeah. There really must be something different when calling prepareShadowTextures directly. Or, at least when calling it directly while normal rendering is suspended.
My site! - Have a look :)
Also on Twitter - extra fluffy
scrawl
OGRE Expert User
OGRE Expert User
Posts: 1119
Joined: Sat Jan 01, 2011 7:57 pm
x 216

Re: Should prepareShadowTextures() ignore visibility flags?

Post by scrawl »

Look at where the scene manager code calls prepareShadowTextures internally. The viewport it's supplying as parameter is not the shadow texture viewport, it's the regular scene viewport.
TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin
x 65

Re: Should prepareShadowTextures() ignore visibility flags?

Post by TheSHEEEP »

Oh my. Silly me. :D
Could've known that this was a different viewport as in your code.

But still,I thought it was so intuitive that I never doubted this was the used viewport. Definitely needs a comment for this :)
I will add that as soon as I get to my machine again.
My site! - Have a look :)
Also on Twitter - extra fluffy
Post Reply