Ogre Forums

Posted: **Fri Nov 15, 2013 1:49 pm**

Hey,

I tried to reduce the number of objects that cast shadows by setting visibility flags.

This is what is done in a custom compositor pass:

Code: Select all

// Pause the rendering, switch to the shadow caster mask, render, then switch mask back
Ogre::SceneManager::RenderContext* context = p_sm->_pauseRendering();
            
unsigned int oldMask = _viewport->getVisibilityMask();
            
_viewport->setVisibilityMask(VISIBILITY_MASK_NONE);
p_sm->prepareShadowTextures(cam, _viewport, &ll);
_viewport->setVisibilityMask(oldMask);
            
p_sm->_resumeRendering(context);

VISIBILITY_MASK_NONE is 0, so in theory, nothing should cast a shadow.
However, everything casts a shadow, so that leads to the conclusion that visibility flags are ignored in the prepareShadowTextures() function.

I know I can use the dedicated function for that (setCastShadows), but IMO, visibility flags should also work here, as they allow some more fine grained control. Also, when an object is not visible in the actual scene due to visibility flags, it should not cast a shadow either. For that to work, you would currently have to setCastShadows to false in addition to certain visibility flags.

So, should this be changed or is it WAD (working as designed)?

Posted: **Fri Nov 15, 2013 2:00 pm**

I'm doing something similar, and it works fine, so maybe it's a problem specific to compositor system?

Code: Select all

        TexturePtr shadowTexture = mSceneMgr->getShadowTexture(i);
        Viewport* vp = shadowTexture->getBuffer()->getRenderTarget()->getViewport(0);
        vp->setVisibilityMask(visibilityMask);

Posted: **Fri Nov 15, 2013 2:07 pm**

Strange. Do you explicitly call prepareShadowTextures?

Also, I somehow doubt that this is related to the compositor system. The prepareShadowTextures() function has nothing to do with it and I'm doing it with a paused context.

Posted: **Fri Nov 15, 2013 2:33 pm**

No, I don't call prepareShadowTextures.

Posted: **Fri Nov 15, 2013 3:11 pm**

I just noticed that when I use visibility flags to hide an entity during normal rendering, it also does not cast shadows.

I think what happens is that all objects that were visible during scene are visible during shadow rendering when you call it the way we do.
So prepareShadowTextures likely does not check for visibility again and just renders all objects that are already visible (and have shadow casting enabled).

Posted: **Fri Nov 15, 2013 3:12 pm**

I can disable shadow casting even for entities that are still visible during the normal render with my code.

Posted: **Fri Nov 15, 2013 3:14 pm**

Yeah. There really must be something different when calling prepareShadowTextures directly. Or, at least when calling it directly while normal rendering is suspended.

Posted: **Fri Nov 15, 2013 3:15 pm**

Look at where the scene manager code calls prepareShadowTextures internally. The viewport it's supplying as parameter is not the shadow texture viewport, it's the regular scene viewport.

Posted: **Sat Nov 16, 2013 9:49 am**

Oh my. Silly me.

Could've known that this was a different viewport as in your code.

But still,I thought it was so intuitive that I never doubted this was the used viewport. Definitely needs a comment for this

I will add that as soon as I get to my machine again.

Ogre Forums

Should prepareShadowTextures() ignore visibility flags?

Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?

Re: Should prepareShadowTextures() ignore visibility flags?