Assert fail on Linux with Texture Shadows

LAva · Post by **LAva** » Tue Sep 25, 2007 10:46 pm

I have some problems with texture shadows on Linux (Ubuntu 7.04).
The program throws an assert_fail in some situations which I can reproduce but I don't know why the program doesn't crash in other situations.

One of the situations where the program crashes is if I load a specific model twice in the scene (I didn't test this in an empty scene, but I think it is a combination of the whole scene and this two models). But there are other constellations where this error is thrown.

I tested my project on windows without any problems!

I'm using the Ogre CVS version but this behavior was also there on earlier releases (1.4.1).

Maybe somebody has a clue if this is a bug or if I'm doing something wrong.

Here is the call stack:

Code: Select all

#0 0xffffe410	__kernel_vsyscall() (??:??)
#1 0xb729a875	raise() (/lib/tls/i686/cmov/libc.so.6:??)
#2 0xb729c201	abort() (/lib/tls/i686/cmov/libc.so.6:??)
#3 0xb7293b6e	__assert_fail() (/lib/tls/i686/cmov/libc.so.6:??)
#4 0xb7dc32ef	Ogre::TextureUnitState::operator=() (../lib_linux/libOgreMain-1.4.4.so:??)
#5 0xb7d4f158	Ogre::SceneManager::deriveShadowCasterPass() (../lib_linux/libOgreMain-1.4.4.so:??)
#6 0xb7d4ffec	Ogre::SceneManager::_setPass() (../lib_linux/libOgreMain-1.4.4.so:??)
#7 0xb7d49127	Ogre::SceneManager::BRAND NAME::visit() (../lib_linux/libOgreMain-1.4.4.so:??)
#8 0xb7d13224	Ogre::QueuedRenderableCollection::acceptVisitorGrouped() (../lib_linux/libOgreMain-1.4.4.so:??)
#9 0xb7d13b51	Ogre::QueuedRenderableCollection::acceptVisitor() (../lib_linux/libOgreMain-1.4.4.so:??)
#10 0xb7d49711	Ogre::SceneManager::renderObjects() (../lib_linux/libOgreMain-1.4.4.so:??)
#11 0xb7d4bc7f	Ogre::SceneManager::renderTextureShadowCasterQueueGroupObjects() (../lib_linux/libOgreMain-1.4.4.so:??)
#12 0xb7d49017	Ogre::SceneManager::_renderQueueGroupObjects() (../lib_linux/libOgreMain-1.4.4.so:??)
#13 0xb7d4ac5d	Ogre::SceneManager::renderVisibleObjectsDefaultSequence() (../lib_linux/libOgreMain-1.4.4.so:??)
#14 0xb7d49778	Ogre::SceneManager::_renderVisibleObjects() (../lib_linux/libOgreMain-1.4.4.so:??)
#15 0xb777a34d	Ogre::TerrainSceneManager::_renderVisibleObjects() (../lib_linux/Plugin_OctreeSceneManager.so:??)
#16 0xb7d4a930	Ogre::SceneManager::_renderScene() (../lib_linux/libOgreMain-1.4.4.so:??)
#17 0xb777a706	Ogre::TerrainSceneManager::_renderScene() (../lib_linux/Plugin_OctreeSceneManager.so:??)
#18 0xb7b8d76e	Ogre::Camera::_renderScene() (../lib_linux/libOgreMain-1.4.4.so:??)
#19 0xb7dd05b0	Ogre::Viewport::update() (../lib_linux/libOgreMain-1.4.4.so:??)
#20 0xb7d1e593	Ogre::RenderTarget::update() (../lib_linux/libOgreMain-1.4.4.so:??)
#21 0xb7d4ca28	Ogre::SceneManager::prepareShadowTextures() (../lib_linux/libOgreMain-1.4.4.so:??)
#22 0xb7d4aa37	Ogre::SceneManager::_renderScene() (../lib_linux/libOgreMain-1.4.4.so:??)
#23 0xb777a706	Ogre::TerrainSceneManager::_renderScene() (../lib_linux/Plugin_OctreeSceneManager.so:??)
#24 0xb7b8d76e	Ogre::Camera::_renderScene() (../lib_linux/libOgreMain-1.4.4.so:??)
#25 0xb7dd05b0	Ogre::Viewport::update() (../lib_linux/libOgreMain-1.4.4.so:??)
#26 0xb7d1e593	Ogre::RenderTarget::update() (../lib_linux/libOgreMain-1.4.4.so:??)
#27 0xb7d225d9	Ogre::RenderWindow::update() (../lib_linux/libOgreMain-1.4.4.so:??)
#28 0xb7d2250c	Ogre::RenderWindow::update() (../lib_linux/libOgreMain-1.4.4.so:??)
#29 0xb7d181db	Ogre::RenderSystem::_updateAllRenderTargets() (../lib_linux/libOgreMain-1.4.4.so:??)

The code where I activate the shadows:

Code: Select all

      mSceneMgr->setShadowTechnique(SHADOWTYPE_TEXTURE_MODULATIVE);
    mSceneMgr->setShadowTextureSize(512);
    mSceneMgr->setShadowTextureCount(10);
    mSceneMgr->setShadowColour(ColourValue(0.7,0.7,0.7));
    mSceneMgr->setShadowFarDistance(50);
    mSceneMgr->setShadowTextureSelfShadow(false);

Post by **tuan kuranes** » Wed Sep 26, 2007 10:49 am

What is needed is more where the assert is raised... did you compile with debug symbols ? can't you read the assert line and see what it's saying ?
That's what assert are for... precise information on requisites to make code working.

I tested my project on windows without any problems!

using opengl too ?
Anyway linux driver are not same quality as windows one...

mSceneMgr->setShadowTextureSize(512);
mSceneMgr->setShadowTextureCount(10);

10 shadow texture means 10 rendertarget of 512x512... that's huge... and I'm not sure it's that supported either by hardware or Ogre (check Ogre log "max render target" for instance...)

LAva · Post by **LAva** » Sat Sep 29, 2007 6:42 pm

There are just two asserts in this method where the error happens.

Code: Select all

    TextureUnitState & TextureUnitState::operator = ( 
        const TextureUnitState &oth )
    {
        assert(mAnimController == 0);
        assert(mEffects.empty());

I'm not sure which one causes the program stop.
I always thought, if I compile Ogre on Linux there are already debug symbols in the libs, but I also tried to set the debug flag manually.
But if I try to debug my program I never get the line number in the Ogre libs. Do the source files have to be on a specific place for this to work?

I tested my project on windows without any problems!
using opengl too ?
Anyway linux driver are not same quality as windows one...

Yes! I tried my program using opengl in Windows as well. Worked without any problems.

10 shadow texture means 10 rendertarget of 512x512... that's huge... and I'm not sure it's that supported either by hardware or Ogre (check Ogre log "max render target" for instance...)

I'm using just one shadow texture now, but the error stays the same.

Post by **tuan kuranes** » Mon Oct 01, 2007 2:52 pm

run it using GDB or any front end to gdb, and you'll get much more information, stack/local var/etc...

LAva · Post by **LAva** » Mon Oct 01, 2007 7:35 pm

The failed assert happens on mEffects.empty() in the TextureUnitState.

Code: Select all

TextureUnitState & TextureUnitState::operator = (
        const TextureUnitState &oth )
    {
        assert(mAnimController == 0);
        assert(mEffects.empty());

Post by **sinbad** » Tue Oct 02, 2007 5:14 pm

We'll need the context in the method that's calling this operator.

SuperMegaMau · Post by **SuperMegaMau** » Mon Jan 28, 2008 11:30 am

hi,

I had the exact same problem, and found out that disabling TextureUnitState effects (transforms) solved the problem. Dont know if this is the issue, but works for me.

TMT · Post by **TMT** » Tue Oct 21, 2008 3:23 pm

I have begun to see this assert also. Mine asserts when being called from SceneManager::deriveShadowCasterPass(). I am studying the code to see exactly what is the cause, but I already see an inconsitency that is suspect:

In SceneManager::deriveShadowCasterPass() you have:

Code: Select all

			// Copy texture state, shift up one since 0 is shadow texture
			unsigned short origPassTUCount = pass->getNumTextureUnitStates();
			for (unsigned short t = 0; t < origPassTUCount; ++t)
			{
				TextureUnitState* tex;
				if (retPass->getNumTextureUnitStates() <= t)
				{
					tex = retPass->createTextureUnitState();
				}
				else
				{
					tex = retPass->getTextureUnitState(t);
				}
				// copy base state
				(*tex) = *(pass->getTextureUnitState(t));
				// override colour function
				tex->setColourOperationEx(LBX_SOURCE1, LBS_MANUAL, LBS_CURRENT,
					isShadowTechniqueAdditive()? ColourValue::Black : mShadowColour);
			}

but in SceneManager::deriveShadowReceiverPass() you have:

Code: Select all

            // Copy texture state, shift up one since 0 is shadow texture
            unsigned short origPassTUCount = pass->getNumTextureUnitStates();
            for (unsigned short t = 0; t < origPassTUCount; ++t)
            {
                unsigned short targetIndex = t+1;
                TextureUnitState* tex;
                if (retPass->getNumTextureUnitStates() <= targetIndex)
                {
                    tex = retPass->createTextureUnitState();
                }
                else
                {
                    tex = retPass->getTextureUnitState(targetIndex);
                }
                (*tex) = *(pass->getTextureUnitState(t));
				// If programmable, have to adjust the texcoord sets too
				// D3D insists that texcoordsets match tex unit in programmable mode
				if (retPass->hasVertexProgram())
					tex->setTextureCoordSet(targetIndex);
            }

These appear to be doing roughly the same thing, but note he shift in the index in the latter. The latter sure seems more correct according to the comment. Anyone familiar with this code know if the former is incorrect or not?

TMT · Post by **TMT** » Fri Oct 24, 2008 5:05 pm

Well, I thought someone would comment. I did change the loop in SceneManager::deriveShadowCasterPass() to be like SceneManager::deriveShadowReceiverPass() and I no longer get the assert. I'm assuming this is a bug that's been there for a while. I'm using 1.4.9, but it appears to be in all subsequent versions as well.

BTW, I'm using Windows and not Linux as the original creator of this thread was using.

Post by **sinbad** » Sat Oct 25, 2008 4:45 pm

Only just saw this. The 2 blocks of code are supposed to be different, the comments have just been inappropriately copied.

In the caster pass, there is in fact no shadow texture, because you're rendering to the shadow texture. Therefore, no offset. There is only an extra texture unit in the receiver pass, since that's where you're pulling in the shadow texture.

I still haven't seen enough detail about this problem to be able to propose a cause, and I haven't come across it yet.

TMT · Post by **TMT** » Sun Oct 26, 2008 7:27 pm

Thanks sinbad for looking at it. I can give some more information in hopes it can clue you or someone in to the specific problem. The material it is asserting on is a flat scrolling (scroll_anim 0.0 0.002) water surface. The receiving TextureUnitState already has the ET_VSCROLL in its mEffects map and hence the assert in the assignment operator fails. None of the other textures have problems...just this one and it's the only one with an effect on it.

As I write this I suddenly am thinking that I can avoid the assert by making sure this water surface doesn't cast a shadow as it really shouldn't anyway, but that wouldn't resolve the issue for a case where an animated texture is on an object that should cast a shadow.

Perhaps the TextureUnitState assignment operator shouldn't care if there are already things in mEffects and simply clear it before copying them.

Shadow code is not my area of expertise, so I'm trying to resolve this somewhat blindly. I apologize for being somewhat dense in this area.

syedhs · Post by **syedhs** » Sat Oct 03, 2009 9:11 am

Just a note, I have hit the same exact error (assert on mEffects.empty() for material with scroll_anim in it) and the workaround is also to disable shadow casting for the entities.

scrawl · Post by **scrawl** » Mon Jan 05, 2015 11:04 pm

5 years later, the bug is still unsolved. I wonder why it was never reported to the issue tracker?

I have the same crash in my project and it occurs 100% of the time when using shadows and a material with a spherical environment map.

Here is my fix attempt: https://bitbucket.org/sinbad/ogre/pull- ... unitstates

Ogre Forums

Assert fail on Linux with Texture Shadows

Assert fail on Linux with Texture Shadows

Re: Assert fail on Linux with Texture Shadows

Re: Assert fail on Linux with Texture Shadows