Shadows looking quite odd

ender81x · Post by **ender81x** » Sun Oct 10, 2004 12:11 pm

I thought I would post a few screnshots of the shadows I am seeing in my program to see if anyone can maybe see what I'm doing wrong. I am using SHADOWTYPE_STENCIL_MODULATIVE, and all the entities in my scene are set to cast shadows...

This is the same scene from two different angles.. thanks for the help.

Post by **sinbad** » Sun Oct 10, 2004 12:22 pm

Your meshes are not manifold. They have to be completely closed, which is to say every edge has to have exactly two triangles meeting at it - no more, no less.

ender81x · Post by **ender81x** » Sun Oct 10, 2004 12:36 pm

Is there any other possible reason? I believe my meshes do have only two triangles for every edge... each face is exactly 2 triangles, and the scene is comprised entirely of boxes (6 faces per submesh)... I am generating the meshes based on a .MAP file (like the file you would get from an editor like Quark), so my objects are defined as a set of planes, and the points of the mesh exist at the places those planes intersect... so is is possible that maybe my meshes are not entirely closed? Or is there some reason besides my mesh not being manifold?

ggambett · Post by **ggambett** » Mon Oct 11, 2004 3:17 pm

ender81x wrote:Is there any other possible reason?

Yes, this one. The patch hasn't been looked at yet, but you can download it and try for yourself.

Post by **sinbad** » Mon Oct 11, 2004 5:52 pm

Good point, I forgot about that. I'm starting to look at patches now so it shouldn't be too long before this is applied.

ender81x · Post by **ender81x** » Mon Oct 11, 2004 7:58 pm

ggambett, Worked like a charm! Thanks a lot for pointing me to that patch.

ender81x · Post by **ender81x** » Tue Oct 12, 2004 2:49 pm

Okay, the patch has fixed my shadows up almost completely... the last problem I am seeing is that it looks almost like my camera node itself is making a hole in the shadow:

The hole moves into the position where the rays from the casting light would pass through my camera... it's looks kind of like my camera has a cylinder mesh attached to it (with a reverse shadow effect...) It happens also with an ogre sample mesh (such as athene.mesh) ... other than that though, it looks great now

Post by **sinbad** » Tue Oct 12, 2004 5:16 pm

I would guess far plane clipping of the shadow volume. See my previous comments / the manual about being careful about that on older hardware. I assume you are running this on oldish (gf2/gf4mx) hardware since you wouldn't have had the problem that ggambett fixed in his patch otherwise. This hardware does not support infinite far planes (or infinite shadow volume extrusion), therefore you have to be careful about far plane clipping (and also lights getting too close to the objects).

If it isn't that, maybe the dark cap is not getting positioned correctly in your case. You can test that by taking a simpler scene with a relatively small light volume, then flying the camera into the shadow of a mesh and looking away from the light. If you see artefacts in that case, it might be a dark cap problem, although I test this scenario quite a lot without problems here.

ender81x · Post by **ender81x** » Tue Oct 12, 2004 7:40 pm

Okay, I went ahead and created a simple scene (just a floor plane, with a cube floating in the middle, and one light above and to the right). I do get a hole in the shadow (the hole is fully lit, with no shadow), and it seems to move with my camera, only when my camera is between the light and the shadow. Also, I don't know if this helps, or it is normal, but the background color (i have it set to green) gets shadowed as well. Since I won't ever have an open background it doesn't really matter, but I just wanted to throw that in in case it helped out. So is this an issue with my video card being a piece of crap? The machine I am testing on right now has a ATI Mobility Radeon 9200 (it's a laptop).

And in case it's useful:

11:33:29: *** Starting Win32GL Subsystem ***
11:33:29: Created Win32Window 'OGRE Render Window' : 1024x768, 32bpp
11:33:29: GL_VERSION = 1.3.3878 WinXP Release
11:33:29: GL_VENDOR = ATI Technologies Inc.
11:33:29: GL_EXTENSIONS = GL_ARB_multitexture GL_EXT_texture_env_add GL_EXT_compiled_vertex_array GL_S3_s3tc GL_ARB_point_parameters GL_ARB_texture_border_clamp GL_ARB_texture_compression GL_ARB_texture_cube_map GL_ARB_texture_env_add GL_ARB_texture_env_combine GL_ARB_texture_env_crossbar GL_ARB_texture_env_dot3 GL_ARB_texture_mirrored_repeat GL_ARB_transpose_matrix GL_ARB_vertex_blend GL_ARB_vertex_buffer_object GL_ARB_vertex_program GL_ARB_window_pos GL_ATI_element_array GL_ATI_envmap_bumpmap GL_ATI_fragment_shader GL_ATI_map_object_buffer GL_ATI_texture_env_combine3 GL_ATI_texture_mirror_once GL_ATI_vertex_array_object GL_ATI_vertex_attrib_array_object GL_ATI_vertex_streams GL_ATIX_texture_env_combine3 GL_ATIX_texture_env_route GL_ATIX_vertex_shader_output_point_size GL_EXT_abgr GL_EXT_bgra GL_EXT_blend_color GL_EXT_blend_func_separate GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_clip_volume_hint GL_EXT_draw_range_elements GL_EXT_fog_coord GL_EXT_multi_draw_arrays GL_EXT_packed_pixels GL_EXT_point_parameters GL_EXT_rescale_normal GL_EXT_secondary_color GL_EXT_separate_specular_color GL_EXT_stencil_wrap GL_EXT_texgen_reflection GL_EXT_texture3D GL_EXT_texture_compression_s3tc GL_EXT_texture_cube_map GL_EXT_texture_edge_clamp GL_EXT_texture_env_combine GL_EXT_texture_env_dot3 GL_EXT_texture_filter_anisotropic GL_EXT_texture_lod_bias GL_EXT_texture_object GL_EXT_texture_rectangle GL_EXT_vertex_array GL_EXT_vertex_shader GL_HP_occlusion_test GL_NV_texgen_reflection GL_NV_blend_square GL_NV_occlusion_query GL_SGI_color_matrix GL_SGIS_texture_edge_clamp GL_SGIS_texture_border_clamp GL_SGIS_texture_lod GL_SGIS_generate_mipmap GL_SGIS_multitexture GL_SUN_multi_draw_arrays GL_WIN_swap_hint WGL_EXT_extensions_string WGL_EXT_swap_control
11:33:29: ***************************
*** GL Renderer Started ***
***************************
11:33:29: RenderSystem capabilities
11:33:29: -------------------------
11:33:29: * Hardware generation of mipmaps: no
11:33:29: * Texture blending: yes
11:33:29: * Anisotropic texture filtering: yes
11:33:29: * Dot product texture operation: yes
11:33:29: * Cube mapping: yes
11:33:29: * Hardware stencil buffer: yes
11:33:29: - Stencil depth: 8
11:33:29: - Two sided stencil support: no
11:33:29: - Wrap stencil values: yes
11:33:29: * Hardware vertex / index buffers: yes
11:33:29: * Vertex programs: yes
11:33:29: - Max vertex program version: arbvp1
11:33:29: * Fragment programs: yes
11:33:29: - Max fragment program version: ps_1_4
11:33:29: * Texture Compression: yes
11:33:29: - DXT: yes
11:33:29: - VTC: no
11:33:29: * Scissor Rectangle: yes
11:33:29: * Hardware Occlusion Query: yes
11:33:29: * User clip planes: yes

Post by **sinbad** » Tue Oct 12, 2004 10:23 pm

I'm quite amazed that the 9200 doesn't have two-sided stencilling. Anyway, that's not a bad card, so it sounds to me like it's a dark cap issue. You have vertex programs available, so I'm not surprised it only happens when the camera is between the light and the object, since this is the only time the dark cap is required when doing vertex program extrusion.

I've tried to recreate this, and have not been able to. Even turning off 2-sided stencilling here, with and without the patch from ggambett, does not create it.
Either on GL or Direct3D.

The only thing I can think of is that for some reason your card is not implementing infinite projection properly. Because you have vertex programs, the volume is being extruded to infinity, and an infinite projection matrix is used to allow it to do that without getting the dark cap clipped at the far plane (this is what would happen on older cards if your extrusion distance was too big). GL always supports infinite projection though, it's only certain D3D drivers that don't. To test this, turn off vertex programs (by commenting out the mCapabilities->setCapability(RSC_VERTEX_PROGRAM); entries in your rendersystem) - this will make it use finite projection (flawed in itself, but if you keep your light attenuation range down and the light not too close to the objects it should be ok). If you get no gap, then it's the infinite projection that's at fault, which I'm afraid is a driver bug.

ender81x · Post by **ender81x** » Wed Oct 13, 2004 5:40 pm

Okay, hopefully this is the last time I have to bug you about this... I commented out mCapabilities->setCapability(RSC_VERTEX_PROGRAM); from my GL rendersystem and rebuilt my ogre source. I am stilling having the same problem... however, I did notice something: each of the faces of my column is built with 2 tris forming a rectangle... When i see the shadow gaps, it looks like one triangle is shading correctly, but the other one is not casting a shadow... so half of each face is not working properly from certain angles... The mesh is all being culled correctly, so i don't think my vertex orientation is the wrong way, but could normals, or something like that have to do with why one tri from each face isn't working? In any case, I seem to be able to set up the lights, etc in a way that I won't have to deal with the problem, so if I can't get this working I can always work around it.. Thanks again for all the help.

Post by **sinbad** » Wed Oct 13, 2004 11:04 pm

Hmm, I'd be susupecting the mesh at this point. Do you get the same issues with our column mesh (the one that's used in the shadow demo that comes with Ogre)? That's built the same way, but I've never witnessed this effect with it.

ender81x · Post by **ender81x** » Thu Oct 14, 2004 3:54 am

Well I'm pretty sure it's my mesh at this point too... the normals are somehow not getting calculated correctly, so I'm in the process of rewriting the code that generates the meshes... thanks for all the help though, and sorry for having you chase the problem around... But I did learn a lot throughout the process.

ender81x · Post by **ender81x** » Fri Oct 15, 2004 7:58 pm

Okay... I have rewritten the code to generate the mesh for my level... and the shadow is still all messed up... the code is a lot prettier now though, so I guess it wasn't all a waste.
Anyway, here are two more screenshots of how it looks at different angles, and I uploaded the source to my project too, if anyone wants to take a look at it. There is a lot of it, but the exe is already compiled in the bin directory in case you just want to see it in action and see if that gives you any more of an idea what's going on.

In the screenshots you can see where the shadows look fine, and then where it is messed up when i move into it with the camera... However, the column.mesh file looks fine all the time so I'm afraid the error is mine somewhere.

source code (It's 3 MB, i left the DLLs in there so you can run it right out the box):
http://www.indpoint.com/screenshots/ogre_map.zip

The source code basically takes a MAP file (which is in text format), and generates the geometry for it... I tried to comment it a lot, but forgive me if it's a mess (or if it's laughably amateur)

Lt.CYX · Post by **Lt.CYX** » Tue Jan 18, 2005 7:29 pm

just a thought... but that column you are generating has its ends capped, right?

Post by **haffax** » Tue Jan 18, 2005 8:16 pm

This thread is like 3 month old. No need to exhume it.

The problem is probably solved by now. If not, 0.15.2 is *much* better in handling shadows.