Shader - detect back face facing camera

A place for users of OGRE to discuss ideas and experiences of utilitising OGRE in their games / demos / applications.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Shader - detect back face facing camera

Post by kubatp »

Hi everybody,
my problem might be very simple for someone experienced in shaders, but I am fighting with it already for half a day:)
Basically I have a manualObject, which works as a fog above terrain. The fog material has scene_blend alpha_blend and depth_check off.
It all works fine but there are cases (specific camera angles) when I can see the faces of the fog which I shouldnt see. I need to somehow detect that these triangles should not be rendered and set its alpha to 0 in shader.

Do you have any ideas how to do this?

As far as I understand it, these triangles are rendered because the depth_check is off (and I need it off) and they are less transparent (more distinct) because the pixel is rendered with the fog triangles which are on the other side and should be rendered (that is why it is less transparent than the rest of the fog). I think that this means that these are actually the back faces of the triangle? How to detect that the rendered triangle faces the camera the other way and should not be rendered?

Thank you for any advice


Image
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 168

Re: Shader - detect back face facing camera

Post by xrgo »

cull_hardware <clockwise|anticlockwise|none> should be enough but if you use none you can detect the backface in pixel shader by using gl_FrontFacing in openGL
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

xrgo wrote: Mon Nov 13, 2017 8:02 pm cull_hardware <clockwise|anticlockwise|none> should be enough
I tried that. When I tried clockwise or none, the result is the same, however when I set it to anticlockwise, it did the exact opossite - it renders only those faces I would like to hide. Any ideas why the clockwise does not work?

Image
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 168

Re: Shader - detect back face facing camera

Post by xrgo »

then... I think the problem is that those are not backfaces, they are actually frontfaces and the just show because are behind another frontface, so its opacity adds up. If that's the case try disabling the transparency and use scene_blend add

Its a bit hard to see without moving the camera.. if you can post a video that would be great =)

edit: I read wrong, it seems that indeed are backfaces.. mmmm... its really strange
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

xrgo wrote: Mon Nov 13, 2017 9:00 pm edit: I read wrong, it seems that indeed are backfaces.. mmmm... its really strange
Ok, I tried the material setting with depth_check on and without cull_hardware setting (so it is default) and it looks like in the attachment. I am surprised that it is transparent when the depth_check and depth_write is turned on?

Any ideas?

Image
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

It seems like the problem is not in the back faces rendering but in rendering of two front faces. Basically when camera is in certain angles, it renders the upper part of the fog closer to the camera and it goes further to the lower part of the fog further from the camera. The back faces are not rendered however because two front faces are rendered and blended together the fog is less transparent.

I was trying to play with scene_blend, separate_scene_blend and separate_scene_blend_op as well, but it didnt fix the problem.

Is there a way how to avoid/set scene blending for certain render group but leave it the same for the rest of the scene? Or maybe reset the colour when the view "ray" goes through the back face?
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 168

Re: Shader - detect back face facing camera

Post by xrgo »

xrgo wrote: Mon Nov 13, 2017 9:00 pm I think the problem is that those are not backfaces, they are actually frontfaces and the just show because are behind another frontface, so its opacity adds up
kubatp wrote: Tue Nov 14, 2017 1:50 pm It seems like the problem is not in the back faces rendering but in rendering of two front faces
lol I was right...
the only solution that come to my mind is to render that mesh without any transparency in another texture and later in post process blend it to the rest of the scene (which was rendered in another texture), but it doesn't seems like a good solution (performance wise mostly)

maybe someone else has another idea?
al2950
OGRE Expert User
OGRE Expert User
Posts: 1227
Joined: Thu Dec 11, 2008 7:56 pm
Location: Bristol, UK
x 157

Re: Shader - detect back face facing camera

Post by al2950 »

If you are trying to create a ground fog effect, I would use a post process effect. I did a very quick google and came across this;
http://in2gpu.com/2014/07/22/create-fog-shader/

It seems to have a nice explanation of how to do fog as a post process and mentions ground fog at the very end.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

al2950 wrote: Tue Nov 14, 2017 4:18 pm If you are trying to create a ground fog effect, I would use a post process effect.
Hi,
thank you for that. I found this already months ago, but it is something else than I need:-/
User avatar
boyamer
Orc
Posts: 459
Joined: Sat Jan 24, 2009 11:16 am
Location: Italy
x 6

Re: Shader - detect back face facing camera

Post by boyamer »

Hi,
Try to digg around with depth bias values, maybe that would fix the issue.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

boyamer wrote: Thu Nov 23, 2017 1:42 pm Try to digg around with depth bias values, maybe that would fix the issue.
Hi,
thank you for the advice, but depth_bias probably cannot help here, because the depth_check is turned off :?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Shader - detect back face facing camera

Post by dark_sylinc »

It took me a while to understand what you were doing and trying to do.

The solution you're trying is not going to work. Alpha blending in order to work properly needs to be rendered back to front.
Right now the problem you have is that when you have the camera like this (ASCII art):

Code: Select all

    /\
C->/  \
  /    \
 /      \
 
Where C is the camera looking over a small mountain of fog, all if fine because it's rendereded back to front.
However when you move the camera to the other side of this mountain, unless you reorder every single triangle, now it's rendered front to back:

Code: Select all

    /\
   /  \ <-C
  /    \
 /      \
You could try to sort each triangle, but this could be expensive. It would be wise to change the approach or flatten the heights to prevent this from happening as much as possible, or better yet, force the camera to look always from above, not from oblique angles.

Alternatively, instead of alpha blending, you could try multiplication (scene_blend modulate) which should be commutative (a * b == b * a)
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

Hi Dark Sylinc,
thank you for joining this.
dark_sylinc wrote: Thu Nov 23, 2017 7:36 pm However when you move the camera to the other side of this mountain, unless you reorder every single triangle, now it's rendered front to back:
Would it change anything when I tell you that the scene (and camera movement) is done the way that viewer can never see it from the other side (only from top)?
dark_sylinc wrote: Thu Nov 23, 2017 7:36 pm flatten the heights
Yes, it would fix the problem. It was actually the initial approach, but it didnt look that nice and created some anomalies together with the terrain itself.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

kubatp wrote: Fri Nov 24, 2017 11:04 am
dark_sylinc wrote: Thu Nov 23, 2017 7:36 pm However when you move the camera to the other side of this mountain, unless you reorder every single triangle, now it's rendered front to back:
Would it change anything when I tell you that the scene (and camera movement) is done the way that viewer can never see it from the other side (only from top)?
Are you saying if I reorder the triangles the way that the further ones (from camera) are added to manual object first, it would fix the problem (if I cannot move the camera the other way, which would reverse the distance of triangles from the camera view)?
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

Is it a dumb question? Or I am wrong?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Shader - detect back face facing camera

Post by dark_sylinc »

You're not wrong. But I cannot say you'll get what you want. You're trying to solve a problem with a very specific approach. I cannot help you there.

If the camera rotates even a bit, there may be changes in how triangle order should be processed. It's very tricky and complicated.

Furthermore I noticed it looks like there's lighting applied to that fog, which could further explain the problems you're encountering.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

The best would probably be to reorganize it and try it, right?

I hope I dont have the lightning applied. I have in my material/technique/pass this

Code: Select all

lighting off
Why do you think there is lightning applied to the fog?
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

I was thinking about another solution to this.
I could detect in vertex shader, if camera is facing back face and if so, change the position of vertex (lower the y position).

Has anyone tried to detect triangle facing camera in shader? I am not a pro in geometry, so any help is appreciated.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1994
Joined: Sun Mar 30, 2014 2:51 pm
x 1074
Contact:

Re: Shader - detect back face facing camera

Post by paroj »

I think your mesh just have the wrong vertex winding for some of the triangles. It determines front/ backface and is readable in the fragment shader as
https://www.khronos.org/registry/OpenGL ... cing.xhtml

however you should be able to just use "cull_hardware". If anticlockwise gives you the backfaces, then they must not be visible with clockwise.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

paroj wrote: Tue Jan 09, 2018 8:45 pm I think your mesh just have the wrong vertex winding for some of the triangles. It determines front/ backface and is readable in the fragment shader as
https://www.khronos.org/registry/OpenGL ... cing.xhtml
Hi Paroj,
thank you for your answer. I am aware of gl_FrontFacing however I use CG shader language. Despite the fact that CG should support this function with the parameter type FACE, it doesnt. It is also reported here https://forums.ogre3d.org/viewtopic.php ... CE#p365728
The error is "error C5108: unknown semantics "FACE".

That is the reason why I ask how to detect this in shader by calculation.
I do not have a problem with culling. I need to modify vertices when the triangle is back faced.
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

There is no easy way to find this out?

What I try to find - I have a "terrain" mesh and once it detects (in vertex shader) that the camera is facing the back face of the mesh, it should change the height of the vertex so it doesnt face the back face (I attach screenshot for easier understading).
Image

What I need is to know how to calculate that face is backfacing the camera. I am aware that I need camera_position, normal of the face and probably also view_direction however I dont know how to combine these to find the information I need.
Thank you for any advices...
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

The solution I was thinking about (in previous post) also doesnt work in such a way that it would improve the visual result.

I was wondering if any kind of usage of separate_scene_blend and separate_scene_blend_op could help here?

What I basically try to achieve is limitation of alpha channel of the blending mesh when the ray from camera crosses the mesh more than once.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Shader - detect back face facing camera

Post by dark_sylinc »

kubatp wrote: Fri Jan 12, 2018 10:40 am Image
That's not the problem you're having, as that gets easily fixed with cull_hardware clockwise (or counterclockwise)

This is the problem you're having:
It's two front faces. The only solutions to that, as I already said is:
  1. Use an OIT algorithm (hard, You need shader knowledge)
  2. Ensure triangles are always drawn back to front (difficulty depends on how your camera plays, you may need to reorder the index buffer every frame)
  3. Use a blending mode that is order independent, such as scene_blend modulate or scene_blend add (easy)
kubatp
Gnome
Posts: 368
Joined: Tue Jan 06, 2009 1:12 pm
x 43

Re: Shader - detect back face facing camera

Post by kubatp »

dark_sylinc wrote: Sat Jan 20, 2018 12:31 am That's not the problem you're having, as that gets easily fixed with cull_hardware clockwise (or counterclockwise)
This is the problem you're having:
Hi Dark Sylinc,
you are right (and I already knew that this is the problem), but it doesnt mean that the solution I drafted is wrong. If you look at your draft, the problem would simply be solved if you lower the right peak (that is the drafted solution I was thinking about) - it would "remove" one front face from the rendering of those texels which were initially rendering two front faces.
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am Use an OIT algorithm (hard, You need shader knowledge)
Would you mind to tell me more about it (or send a link with description)? I was googling that but it seems like OIT is a set of different algorithms rather than one and you mind have one particular in mind?
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am Ensure triangles are always drawn back to front (difficulty depends on how your camera plays, you may need to reorder the index buffer every frame)
This is interesting, because I already have that. Basically I create the manual object from the most distant vertices to the closest ones (on X and Z axes). When camera doesnt rotate and look directly to NEGATIVE Z, it should always be rendered from back to front. But it doesnt help or maybe I need to specify something else?
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am Use a blending mode that is order independent, such as scene_blend modulate or scene_blend add (easy)
Yes, this is easy but how would I make the result transparent to see the meshes below?
paroj
OGRE Team Member
OGRE Team Member
Posts: 1994
Joined: Sun Mar 30, 2014 2:51 pm
x 1074
Contact:

Re: Shader - detect back face facing camera

Post by paroj »

kubatp wrote: Sat Jan 20, 2018 1:48 pm Would you mind to tell me more about it (or send a link with description)? I was googling that but it seems like OIT is a set of different algorithms rather than one and you mind have one particular in mind?
I guess the easiest to implement is https://docs.nvidia.com/gameworks/conte ... sample.htm
kubatp wrote: Sat Jan 20, 2018 1:48 pm Yes, this is easy but how would I make the result transparent to see the meshes below?
for instance use RENDER_QUEUE_SKIES_LATE
Post Reply