Page 1 of 2

Shader - detect back face facing camera

Posted: Mon Nov 13, 2017 6:55 pm
by kubatp
Hi everybody,
my problem might be very simple for someone experienced in shaders, but I am fighting with it already for half a day:)
Basically I have a manualObject, which works as a fog above terrain. The fog material has scene_blend alpha_blend and depth_check off.
It all works fine but there are cases (specific camera angles) when I can see the faces of the fog which I shouldnt see. I need to somehow detect that these triangles should not be rendered and set its alpha to 0 in shader.

Do you have any ideas how to do this?

As far as I understand it, these triangles are rendered because the depth_check is off (and I need it off) and they are less transparent (more distinct) because the pixel is rendered with the fog triangles which are on the other side and should be rendered (that is why it is less transparent than the rest of the fog). I think that this means that these are actually the back faces of the triangle? How to detect that the rendered triangle faces the camera the other way and should not be rendered?

Thank you for any advice


Image

Re: Shader - detect back face facing camera

Posted: Mon Nov 13, 2017 8:02 pm
by xrgo
cull_hardware <clockwise|anticlockwise|none> should be enough but if you use none you can detect the backface in pixel shader by using gl_FrontFacing in openGL

Re: Shader - detect back face facing camera

Posted: Mon Nov 13, 2017 8:26 pm
by kubatp
xrgo wrote: Mon Nov 13, 2017 8:02 pm cull_hardware <clockwise|anticlockwise|none> should be enough
I tried that. When I tried clockwise or none, the result is the same, however when I set it to anticlockwise, it did the exact opossite - it renders only those faces I would like to hide. Any ideas why the clockwise does not work?

Image

Re: Shader - detect back face facing camera

Posted: Mon Nov 13, 2017 9:00 pm
by xrgo
then... I think the problem is that those are not backfaces, they are actually frontfaces and the just show because are behind another frontface, so its opacity adds up. If that's the case try disabling the transparency and use scene_blend add

Its a bit hard to see without moving the camera.. if you can post a video that would be great =)

edit: I read wrong, it seems that indeed are backfaces.. mmmm... its really strange

Re: Shader - detect back face facing camera

Posted: Mon Nov 13, 2017 10:56 pm
by kubatp
xrgo wrote: Mon Nov 13, 2017 9:00 pm edit: I read wrong, it seems that indeed are backfaces.. mmmm... its really strange
Ok, I tried the material setting with depth_check on and without cull_hardware setting (so it is default) and it looks like in the attachment. I am surprised that it is transparent when the depth_check and depth_write is turned on?

Any ideas?

Image

Re: Shader - detect back face facing camera

Posted: Tue Nov 14, 2017 1:50 pm
by kubatp
It seems like the problem is not in the back faces rendering but in rendering of two front faces. Basically when camera is in certain angles, it renders the upper part of the fog closer to the camera and it goes further to the lower part of the fog further from the camera. The back faces are not rendered however because two front faces are rendered and blended together the fog is less transparent.

I was trying to play with scene_blend, separate_scene_blend and separate_scene_blend_op as well, but it didnt fix the problem.

Is there a way how to avoid/set scene blending for certain render group but leave it the same for the rest of the scene? Or maybe reset the colour when the view "ray" goes through the back face?

Re: Shader - detect back face facing camera

Posted: Tue Nov 14, 2017 3:50 pm
by xrgo
xrgo wrote: Mon Nov 13, 2017 9:00 pm I think the problem is that those are not backfaces, they are actually frontfaces and the just show because are behind another frontface, so its opacity adds up
kubatp wrote: Tue Nov 14, 2017 1:50 pm It seems like the problem is not in the back faces rendering but in rendering of two front faces
lol I was right...
the only solution that come to my mind is to render that mesh without any transparency in another texture and later in post process blend it to the rest of the scene (which was rendered in another texture), but it doesn't seems like a good solution (performance wise mostly)

maybe someone else has another idea?

Re: Shader - detect back face facing camera

Posted: Tue Nov 14, 2017 4:18 pm
by al2950
If you are trying to create a ground fog effect, I would use a post process effect. I did a very quick google and came across this;
http://in2gpu.com/2014/07/22/create-fog-shader/

It seems to have a nice explanation of how to do fog as a post process and mentions ground fog at the very end.

Re: Shader - detect back face facing camera

Posted: Tue Nov 14, 2017 7:33 pm
by kubatp
al2950 wrote: Tue Nov 14, 2017 4:18 pm If you are trying to create a ground fog effect, I would use a post process effect.
Hi,
thank you for that. I found this already months ago, but it is something else than I need:-/

Re: Shader - detect back face facing camera

Posted: Thu Nov 23, 2017 1:42 pm
by boyamer
Hi,
Try to digg around with depth bias values, maybe that would fix the issue.

Re: Shader - detect back face facing camera

Posted: Thu Nov 23, 2017 2:59 pm
by kubatp
boyamer wrote: Thu Nov 23, 2017 1:42 pm Try to digg around with depth bias values, maybe that would fix the issue.
Hi,
thank you for the advice, but depth_bias probably cannot help here, because the depth_check is turned off :?

Re: Shader - detect back face facing camera

Posted: Thu Nov 23, 2017 7:36 pm
by dark_sylinc
It took me a while to understand what you were doing and trying to do.

The solution you're trying is not going to work. Alpha blending in order to work properly needs to be rendered back to front.
Right now the problem you have is that when you have the camera like this (ASCII art):

Code: Select all

    /\
C->/  \
  /    \
 /      \
 
Where C is the camera looking over a small mountain of fog, all if fine because it's rendereded back to front.
However when you move the camera to the other side of this mountain, unless you reorder every single triangle, now it's rendered front to back:

Code: Select all

    /\
   /  \ <-C
  /    \
 /      \
You could try to sort each triangle, but this could be expensive. It would be wise to change the approach or flatten the heights to prevent this from happening as much as possible, or better yet, force the camera to look always from above, not from oblique angles.

Alternatively, instead of alpha blending, you could try multiplication (scene_blend modulate) which should be commutative (a * b == b * a)

Re: Shader - detect back face facing camera

Posted: Fri Nov 24, 2017 11:04 am
by kubatp
Hi Dark Sylinc,
thank you for joining this.
dark_sylinc wrote: Thu Nov 23, 2017 7:36 pm However when you move the camera to the other side of this mountain, unless you reorder every single triangle, now it's rendered front to back:
Would it change anything when I tell you that the scene (and camera movement) is done the way that viewer can never see it from the other side (only from top)?
dark_sylinc wrote: Thu Nov 23, 2017 7:36 pm flatten the heights
Yes, it would fix the problem. It was actually the initial approach, but it didnt look that nice and created some anomalies together with the terrain itself.

Re: Shader - detect back face facing camera

Posted: Mon Nov 27, 2017 11:50 pm
by kubatp
kubatp wrote: Fri Nov 24, 2017 11:04 am
dark_sylinc wrote: Thu Nov 23, 2017 7:36 pm However when you move the camera to the other side of this mountain, unless you reorder every single triangle, now it's rendered front to back:
Would it change anything when I tell you that the scene (and camera movement) is done the way that viewer can never see it from the other side (only from top)?
Are you saying if I reorder the triangles the way that the further ones (from camera) are added to manual object first, it would fix the problem (if I cannot move the camera the other way, which would reverse the distance of triangles from the camera view)?

Re: Shader - detect back face facing camera

Posted: Wed Dec 06, 2017 5:59 pm
by kubatp
Is it a dumb question? Or I am wrong?

Re: Shader - detect back face facing camera

Posted: Wed Dec 06, 2017 9:06 pm
by dark_sylinc
You're not wrong. But I cannot say you'll get what you want. You're trying to solve a problem with a very specific approach. I cannot help you there.

If the camera rotates even a bit, there may be changes in how triangle order should be processed. It's very tricky and complicated.

Furthermore I noticed it looks like there's lighting applied to that fog, which could further explain the problems you're encountering.

Re: Shader - detect back face facing camera

Posted: Wed Dec 06, 2017 10:21 pm
by kubatp
The best would probably be to reorganize it and try it, right?

I hope I dont have the lightning applied. I have in my material/technique/pass this

Code: Select all

lighting off
Why do you think there is lightning applied to the fog?

Re: Shader - detect back face facing camera

Posted: Tue Jan 09, 2018 4:08 pm
by kubatp
I was thinking about another solution to this.
I could detect in vertex shader, if camera is facing back face and if so, change the position of vertex (lower the y position).

Has anyone tried to detect triangle facing camera in shader? I am not a pro in geometry, so any help is appreciated.

Re: Shader - detect back face facing camera

Posted: Tue Jan 09, 2018 8:45 pm
by paroj
I think your mesh just have the wrong vertex winding for some of the triangles. It determines front/ backface and is readable in the fragment shader as
https://www.khronos.org/registry/OpenGL ... cing.xhtml

however you should be able to just use "cull_hardware". If anticlockwise gives you the backfaces, then they must not be visible with clockwise.

Re: Shader - detect back face facing camera

Posted: Tue Jan 09, 2018 9:17 pm
by kubatp
paroj wrote: Tue Jan 09, 2018 8:45 pm I think your mesh just have the wrong vertex winding for some of the triangles. It determines front/ backface and is readable in the fragment shader as
https://www.khronos.org/registry/OpenGL ... cing.xhtml
Hi Paroj,
thank you for your answer. I am aware of gl_FrontFacing however I use CG shader language. Despite the fact that CG should support this function with the parameter type FACE, it doesnt. It is also reported here https://forums.ogre3d.org/viewtopic.php ... CE#p365728
The error is "error C5108: unknown semantics "FACE".

That is the reason why I ask how to detect this in shader by calculation.
I do not have a problem with culling. I need to modify vertices when the triangle is back faced.

Re: Shader - detect back face facing camera

Posted: Fri Jan 12, 2018 10:40 am
by kubatp
There is no easy way to find this out?

What I try to find - I have a "terrain" mesh and once it detects (in vertex shader) that the camera is facing the back face of the mesh, it should change the height of the vertex so it doesnt face the back face (I attach screenshot for easier understading).
Image

What I need is to know how to calculate that face is backfacing the camera. I am aware that I need camera_position, normal of the face and probably also view_direction however I dont know how to combine these to find the information I need.
Thank you for any advices...

Re: Shader - detect back face facing camera

Posted: Fri Jan 19, 2018 10:50 pm
by kubatp
The solution I was thinking about (in previous post) also doesnt work in such a way that it would improve the visual result.

I was wondering if any kind of usage of separate_scene_blend and separate_scene_blend_op could help here?

What I basically try to achieve is limitation of alpha channel of the blending mesh when the ray from camera crosses the mesh more than once.

Re: Shader - detect back face facing camera

Posted: Sat Jan 20, 2018 12:31 am
by dark_sylinc
kubatp wrote: Fri Jan 12, 2018 10:40 am Image
That's not the problem you're having, as that gets easily fixed with cull_hardware clockwise (or counterclockwise)

This is the problem you're having:
It's two front faces. The only solutions to that, as I already said is:
  1. Use an OIT algorithm (hard, You need shader knowledge)
  2. Ensure triangles are always drawn back to front (difficulty depends on how your camera plays, you may need to reorder the index buffer every frame)
  3. Use a blending mode that is order independent, such as scene_blend modulate or scene_blend add (easy)

Re: Shader - detect back face facing camera

Posted: Sat Jan 20, 2018 1:48 pm
by kubatp
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am That's not the problem you're having, as that gets easily fixed with cull_hardware clockwise (or counterclockwise)
This is the problem you're having:
Hi Dark Sylinc,
you are right (and I already knew that this is the problem), but it doesnt mean that the solution I drafted is wrong. If you look at your draft, the problem would simply be solved if you lower the right peak (that is the drafted solution I was thinking about) - it would "remove" one front face from the rendering of those texels which were initially rendering two front faces.
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am Use an OIT algorithm (hard, You need shader knowledge)
Would you mind to tell me more about it (or send a link with description)? I was googling that but it seems like OIT is a set of different algorithms rather than one and you mind have one particular in mind?
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am Ensure triangles are always drawn back to front (difficulty depends on how your camera plays, you may need to reorder the index buffer every frame)
This is interesting, because I already have that. Basically I create the manual object from the most distant vertices to the closest ones (on X and Z axes). When camera doesnt rotate and look directly to NEGATIVE Z, it should always be rendered from back to front. But it doesnt help or maybe I need to specify something else?
dark_sylinc wrote: Sat Jan 20, 2018 12:31 am Use a blending mode that is order independent, such as scene_blend modulate or scene_blend add (easy)
Yes, this is easy but how would I make the result transparent to see the meshes below?

Re: Shader - detect back face facing camera

Posted: Sat Jan 20, 2018 6:40 pm
by paroj
kubatp wrote: Sat Jan 20, 2018 1:48 pm Would you mind to tell me more about it (or send a link with description)? I was googling that but it seems like OIT is a set of different algorithms rather than one and you mind have one particular in mind?
I guess the easiest to implement is https://docs.nvidia.com/gameworks/conte ... sample.htm
kubatp wrote: Sat Jan 20, 2018 1:48 pm Yes, this is easy but how would I make the result transparent to see the meshes below?
for instance use RENDER_QUEUE_SKIES_LATE