Few serious questions for Ogre-Next, moving Stunt Rally

[dark_sylinc]

That was a bug. Thanks! Fixed.

Every user-available setting must be controllable through JSON materials.

[Crystal Hammer]

So area approx lights aren't working with Terra right?
Just tried doing that under car glow light.
Works okay on objects, but gets weird on terrain, tere is a negative light halo or some effect around and it's bad.
I tried with and w/o light texture.
It's not important for me at all, but would be cool if it worked.

1339`s.jpg

1219`s.jpg

1269`s.jpg

[dark_sylinc]

I never tried, but since it inherits the code from PBS, I can't think of a reason it wouldn't work.

It probably has some normals issue or something like that. Could you edit the Terrain sample to add an area light and post the patch so I can take a look?

[Crystal Hammer]

Okay I added in createScene01, code below: test area light
It's a big skyblue area light above (where cube is), and does that bad red glow on terrain. While houses are lit okay.

03_16-08-30.jpg

Tutorial_TerrainGameState.cpp

[Crystal Hammer]

I never tried, but since it inherits the code from PBS, I can't think of a reason it wouldn't work.

Well if you never tried, then that's already a good reason to assume it doesn't work. After all, there is test driven development where they first write tests, to assure that stuff will work, even later when code changes could affect it.
At least from my view, half of the issues I discovered here were because probably nobody yet tried this with terrain or that with planar reflections etc.

Anyways new things I want to ask about lights visibility_mask:
I added a bunch of lights to scene. Main render should see it only.
I also have a cubemap rendered and I want it to not have those many lights visible.
So I've set in:

Code: Select all

abstract target cubemap_target
{
	pass render_scene
	{
		light_visibility_mask 0x001

and when I create these many lights I set:
light->setLightMask(0x10);
This should work right? Why does it not, am I missing something?

Another topic. I had trouble moving my nicer water (from my terrain demo, which has depth buffer access and refractions) to SR3 (where there are many render queue groups and visibility masks and in editor also viewports and RTTs).
So, I'd try a simpler approach:
First I'd add depth buffer to .compostior, split it for objects that want to access depth buffer etc.
If I render stuff that will access depth texture, it needs own pass and visibility mask, am I right?.
In Hlms/Pbs/Any/Refractions_piece_ps.any there is this:

Code: Select all

	midf3 refractionCol = OGRE_SampleLevelF16( refractionMap, refractionMapSampler, refractUv, 0 ).xyz;
	midf refractionDepth = OGRE_SampleLevelF16( depthTextureNoMsaa, refractionMapSampler, refractUv, 0 ).x;

How are those samplers set, when are they available in shader? Can I use this piece and sample from depth texture in any other place? Do I need to set this sampler? E.g. if I wanted to do soft particles I'd need to access it (I need it in Pbs not unlit).
If I get a crash at start from compiling shaders with them missing what could be wrong?
I have again too many questions, this seems way too hard to do. But I think it's probably easier than adding SSAO.

Could we have sample(s) that would have all or most things at once, not each sample having just one component? That's why I started doing my Terrain demo, but it's still missing SSAO, HDR, GI, etc. And I can't even. Do you think it'd be east to add those there? Which GI technique would be good for terrain maps?

Lastly (and least important), there isn't any occlusion in OgreNext right?
I've seen in some video that this is important in big games with lots of trees or objects. Surely for many of our mountain terrain tracks lots of trees would be occluded by terrain. Even further terrain would be occluded by near terrain sometimes.
It seems to be in bgfx already, sample here with few links.
What do you think of occlusion, is it worth? Of course, I surely can't code it myself.

[dark_sylinc]

Hi!

I completely forgot to checkout the Area lights, sorry about that!

Anyways new things I want to ask about lights visibility_mask:

Lightmasks are tricky because they're easy to enforce for Directional lights but much harder for everything else. See this post.
It's also a feature I am not that familiar myself so I often forget how it works.

How are those samplers set, when are they available in shader? Can I use this piece and sample from depth texture in any other place? Do I need to set this sampler? E.g. if I wanted to do soft particles I'd need to access it (I need it in Pbs not unlit).

First remember that access to the depth buffer being rendered to means that all depth access must be read only.

Second see the relevant piece of code:

Code: Select all

   mDepthTexture = sceneManager->getCurrentPrePassDepthTexture();
   mDepthTextureNoMsaa = sceneManager->getCurrentPassDepthTextureNoMsaa();
        

   // ... much later on
  const bool refractionsAvailable = getProperty( tid, HlmsBaseProp::SsRefractionsAvailable );
  if( refractionsAvailable )
  {
      if( !depthTextureDefined && !getProperty( tid, HlmsBaseProp::UsePrePassMsaa ) )
      {
          setTextureReg( tid, PixelShader, "gBuf_depthTexture", texUnit++ );
          depthTextureDefined = true;
      }
      else
      {
          setTextureReg( tid, PixelShader, "depthTextureNoMsaa", texUnit++ );
      }
      setTextureReg( tid, PixelShader, "refractionMap", texUnit++ );
  }

  OGRE_ASSERT_HIGH(
      ( refractionsAvailable || getProperty( tid, HlmsBaseProp::ScreenSpaceRefractions ) == 0 ) &&
      "A material that uses refractions is used in a pass where refractions are unavailable! See "
      "Samples/2.0/ApiUsage/Refractions for which pass refractions must be rendered in" );

The key part is when depthTextureDefined gets set to true. The other key part is when gBuf_depthTexture (or depthTextureNoMsaa) gets set.

The other bit of code that you need to look for is:

Code: Select all

if( mDepthTexture )
{
  *commandBuffer->addCommand<CbTexture>() =
      CbTexture( (uint16)texUnit++, mDepthTexture, mDecalsSamplerblock,
                 PixelFormatGpuUtils::isDepth( mDepthTexture->getPixelFormat() ) );
}

if( mDepthTextureNoMsaa && mDepthTextureNoMsaa != mPrePassMsaaDepthTexture )
{
  *commandBuffer->addCommand<CbTexture>() = CbTexture(
      (uint16)texUnit++, mDepthTextureNoMsaa, mDecalsSamplerblock,
      PixelFormatGpuUtils::isDepth( mDepthTextureNoMsaa->getPixelFormat() ) );
}

Could we have sample(s) that would have all or most things at once, not each sample having just one component? That's why I started doing my Terrain demo, but it's still missing SSAO, HDR, GI, etc. And I can't even. Do you think it'd be east to add those there? Which GI technique would be good for terrain maps?

I tried to get to address that with livingroom sample but I did not get the feedback I was expecting (it's still too "demoish").

Lastly (and least important), there isn't any occlusion in OgreNext right?
I've seen in some video that this is important in big games with lots of trees or objects. Surely for many of our mountain terrain tracks lots of trees would be occluded by terrain. Even further terrain would be occluded by near terrain sometimes.
It seems to be in bgfx already, sample here with few links.
What do you think of occlusion, is it worth? Of course, I surely can't code it myself.

No, we don't. If you play with render queues to force the terrain to render last, the Z buffer will reject objects behind mountains.
However there is still vertex shader and CPU draw call cost.
OgreNext is very efficient about draw call cost.

IIRC you are already using MovableObject::setRenderingDistance to make objects disappear after a certain distance from camera, which also helps a lot.

Anyway, you'd get very low value in occlusion culling because that only works when the system can 100% guarantee the object must not be rendered and it will look fine. Which is often not the case with thin trees.

You get more bang for the buck w/ Occlusion culling when using it in-doors or in a highly populated city with big tall buildings (recent Spiderman titles?).

[Crystal Hammer]

Ok thanks.
So how is it going with the area lights? I'm guessing you had no time to spare.

From new things I have this crash in Debug with Vulkan, compiling shader (I'm using v3-0 branch now):

Code: Select all

09:01:57: Parsing script EsmGaussianBlurLogFilter.material
09:01:57: Vulkan GLSL compiler error in ESM/GaussianLogFilterH_ps_VK:
ERROR: 0:31: 'limitations' : inductive-loop init-declaration requires the form "type-specifier loop-index = constant-expression" 
ERROR: 0:36: '' : compilation terminated 
ERROR: 0:35: 'limitations' : Non-constant-index-expression 
ERROR: 3 compilation errors.  No code generated.


DEBUG LOG:

09:01:57: 
## ROOT LAYOUT BEGIN
{
	"0" :
	{
		"has_params" : ["vs", "ps"],
		"textures" : [0, 4],
		"samplers" : [0, 4]
	}
}

## ROOT LAYOUT END
#extension GL_EXT_samplerless_texture_functions : require
#define vulkan_layout layout
#define vulkan( x ) x
#define vk_comma ,
#define vkSampler1D sampler1D
#define vkSampler2D sampler2D
#define vkSampler2DArray sampler2DArray
#define vkSampler3D sampler3D
#define vkSamplerCube samplerCube
#define ogre_P0 set = 0, binding = 1
#define ogre_t0 set = 0, binding = 2
#define ogre_t1 set = 0, binding = 3
#define ogre_t2 set = 0, binding = 4
#define ogre_t3 set = 0, binding = 5
#define ogre_s0 set = 0, binding = 6
#define ogre_s1 set = 0, binding = 7
#define ogre_s2 set = 0, binding = 8
#define ogre_s3 set = 0, binding = 9
#define NUM_WEIGHTS 9
#define K 80
#define K 80
#define HORIZONTAL_STEP 1
#define VERTICAL_STEP 0

09:01:57: OGRE EXCEPTION(3:RenderingAPIException): Fragment Program ESM/GaussianLogFilterH_ps_VK failed to compile. See compile log above for details. in VulkanProgram::compile at /home/ch/_sr/Ogre/ogre-next/RenderSystems/Vulkan/src/OgreVulkanProgram.cpp (line 674)

I got default shaders copied from OgreNext. IDK why now it's happening or if it was before.

But fix for this is not using --i or ++i, but i-- and i++ there, in GaussianBlurLogFilter_ps.glsl.

Edit: now I get this twice:

Code: Select all

11:05:12: Vulkan GLSL to SPIRV 300000000VertexShader_vs:

11:05:12: Shader 300000000VertexShader_vs compiled successfully.
11:05:12: Vulkan GLSL compiler error in 300000000PixelShader_ps:
ERROR: 0:328: 'limitations' : Non-constant-index-expression 
ERROR: 0:336: 'limitations' : Non-constant-index-expression 
ERROR: 2 compilation errors.  No code generated.


DEBUG LOG:

11:05:12: 
## ROOT LAYOUT BEGIN
{
	"0" :
	{
		"const_buffers" : [0, 3],
		"readonly_buffers" : [0, 1]
	},
	"1" :
	{
		"baked" : true,
		"textures" : [2, 3],
		"samplers" : [2, 3]
	}
}

## ROOT LAYOUT END
#extension GL_EXT_samplerless_texture_functions : require
#define vulkan_layout layout
#define vulkan( x ) x
#define vk_comma ,
#define vkSampler1D sampler1D
#define vkSampler2D sampler2D
#define vkSampler2DArray sampler2DArray
#define vkSampler3D sampler3D
#define vkSamplerCube samplerCube
#define ogre_B0 set = 0, binding = 0
#define ogre_B1 set = 0, binding = 1
#define ogre_B2 set = 0, binding = 2
#define ogre_R0 set = 0, binding = 3
#define ogre_t2 set = 1, binding = 0
#define ogre_s2 set = 1, binding = 1

from some unlit shader, I've attached it (300000000PixelShader_ps.glsl), it's on these 2 lines:

Code: Select all

			ushort materialId	= worldMaterialIdx[inPs.drawId].x & 0x1FFu;
..
		ushort diffuseIdx0 = material.indices0_3.x & 0x0000FFFFu;															

And no idea why, I think that's enough for me, of trying to run Vulkan in Debug.

BTW I couldn't get ESM to work or show anything normal.
If you ever wanted to check, code is commented out in src/common/AppGui_Shadows.cpp (just search for esm).

[Crystal Hammer]

Next issue, I have this

Code: Select all

void App::update( float dt )
{
	//  fixes white texture flashes
	if (bLoading)
	{
		auto* texMgr = mRoot->getRenderSystem()->getTextureGpuManager();
		texMgr->waitForStreamingCompletion();
	}

and it seems working for scene textures etc.
But I still get white flashes on each loaded texture by MyGui.
It is annoying, but not critical for me. IDK how to even try fixing this.

Another weird crash I got in Debug only (it works in Release but on 2nd try) is in my bool PreviewTex::Load(, goes like so:

Code: Select all

			Image2 img;

			....
			(loading img from FileStreamDataStream)
			createTexture also done here
			....

			auto pf = PFG_RGBA8_UNORM_SRGB;
			const TextureBox& imgData = img.getData(0);

			StagingTexture* tex = mgr->getStagingTexture(
				xSize, ySize, 1u, 1u, pf );
			tex->startMapRegion();
			TextureBox texBox = tex->mapRegion(
				xSize, ySize, 1u, 1u, pf );

			tex->stopMapRegion();
			if (imgData.width == texBox.width && texBox.data)
			try
			{
				texBox.copyFrom( imgData );  // upload

Any idea what else to check foror what could crash in TextureBox copyFrom? There are plenty of asserts inside, none trigger. But in debugger I see that texBox.data is <invalid address> and that's it. Then it crashes saying:

Code: Select all

Exception has occurred: Signal
signal SIGSEGV: address not mapped to object (fault address: 0x7fffc2800040)

inside OgreTextureBox.h in void copyFrom( const TextureBox &src ) at first memcpy( dstData, .. at top, because dstData is invalid.
I'm guessing I should check if the texture is loaded or exists in GPU or something first, how to do it? Maybe like so?:
if (prvTex->getResidencyStatus() != GpuResidency::Resident)

I also get this in log:

Code: Select all

11:00:14: DD== MinimizeMemory ====DD
11:00:14: OGRE EXCEPTION(9:UnimplementedException):  in VulkanVaoManager::cleanupEmptyPools at /home/ch/_sr/Ogre/ogre-next/RenderSystems/Vulkan/src/Vao/OgreVulkanVaoManager.cpp (line 580)

IDK if related.

[Crystal Hammer]

But actually I have a way more important problem.
I was again trying to get depth buffer and refractions into SR3 (like I did in terrain demo).
I followed your advice from before post.
But I'm getting nowhere with this. I'm pretty sure I did all right, but I still get that assert

Code: Select all

/home/ch/_sr/Ogre/ogre-next/Components/Hlms/Pbs/src/OgreHlmsPbs.cpp(1219): Assert Failure: '( refractionsAvailable || getProperty( HlmsBaseProp::ScreenSpaceRefractions ) == 0 ) && "A material that uses refractions is used in a pass where refractions are unavailable! See " "Samples/2.0/ApiUsage/Refractions for which pass refractions must be rendered in"' 

I test this in game, on new branch refract, all relevant changes can be seen in the diff.
and I'm loading track = Test7-FluidsSmall in game.cfg, which has few fluids, but now "Water_cyan" (from all.material.json) has:

Code: Select all

            "refraction_strength" : 0.9,
			"transparency" :
			{
				"value" : 0.8061712,
				"mode" : "Refractive",

it crashes in Debug on that assert.
If I use "mode" : "Transparent", it works fine, although I think lost shadows and Hud.

Water is made in FluidsReflect::CreateFluids() using
item[n]->setRenderQueueGroup( RQG_Fluid ); item[n]->setVisibilityFlags( RV_Fluid );
defined in src/common/RenderConst.h as 210 and 2 respectively.
And adjusted all rq_last and visibility_mask etc, in SR3.compositor,
now abstract compositor_node SR3_RenderAbs has depth buffer and renamed old to SR3_RenderAbsOld.

Can you help me with this? Or at least check if I still forgot something or did wrong?

[Crystal Hammer]

Okay I fixed some stuff, not all yet. But I finally don't get assert and have refractions. I had later in workspace inheritance that changed my first definition.

So how do I set alpha (.w) per pixel for a refractive Pbs water?
I've tried setting outColour.w and it doesn't change it.
I want to do small soft alpha transition on border, and use it for soft particles too someday.

[Crystal Hammer]

Okay what's this? I'm not an expert in reading Ogre code.

Code: Select all

void HlmsPbsDatablock::setTransparency( 
...
        if( changeBlendblock )
        {
            HlmsBlendblock newBlendblock;

            if( mTransparencyMode == None || mTransparencyMode == Refractive )
            {
                newBlendblock.mSourceBlendFactor = SBF_ONE;
                newBlendblock.mDestBlendFactor = SBF_ZERO;
                if( mTransparencyMode == Refractive )
                    newBlendblock.setForceTransparentRenderOrder( true );
            }
            else if( mTransparencyMode == Transparent )
            {
                newBlendblock.mSourceBlendFactor = SBF_ONE;
                newBlendblock.mDestBlendFactor = SBF_ONE_MINUS_SOURCE_ALPHA;
            }

Is it called when I just have json e.g.:

Code: Select all

			"refraction_strength" : 0.7,
			"transparency" :
			{
				"value" : 0.28,
				"mode" : "Refractive",
				"use_alpha_from_textures" : true
			},

Because I have the feeling I won't be able to change alpha per pixel in a refractive material.
Can I do that? What Blendblock should I use when I want to just control opacity of that refractive water to fade it out on border?

Also BTW, if I want to allow users to have an option for refractive water and not force them to always have refractive, I need to duplicate all my water and river materials now, right? This is insane already, in that material json already has 57848 lines and like 100 fluid materials already duplicated for rivers, meh.

Is there a way of having blurry refractions? E.g. doing mip-maps for them and sampling somewhat like for rough reflection.

[dark_sylinc]

Hi!

My backlog of OgreNext things to do has become gigantic.

From new things I have this crash in Debug with Vulkan, compiling shader (I'm using v3-0 branch now):

It looks like something got lightly corrupted (could be libOgreMain version mismatch, memory corruption, Vulkan RenderSystem is using the wrong shaderc compiler like the one from your system instead of the one that comes with ogre-next-deps, etc).

For two reasons:

1. AFAICT Fragment Program ESM/GaussianLogFilterH_ps_VK should not fail to compile.
2. ESM/GaussianLogFilterH_ps* should not be a limiter because it's only used as a fallback when Compute Shaders aren't supported.

Maybe valgrind has more info. But my prime suspect would be our Vulkan RenderSystem picking up the wrong shaderc/glslang compiler when building (i.e. it picked the one installed in your system).

Try checking your CMake config's Vulkan_SHADERC_LIB_REL, Vulkan_SHADERC_LIB_DBG and Vulkan_SHADERC_INCLUDE_DIR:

and it seems working for scene textures etc.
But I still get white flashes on each loaded texture by MyGui.
It is annoying, but not critical for me. IDK how to even try fixing this.

Eugene ran into this same problem because he is using a ton of listeners (just like you) and in one of those listeners, image loading is triggered. This of course happens after your call to waitForStreamingCompletion, which is why it happens.

His workaround is the following, which is basically to detect if this scenario happened and re-render the whole frame again:

Code: Select all

textureMgr->waitForStreamingCompletion();
const oldValue = textureMgr->getLoadRequestsCounter();
root->renderOneFrame();
if( oldValue != textureMgr->getLoadRequestsCounter() )
{
    textureMgr->waitForStreamingCompletion();
    root->renderOneFrame();
}

This may need to be changed to a while() loop if it happens multiple times in a row.

Ideally you should try to move most stuff outside of listeners.

Any idea what else to check foror what could crash in TextureBox copyFrom? There are plenty of asserts inside, none trigger. But in debugger I see that texBox.data is <invalid address> and that's it. Then it crashes saying:

I don't know, but you didn't post a callstack.
Also valgrind can help a lot here. It's super slow but it is really good (I recommend to use Valgrind with QtCreator, and set QtCreator to show "External Errors"):

I also get this in log:

Code: Select all

11:00:14: DD== MinimizeMemory ====DD
11:00:14: OGRE EXCEPTION(9:UnimplementedException):  in VulkanVaoManager::cleanupEmptyPools at /home/ch/_sr/Ogre/ogre-next/RenderSystems/Vulkan/src/Vao/OgreVulkanVaoManager.cpp (line 580)

Yeah, that's expected. Just enclose it in a try/catch block and let your code continue.

Okay what's this? I'm not an expert in reading Ogre code.

changeBlendblock = true means we will setup the ideal blendblock for you automatically.

In case of refractions, it uses an opaque blendblock (i.e. no alpha blending at all) because the blending operation is done inside a shader by hand.

Because I have the feeling I won't be able to change alpha per pixel in a refractive material.

The refraction is controlled by HlmsPbsDatablock::mTransparencyMode.
But if useAlphaFromTextures = true, then you can use the alpha from the diffuse texture.

What Blendblock should I use when I want to just control opacity of that refractive water to fade it out on border?

"Border" can mean many things. Do you mean the border between the lake and the terrain, or do you mean calculated via a complex math in a shader?

Btw the relevant section you're looking for is in Refractions_piece_ps.any:

Code: Select all

	@property( use_texture_alpha )
		midf refractionAlpha = midf_c( material.F0.w ) * pixelData.diffuse.w;
	@else
        midf refractionAlpha = midf_c( material.F0.w );
	@end

finalColour += refractionCol.xyz * (_h( 1.0 ) - refractionAlpha);

When refractionAlpha = 0, the refraction becomes transparent. When refractionAlpha = 1, the object becomes opaque.

If you want to just fade out the refraction; I think you'd have to sample the refraction texture again at the right coordinates (untested code):

Code: Select all

finalColour += refractionCol.xyz * (_h( 1.0 ) - refractionAlpha);

midf3 originalColBehindObject = OGRE_SampleLevelF16( refractionMap, refractionMapSampler, screenPosUv.xy, 0 ).xyz;
finalColour = lerp( originalColBehindObject, finalColour, myWeight );

[Crystal Hammer]

Allright thanks for helping.
Your last code to fade out the refraction works, it's what I needed.
Just strangely somehow water depth is scaling with distance in SR3 but it's not in Terrain Demo, odd. Anyway I probably need to pass inversed projection matrix to shaders, instead or doing that matrix inverse per pixel.

[Crystal Hammer]

Let's say I have a file a.material.json that has a "sampler_a" used in 1 material,
and a file b.material.json that also has a "sampler_a" used in 1 other material.
Does it create 2 samplers or just 1 since they have same parameters and there is some hash for that? If it does 2 then is is it slower?
I'm splitting my huge json to smaller.
Also can I just put all samplers in each .json even if not used there? Would be easier.
But I can't just put all samplers in own .json and just materials (w/o samplers defined) in other .json right? I think this would be better. That way if I ever decided to do any change in a sampler (block) I wouldn't need to apply it in all other .json files, to have it the same.

[dark_sylinc]

Hi!

If the samplers have the exact same parameters, then internally they will share the same samplerblock (unless you hit some sort of bug).

The "name" a samplerblock has in a JSON material is simply for referencing it within that same JSON file.

• If two JSON files have a samplerblock with different names but exact same parameters, they will share the same samplerblock pointer.

• If two JSON files have a samplerblock with the same name but different parameters, they will NOT share the same samplerblock pointer.

The only performance issue I can foresee is that if you split it into a 10000 different JSON files, OgreNext will have to JSON-parse the same samplerblock 10000 times, thus slowing down material loading (whether it'd be noticeable, I don't know).

The same thing applies to macroblocks and blendblocks.

[Crystal Hammer]

So, I have removed almost all from AtmosphereNpr and I'm left with what I need: a ConstBufferPacked and a bunch of my parameters that I want to feed ALL shaders with, I mean all of them.
I need some for wind, that has to be even for shadow caster, some are for vertex shaders, some for pixel shaders. Even for terrain someday, I don't care if particles etc have them too.
My reduced code is here:
Atmosphere.h
Atmosphere.cpp
Works well for all that worked before, I was using modified AtmosphereNpr until now.

Now why am I still getting those freaking errors:

Code: Select all

15:44:21: GLSL compile log: 100000034VertexShader_vs
0:505(15): error: `atmo' undeclared
0:505(15): error: type mismatch
0:505(15): error: type mismatch
0:507(22): warning: `time' used uninitialized

100000034VertexShader_vs.glsl

Now clearly atmo isn't declared in glsl there (but is being used). Why?

AtmoSettings atmo; is declared in:
200.Atmo_piece_all.any
as @piece( AtmoStructDecl )
needs @property( atmosky_npr ) but even without this, it still failed.

AtmoStructDecl is inserted in DefaultHeaderVS
800.VertexShader_piece_vs.any

and in DefaultHeaderPS
800.PixelShader_piece_ps.any

I'm assuming the code in Atmosphere.cpp
does what I see, so sets this buffer (in bindConstBuffers) for both VS and PS, so why isn't it in VS?
Also sets atmosky_npr in preparePassHash, for all shaders right?

What's even more weird that I use it for grass in VS already. It only now fails when I try to add it to regular meshes for trees, ferns etc but only in editor, it works fine in game. I can't get what's going on here.

Sorry for editing this so much.
Hmm, I changed that last time member from float to Vector4, it doesn't crash now.

It even does shadow casters, but not right, I get weird blinking shadows.
Ok what is different in shadowcaster vertex shader? Seems like I can't just change postions like in regular render?

[Crystal Hammer]

Okay so from above post, one left question:
I got tree wind animation adjusting pos of each vertex in vertex shader, so what variable and when should I change?
I am now adjusting worldPos just before @insertpiece( PoseTransform )
and it seems working. But it does some garbage for shadow casters. I'd like both doing the same.
Later is @insertpiece( VertexTransform ) and it uses this no-spaces-at-all, hard to read thing, IDK:
@piece( CalculatePsPos )mul( @insertpiece(local_vertex), worldViewMat ).xyz@end

I could be wrong but I really think now this code is so cryptic, that no wonder only around 10 people can read it and modify (after year(s) of studying) .
I understand there is @insertpiece( custom_vs_preTransform ) which BTW still has TBD in doc. What could I even access or change inside such piece? How am I supposed to know? Only by knowing the whole code from .any right? I think that's the biggest problem for starting with shaders. Can't do anything without studying half or whole code first. Which has hundreds of lines in few files with rare comments.

So yeah in any big engine like UE, Unity, Godot you could just put a PBS or material block, and use its input for position, then write such tree wind code in blocks etc, would work for caster too I assume, and without looking and studying whole inside details of that PBS block.

BTW what's inside #include "/media/matias/Datos/SyntaxHighlightingMisc.h", and for which editor is it? I see it in like every file on top.
Is there any syntax definition for .any files e.g. for something modern like VSCodium (VS Code)? That could be useful.
It'd be even greater if we could just use "go to definition" on symbols in .any files, but that's very unlikely right?

[dark_sylinc]

Hi!

First I strongly suggest that you separate your custom files from OgreNext's default files.

Like so:

• Hlms

  • Unlit

  • Pbs

  • PbsSR3

    • 200.Atmo_piece_all.any

This way when updating OgreNext, upgrading goes much smoother.
Just make sure that you insert PbsSR3 to the paths returned by HlmsPbs::getDefaultPaths, likely before "Hlms/Pbs/Any/Main" entry (parsing order matters).

Second, prefix or suffix all your custom code. Instead of AtmoStructDecl, use SR3_AtmoStructDecl.
This will make tracking your changes easier, as well avoid potential name clashing with OgreNext' default code.

You sould define @piece( SR3_AtmoStructDecl ) like you're doing now. But instead of modifying 800.VertexShader_piece_vs.any directly just do:

Code: Select all

@piece( custom_vs_uniformStructDeclaration )
    @insertpiece( SR3_AtmoStructDecl )
@end

That way you can insert your customizations without touching OgreNext's default code, minimizing merge conflicts when upgrading. Of course working on 800.VertexShader_piece_vs.any directly is faster and makes things easier when troubleshooting why the Hlms is not generating the code you want.
But once you figured that out, it's better to put it isolated without touching OgreNext's data files as much as possible.

Now clearly atmo isn't declared in glsl there (but is being used). Why?

I have no idea. After quick glance and your code, your AtmoSettings atmo should always be defined.

Try looking in both the console and Ogre.log for any Hlms parser error. In very rare cases the Hlms can log out errors and still spit out shader code that appears ok but is not.

The thing that would make the most sense is that your app is loading the Hlms from another folder where 200.Atmo_piece_all.any does not exist but 800.VertexShader_piece_vs.any is up to date; thus causing your problem.

BTW what's inside #include "/media/matias/Datos/SyntaxHighlightingMisc.h", and for which editor is it? I see it in like every file on top.

I'm simply using this file: to get C++ syntax highlighters in QtCreator to work with GLSL/HLSL/Metal. Of course it's not valid code; it's just so that I get something decent to work with, like code autocomplete.

I am now adjusting worldPos just before @insertpiece( PoseTransform )
and it seems working. But it does some garbage for shadow casters. I'd like both doing the same.
Later is @insertpiece( VertexTransform ) and it uses this no-spaces-at-all, hard to read thing, IDK:
@piece( CalculatePsPos )mul( @insertpiece(local_vertex), worldViewMat ).xyz@end

Ugh, I hate it too.

That code was written like that with the awful mobile GLES parsers in mind (which were really bad at optimizing), so I tried to avoid things like copying position into a temporary variable if possible.

The most common case (rendering non skeletally animated objects) would end up as:

Code: Select all

outVs.pos = mul( inVs_vertex, worldViewMat );

But now in 2024 (with GLES out of the question), PRs to declutter that madness are welcome.
Like always copying into a temporary variable:

Code: Select all

float4 inputPos = inVs_vertex; // no more #define inputPos inVs_vertex

@property( !hlms_skeleton )
   // This is in local space. Gets multiplied by worldViewProj.
   float4 readyForViewProj = inputPos; // No more @piece( local_vertex )inputPos@end
@else
   // This is in world space. Local space got converted by skeleton transformations into world space.
   // Now all that's left is to get multiplied by viewProj.
   float4 readyForViewProj = worldPos; // No more @piece( local_vertex )worldPos@end
@end

[Crystal Hammer]

Thanks.
I tried this syntax.h but clangd doesn't get it right, lots of issues. Mainly because @ syntax words not being covered. I got back to C# for just syntax coloring.

There is this nice bunch of parameters to use in pixel shader: material.detailOffsetScale and material.userValue.
Is is possible to use it also in vertex shader? I need a way to get few params from material in .json to get into vertex shader to scale wind effect for trees and bushes etc.

[dark_sylinc]

Thanks.
I tried this syntax.h but clangd doesn't get it right, lots of issues. Mainly because @ syntax words not being covered. I got back to C# for just syntax coloring.

Yeah it depends on the IDE.
Sometimes doing:
// @property( xx )
// @end

works because the Hlms doesn't recognize those as being inside a comment and it becomes valid C++.

There is this nice bunch of parameters to use in pixel shader: material.detailOffsetScale and material.userValue.
Is is possible to use it also in vertex shader? I need a way to get few params from material in .json to get into vertex shader to scale wind effect for trees and bushes etc.

Mmmm usually I just look at what magic @pset( ogre_property, 1 ) incantation I need to do to get it defined on the VS. But when you say it out loud; I realize we could use a hlms_force_material_vs / hlms_force_material_ps that you can set (e.g. via @pset( hlms_force_material_vs, 1 ) or via C++) to make it user-friendlier.

Feel free to submit a PR for this!

[Crystal Hammer]

Mmmm usually I just look at what magic @pset( ogre_property, 1 ) incantation I need to do to get it defined on the VS. But when you say it out loud; I realize we could use a hlms_force_material_vs / hlms_force_material_ps that you can set (e.g. via @pset( hlms_force_material_vs, 1 ) or via C++) to make it user-friendlier.

I don't think I understand. And I don't have any slots or know any magic spells at that high level.
So what is actually capable in C++ of sending different parameter values to each datablock (VS and PS)?

Documentation is probably HlmsCreationOfShadersCustomizing and I keep my notes in my Developing.md.

• Only capable is a custom datablock right
  class HlmsPbsDb2 : public Ogre::HlmsPbsDatablock
  where I can set mUserValue[.. (to values read from .json before), in
  void HlmsPbsDb2::uploadToConstBuffer( char *dstPtr, uint8 dirtyFlags )
• And in void HlmsPbs2::calculateHashForPreCreate(
  we can call setProperty(, to anything, even getDatablock() from Renderable
  But this creates a new shader every time. Is not meant for float or int values really, only for bool flags. Could be used as last resort.
• HlmsListener, which can change params for all Pbs shaders. In HlmsPbsTerraShadows::preparePassBuffer. So not usable here.
  Demo in Tutorial_Hlms01_Customization
• Somewhat similar is using Atmosphere2Npr or inheriting from Ogre::AtmosphereComponent and setting params for all shaders, in _update method which fills a ConstBuffer.

MaterialStructDecl has //Uniforms that change per Item/Entity, but change very infrequently.
There are userValue and detailOffsetScale arrays that I use for my other stuff.
But this is inside PS @piece( DefaultHeaderPS ), same place where @insertpiece( AtmoStructDecl ) was.
So how do I get (all) that MaterialStructDecl in VS? Is that even okay to do?

Or should I just add 1 float4 instead? And if so, how to do it?
I mean send custom params (not for all but) per Datablock / material, and for VS not PS.
I think I'm missing like 2 more samples Tutorial_Hlms05_Customization, 6 etc. Which would do that.

I'm seeing something close in TerraMaterialStructDecl in @piece( DefaultTerraHeaderVS ), it is also in @piece( DefaultTerraHeaderPS ) It even has mDetailsOffsetScale filled in HlmsTerraDatablock::uploadToConstBuffer.
But there is only one HlmsTerraDatablock::MaterialSizeInGpu (not possible to change in Pbs IIRC) so is a ConstBuffer always accessible from VS and PS?
Also TerraInstanceStructDecl filled in TerrainCell::uploadToGpu but that's too much.

BTW is this .inl and #define still needed?

Code: Select all

#define OGRE_HLMS_TEXTURE_BASE_CLASS HlmsTerraBaseTextureDatablock
#define OGRE_HLMS_TEXTURE_BASE_MAX_TEX NUM_TERRA_TEXTURE_TYPES
#define OGRE_HLMS_CREATOR_CLASS HlmsTerra
#include "OgreHlmsTextureBaseClass.inl"

I also think the table here, with many TBD, could use some examples, if not full info of what is availabe at this point, and what could be changed in each custom_ block. And yeah maybe even some deserve their own Tutorial Sample.
https://ogrecave.github.io/ogre-next/ap ... ustomizing

[Crystal Hammer]

So I tried once again to turn my particles from unlit to pbs.
Whatever I do or change they are always solid white, no warning in logs etc.
I see that wireframe works when I set it in that json, but nothing else.
I did set finalColour.xyz to something, for all Pbs shaders, and particles are still white.
I have vertex colors applied in pixel shader, I added it BTW of road.

Hmm I'm looking in renderDoc and those particle white pixels are marked "invalidated", what does it mean?
Ok there is almost nothing in shaders:

100000100PixelShader_ps.glsl

100000100VertexShader_vs.glsl

only this in pixel main outPs_colour0 = midf4_c( 1.0, 1.0, 1.0, 1.0 ); even though I have textures etc.

Aah it's because this fails for particles:
@property( hlms_normal || hlms_qtangent )
Ugh how can I fix this? I would set normal to 0,0,1 or something constant since they are always same aligned as screen. But then I'm missing all other stuff in pixel shader that needs hlms_normal || hlms_qtangent defined.
So I've added a custom property particle, and I'm doing this for my new particle pbs

Code: Select all

			setProperty( "hlms_normal", 1);
			setProperty( "hlms_qtangent", 1);
			setProperty( "particle", 1 );

[dark_sylinc]

Yeah the lack of normals is what's breaking PBS.

You could try lying to HlmsPbs by telling it in calculateHashForPreCreate that it's got a normal e.g.

Code: Select all

setProperty( kNoTid, HlmsBaseProp::Normal, 1 );

But you would also have to deal with patching the generated shader so that it avoids generating broken shader code. It's not pretty.

Would you be able to try the ParticleFX branch? You can use it in combination with PFX1. With luck, PFX2 limitations won't be a hindrance to the applications you're trying to use it for; and it has got normal generation (it works with PBS).

[dark_sylinc]

Something about your problem felt familiar and I finally remembered!

PFX2 had indeed the same problem you're facing.

1. I modified Pbs shaders so that when hlms_particle_system property is set, invalid code isn't generated that tries to fetch non-existent normals (and also non-existent position & UVs because that's how PFX2 works).
2. HlmsPbs::notifyPropertiesMergedPreGenerationStep lies by setting hlms_normal when hlms_particle_system is set.
3. When hlms_particle_system is set, it also defines DoParticleSystemVS piece which sets all the data fetched from somewhere else (i.e. a Const buffer or a ReadOnly buffer) and sent to the Pixel Shader.

[Crystal Hammer]

Thanks.
Yes I did check out PFX2 branch months ago, and I'm still waiting for merge to master. And into 3.0 since I'm not even using master. I can't merge PFX2 with 3-0 in my .py build script for SR3.
Honestly, the alpha hashing noise was getting on my nerves. But I think this only matters for dust on deserts, sandstorms and such where I would still use PFX1. I guess all other particles could use PFX2 in SR3 like: rain, snow, gravel from road, water, mud splashes, and all additive like: fire, sparks, boost, etc.