Page 1 of 2

[2.1] Easier way to communicate with hlms shader?

Posted: Sun Apr 05, 2015 5:02 am
by xrgo
Hello, in my previous topic ( http://www.ogre3d.org/forums/viewtopic.php?f=25&t=82878 ) I learned some ways to communicate with the pbs shader from code:
1. modifying the OgreHlmsPbs implementation ( http://www.ogre3d.org/forums/viewtopic. ... 78#p515450 )
2. adding new parameters in HlmsPbsDatablock ( http://www.ogre3d.org/forums/viewtopic. ... 78#p516178 )

And they work flawlessly =)... thing is I don't like the idea of modify Ogre source, because sometimes it gets complicated to merge when there are changes on the repo.

Is there any other (not so invasive) way to communicate with hlms shaders?
like in the old way:
glsl: uniform float myParameter;
c++: myMaterial->getTechnique(0)->getPass(0)->getFragmentProgramParameters()->setNamedConstant("myParameter", value);

Thanks in advance!!

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Sun Apr 05, 2015 7:27 am
by dark_sylinc
Not yet.

There was/is a plan to allow for customizations interfaces. While they might not end up offering the full flexibility you need (depends on the features to be added...) as too much flexibility will hurt performance or make things harder, it could make the merging easier (or non existant if you're lucky).

The basic idea was that was that you would provide a template where custom pieces would be collected to be inserted in the shaders (so that you can customize the templates without actually modifying the original ones); and provide some sort of listener to setup per-pass data or setup properties at shader creation time (don't expect a listener to be called per object though, that ain't gonna happen!).
This isn't too hard, but might need to be handled with a bit of care.

Don't know when this feature is going to arrive though. There's still some important threading left to do to the Hlms (fillBuffersFor) which shouldn't affect such system, but if I'm wrong then that would make things more complicated if the customization system is already there, or may be not.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Sun Apr 05, 2015 6:14 pm
by xrgo
Thank you for your answer, hope to see that feature soon =) thanks

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Sat Apr 25, 2015 8:06 pm
by dark_sylinc
Hi xrgo. I just pushed this feature to the repo: https://bitbucket.org/sinbad/ogre/commi ... 6a3b3d8f26.

The manual explains a bit on how to do it. I haven't written a sample yet, showing how to do it. Do you think you can figure it out?

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Sat Apr 25, 2015 9:25 pm
by xrgo
Thank you so much!!!!!!!
dark_sylinc wrote:Do you think you can figure it out?
yes I think I can :)

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Mon May 18, 2015 8:28 pm
by xrgo
I just tried it and it worked like a charm =)
Thank you so much for this

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Jul 14, 2015 2:28 pm
by aymar
Hey xrgo,

Can you post some info on how to use the HlmsListener to customize the PBS?

Thanks.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Jul 14, 2015 9:26 pm
by xrgo
Here is how I made some simple linear fog:

You have to inherit some class from HlmsListener

Code: Select all

class SomeClass : public Ogre::HlmsListener
and implement these methods:

Code: Select all

    //Comunicate stuffs to the PBS shader
    Ogre::uint32 getPassBufferSize( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager );

    float* preparePassBuffer( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager,
                                      float *passBufferPtr );
for instance for my Fog I implement them like this:

Code: Select all

Ogre::uint32 SomeClass::getPassBufferSize( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager ){
        return 32; //( vec4 fogParams + vec4 fogColor )*4
    }

float* SomeClass::preparePassBuffer( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                  bool dualParaboloid, Ogre::SceneManager *sceneManager,
                                  float *passBufferPtr ){

    //Linear Fog parameters
    *passBufferPtr++ = sceneManager->getFogStart();
    *passBufferPtr++ = sceneManager->getFogEnd();
    *passBufferPtr++ = 0.0;
    *passBufferPtr++ = 0.0f;

    *passBufferPtr++ = sceneManager->getFogColour().r;
    *passBufferPtr++ = sceneManager->getFogColour().g;
    *passBufferPtr++ = sceneManager->getFogColour().b;
    *passBufferPtr++ = 1.0f;
    return passBufferPtr;
}
Then you have to tell the hlmsPbs that SomeClass is the listener

Code: Select all

hlmsPbs->setListener(mySomeClass);
Then in the shaders:
If you notice in the file "Structs_piece_vs_piece_ps.glsl" in the "PassBuffer" there is a "@insertpiece( custom_passBuffer )", this means that you can make your own custom passBuffer extension that the hlms precompiler will automatically put where that "@insertpiece( custom_passBuffer )" is.
So you can make a file named: "CustomPassBuffer_piece_vs_piece_ps.glsl" with this inside:

Code: Select all

@piece( custom_passBuffer )

//Fog
	vec4 fogParams;
	vec4 fogColor;
	
@end
now you have in your "fogParams" and "fogColor" that you can use in the pbs shader!
And in the same way if you notice in the file "PixelShader_ps.glsl" there is a "@insertpiece( custom_ps_posExecution )", so you can make a file named "CustomPost_piece_ps.glsl" with this inside:

Code: Select all

@piece( custom_ps_posExecution )
@property( !hlms_shadowcaster )
//Fog
	float fogDistance = length( gl_FragCoord.z / gl_FragCoord.w );   // or better "length( inPs.pos )" for a more accurate distance
	float fogFactor = 1.0-clamp((pass.fogParams.y - fogDistance) / (pass.fogParams.y - pass.fogParams.x),0.0,1.0);
	vec3  fogColor  = vec3(pass.fogColor.xyz);
	outColour.xyz   = mix(finalColour, fogColor, fogFactor );
	
@end
@end
and that way you are actually modifying the shader without modifying the original files =)

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Jul 14, 2015 10:54 pm
by aymar
That helps a lot!

Do you know if low level materials still support auto params?

Thanks!

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Jul 14, 2015 11:12 pm
by xrgo
yes! low level mats works as always, just no Fixed function pipeline and you need to put the entity on a V1_LEGACY renderqueue group

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Wed Mar 28, 2018 12:00 pm
by Lax
Hi,

I have a question, will this fog code snippets mentioned above be implemented as part of Ogre2.x?
Because if you update the ogre repository and have your own Fog implementation, this could become a mess quickly...

Regards
Lax

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Wed Mar 28, 2018 1:43 pm
by al2950
Lax wrote: Wed Mar 28, 2018 12:00 pm Because if you update the ogre repository and have your own Fog implementation, this could become a mess quickly...
I dont see this is as the case, in fact it is not the case! The fog implementation uses the standard interfaces for customizing a HLMS implementation. So you should face no difficulties in updating Ogre using your custom fog functions.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Thu Mar 29, 2018 3:11 pm
by Lax
Hi al2950,

ok thanks, I will try it out.

Regards
Lax

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue May 01, 2018 9:36 am
by Lax
Hi,

I'm looking at the fog implementation and the following came in my mind:

Code: Select all

hlmsPbs->setListener(mySomeClass);
Sets an active listener for hlms pbs, but if terra is used, there is already a listener "Ogre::HlmsPbsTerraShadows" set, so one listener would overwrite the other one.

I'm wondering, why several listeners cannot be registered like:

Code: Select all

hlmsPbs->addListener(mySomeClass);
Regards
Lax

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Fri May 18, 2018 10:20 am
by hedphelym
Hlsl version:

CustomPassBufferPiece_vs_piece_ps.hlsl

Code: Select all

@piece( custom_passBuffer )
	float4 fogParams;
	float4 fogColor;
@end
CustomPost_piece_ps.hlsl

Code: Select all

@piece( custom_ps_posExecution )
@property( !hlms_shadowcaster )
	float fogDistance =  length( inPs.pos );
	float fogFactor = clamp((passBuf.fogParams.y - fogDistance) / (passBuf.fogParams.y - passBuf.fogParams.x),0.0,1.0);
	outPs.colour0.rgb  = lerp(passBuf.fogColor.rgb,finalColour, fogFactor );
@end
@end

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Thu Aug 23, 2018 1:12 pm
by hedphelym
When I run the fog with directx in debug mode it keeps hitting this assert in 'OgreHlmsPbs.cpp' :

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
Has anyone of you tested this? Or know why that happens?
I have used the code exactly as mentioned in the first post, with the HLSL code I posted right above this post.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Fri Aug 24, 2018 3:14 am
by dark_sylinc
Check that both getPassBufferSize and preparePassBuffer from your listener are getting called.

The listener's getPassBufferSize is supposed to tell Ogre that the buffer must be 32 bytes bigger. And preparePassBuffer should return a pointer that is advanced by 32 bytes (hence the post's code performs *passBufferPtr++ eight times).

The assert is there to check that everything that had to be sent to GPU got written, and that we didn't go past the bounds.
If the assert is failing, either getPassBufferSize/preparePassBuffer are not getting called, they are in disagreement, or something very weird is going on.

Watchout for alignToNextMultiple calls. I can't remember if they affect listener code, but it has bitten a few people a few times. Most variables have to be aligned to 16 bytes and this is painful to enforce when we allow arbitrary code to be inserted.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Fri Aug 24, 2018 7:30 am
by hedphelym
The solution was very simple when I found it.
The getPassBufferSize function need to be const for it to be called.

Code: Select all

Ogre::uint32 PbsFog::getPassBufferSize( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager ) const
                                      {
                                      	.....
                                      }
                                      
What happened was that It was not defined as const, thereby not being called, and ogre got '0' instead of '32'.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Jun 16, 2020 7:51 am
by PBY
As this seems to be the only place where this simple way of adding small shader programs, and the documentation is useless, here is one point missing above:

Not only should your custom file end with the correct extension (like .hlsl), and declare the wanted @piece name, but the filename is also important.
OgreHlms will enumerate all files from the Hlms archive, but will only take file ending (before file extension) with one the following strings:
  • piece_vs
  • piece_ps
  • piece_gs
  • piece_hs
  • piece_ds
It will use the following list for finding the corresponding files:

Code: Select all

PieceFilePatterns[] = { "piece_vs", "piece_ps", "piece_gs", "piece_hs", "piece_ds" };
You should also use the correct ending, as it will take only one of the strings for each type, i.e. for creating the vertex shader, it will only search for piece_vs.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Jun 16, 2020 4:33 pm
by dark_sylinc
A few details to what you just said:
  1. You can concatenate these suffixes. MyFile_piece_vs_piece_ps.glsl means it will be parsed both during vertex and pixel shader stage.
  2. Use piece_all means it will be parsed in all shader stages (a shortcut to avoid concatenating all piece_*s suffixes)
  3. You can use the .any extension for files that should be parsed by any RenderSystem, ie. it gets parsed regardless of whether glsl/hlsl/metal is the currently active shader languague. You can also use @property( syntax == glsl )@end or @property( syntax != hlsl )@else @end to handle the minor differences

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Mon Feb 22, 2021 9:14 pm
by Lax
Hi dark_sylinc,

I have an request for a new functionality for Hlms. At this time its only possible to set one listener, but I need two listeners. Because I use terra and hence for hlms the terra shadows listener is set. Now I want also set a custom fog listener. Which does work, but after that, terra does not work anymore, because its shadow listener has been overwritten.

Would it be possible to introduction a new function e.g.

Code: Select all

addListener
and

Code: Select all

removeListener
. So that custom several listener could be added for hlms?

That would be really great! What do you think?

Best Regards
Lax

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Mon Feb 22, 2021 10:12 pm
by dark_sylinc
Can it be solved with a meta listener that calls the other listeners; and register them to your listener (instead of directly to Hlms)?

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Feb 23, 2021 12:46 pm
by Lax
hm, do you have an idea, how this could work in detail?
Because its required, that the HlmsPbs calls all virtual functions of all listeners...

Do you mean:
I use a base listener, which is registered to HlmsPbs and in this base listener, I call other custom listeners in all virtual functions?

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Feb 23, 2021 1:17 pm
by Zonder
Lax wrote: Tue Feb 23, 2021 12:46 pm Do you mean:
I use a base listener, which is registered to HlmsPbs and in this base listener, I call other custom listeners in all virtual functions?
Yes I am pretty sure thats what he means.

I don't think we would want the extra complexity on the HLMS why he says this.

Re: [2.1] Easier way to communicate with hlms shader?

Posted: Tue Feb 23, 2021 1:48 pm
by Lax
ok, I did as proposed, but its no more working. I always get an assert:

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
I now have a base listener, which gets registered directly to HlmsPbs and this listener has a vector list of contrete Listeners. Which are added.

Code: Select all

std::vector<Ogre::HlmsListener*> concreteListeners;
I tested it with a fog listener:

Code: Select all


	Ogre::uint32 HlmsFogListener::getPassBufferSize(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager) const
	{
		Ogre::uint32 size = 0;

		if (!casterPass)
		{
		        // size = 32
			size = sizeof(float) * 4 * 2; // float4 + float4 in bytes
		}
		return size;
	}
	
	float* HlmsFogListener::preparePassBuffer(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager, float* passBufferPtr)
	{
		if (!casterPass)
		{
			//Linear Fog parameters
			*passBufferPtr++ = sceneManager->getFogStart();
			*passBufferPtr++ = sceneManager->getFogEnd();
			*passBufferPtr++ = 0.0f;
			*passBufferPtr++ = 0.0f;

			*passBufferPtr++ = sceneManager->getFogColour().r;
			*passBufferPtr++ = sceneManager->getFogColour().g;
			*passBufferPtr++ = sceneManager->getFogColour().b;
			*passBufferPtr++ = 1.0f;
		}
		return passBufferPtr;
	}
And in the base listener, the function 'getPassBufferSize' for all listeners is called. In my example just the fog listener:

Code: Select all

Ogre::uint32 HlmsBaseListenerContainer::getPassBufferSize(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager) const
	{
		Ogre::uint32 size = 0;

		for (auto& it = this->concreteListeners.cbegin(); it != this->concreteListeners.cend(); ++it)
		{
			size += (*it)->getPassBufferSize(shadowNode, casterPass, dualParaboloid, sceneManager);
		}
		return size;
	}
	
	float* HlmsBaseListenerContainer::preparePassBuffer(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager, float* passBufferPtr)
	{
		for (auto& it = this->concreteListeners.cbegin(); it != this->concreteListeners.cend(); ++it)
		{
			(*it)->preparePassBuffer(shadowNode, casterPass, dualParaboloid, sceneManager, passBufferPtr);
		}
		
		return passBufferPtr;
	}
The size is incremented by the outcome of each concrete Listener. But somehow, Ogre does not accept the size anymore. The size is 256, but the assert is thrown:

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
--> 224 != 256
--> Somehow 32 bytes are missing

The strange thing is, that it did work before, when I registered directly the fog listener. So the question is, what is now different?? It should behave the same as before...

When I use the fog listener directly:

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
--> 256 == 256
[Edit] Its because 'passBufferPtr' will become different, when its used in HlmsBaseListenerContainer! But I have no clue how to solve this...

[Edit2]: Ok I've got it! 'pussBufferPtr' must become a reference to pointer (in/out) variable, so that the adress is always the same. See:

Code: Select all

float* HlmsBaseListenerContainer::preparePassBuffer(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager, float*& passBufferPtr)
@dark_sylinc: Would It be possible, that you add the '&' reference? It will not harm the pbs implementation and make it possible to use meta listeners!

Best Regards
Lax