[2.1] Easier way to communicate with hlms shader?

Discussion area about developing with Ogre2 branches (2.1, 2.2 and beyond)
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 167

[2.1] Easier way to communicate with hlms shader?

Post by xrgo »

Hello, in my previous topic ( http://www.ogre3d.org/forums/viewtopic.php?f=25&t=82878 ) I learned some ways to communicate with the pbs shader from code:
1. modifying the OgreHlmsPbs implementation ( http://www.ogre3d.org/forums/viewtopic. ... 78#p515450 )
2. adding new parameters in HlmsPbsDatablock ( http://www.ogre3d.org/forums/viewtopic. ... 78#p516178 )

And they work flawlessly =)... thing is I don't like the idea of modify Ogre source, because sometimes it gets complicated to merge when there are changes on the repo.

Is there any other (not so invasive) way to communicate with hlms shaders?
like in the old way:
glsl: uniform float myParameter;
c++: myMaterial->getTechnique(0)->getPass(0)->getFragmentProgramParameters()->setNamedConstant("myParameter", value);

Thanks in advance!!
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4665
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 997
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by dark_sylinc »

Not yet.

There was/is a plan to allow for customizations interfaces. While they might not end up offering the full flexibility you need (depends on the features to be added...) as too much flexibility will hurt performance or make things harder, it could make the merging easier (or non existant if you're lucky).

The basic idea was that was that you would provide a template where custom pieces would be collected to be inserted in the shaders (so that you can customize the templates without actually modifying the original ones); and provide some sort of listener to setup per-pass data or setup properties at shader creation time (don't expect a listener to be called per object though, that ain't gonna happen!).
This isn't too hard, but might need to be handled with a bit of care.

Don't know when this feature is going to arrive though. There's still some important threading left to do to the Hlms (fillBuffersFor) which shouldn't affect such system, but if I'm wrong then that would make things more complicated if the customization system is already there, or may be not.
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 167

Re: [2.1] Easier way to communicate with hlms shader?

Post by xrgo »

Thank you for your answer, hope to see that feature soon =) thanks
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4665
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 997
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by dark_sylinc »

Hi xrgo. I just pushed this feature to the repo: https://bitbucket.org/sinbad/ogre/commi ... 6a3b3d8f26.

The manual explains a bit on how to do it. I haven't written a sample yet, showing how to do it. Do you think you can figure it out?
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 167

Re: [2.1] Easier way to communicate with hlms shader?

Post by xrgo »

Thank you so much!!!!!!!
dark_sylinc wrote:Do you think you can figure it out?
yes I think I can :)
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 167

Re: [2.1] Easier way to communicate with hlms shader?

Post by xrgo »

I just tried it and it worked like a charm =)
Thank you so much for this
aymar
Greenskin
Posts: 145
Joined: Fri Jun 12, 2015 6:53 pm
Location: Florianopolis, Brazil
x 17

Re: [2.1] Easier way to communicate with hlms shader?

Post by aymar »

Hey xrgo,

Can you post some info on how to use the HlmsListener to customize the PBS?

Thanks.
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 167

Re: [2.1] Easier way to communicate with hlms shader?

Post by xrgo »

Here is how I made some simple linear fog:

You have to inherit some class from HlmsListener

Code: Select all

class SomeClass : public Ogre::HlmsListener
and implement these methods:

Code: Select all

    //Comunicate stuffs to the PBS shader
    Ogre::uint32 getPassBufferSize( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager );

    float* preparePassBuffer( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager,
                                      float *passBufferPtr );
for instance for my Fog I implement them like this:

Code: Select all

Ogre::uint32 SomeClass::getPassBufferSize( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager ){
        return 32; //( vec4 fogParams + vec4 fogColor )*4
    }

float* SomeClass::preparePassBuffer( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                  bool dualParaboloid, Ogre::SceneManager *sceneManager,
                                  float *passBufferPtr ){

    //Linear Fog parameters
    *passBufferPtr++ = sceneManager->getFogStart();
    *passBufferPtr++ = sceneManager->getFogEnd();
    *passBufferPtr++ = 0.0;
    *passBufferPtr++ = 0.0f;

    *passBufferPtr++ = sceneManager->getFogColour().r;
    *passBufferPtr++ = sceneManager->getFogColour().g;
    *passBufferPtr++ = sceneManager->getFogColour().b;
    *passBufferPtr++ = 1.0f;
    return passBufferPtr;
}
Then you have to tell the hlmsPbs that SomeClass is the listener

Code: Select all

hlmsPbs->setListener(mySomeClass);
Then in the shaders:
If you notice in the file "Structs_piece_vs_piece_ps.glsl" in the "PassBuffer" there is a "@insertpiece( custom_passBuffer )", this means that you can make your own custom passBuffer extension that the hlms precompiler will automatically put where that "@insertpiece( custom_passBuffer )" is.
So you can make a file named: "CustomPassBuffer_piece_vs_piece_ps.glsl" with this inside:

Code: Select all

@piece( custom_passBuffer )

//Fog
	vec4 fogParams;
	vec4 fogColor;
	
@end
now you have in your "fogParams" and "fogColor" that you can use in the pbs shader!
And in the same way if you notice in the file "PixelShader_ps.glsl" there is a "@insertpiece( custom_ps_posExecution )", so you can make a file named "CustomPost_piece_ps.glsl" with this inside:

Code: Select all

@piece( custom_ps_posExecution )
@property( !hlms_shadowcaster )
//Fog
	float fogDistance = length( gl_FragCoord.z / gl_FragCoord.w );   // or better "length( inPs.pos )" for a more accurate distance
	float fogFactor = 1.0-clamp((pass.fogParams.y - fogDistance) / (pass.fogParams.y - pass.fogParams.x),0.0,1.0);
	vec3  fogColor  = vec3(pass.fogColor.xyz);
	outColour.xyz   = mix(finalColour, fogColor, fogFactor );
	
@end
@end
and that way you are actually modifying the shader without modifying the original files =)
Last edited by xrgo on Sun Apr 17, 2016 5:54 pm, edited 1 time in total.
aymar
Greenskin
Posts: 145
Joined: Fri Jun 12, 2015 6:53 pm
Location: Florianopolis, Brazil
x 17

Re: [2.1] Easier way to communicate with hlms shader?

Post by aymar »

That helps a lot!

Do you know if low level materials still support auto params?

Thanks!
xrgo
OGRE Expert User
OGRE Expert User
Posts: 1148
Joined: Sat Jul 06, 2013 10:59 pm
Location: Chile
x 167

Re: [2.1] Easier way to communicate with hlms shader?

Post by xrgo »

yes! low level mats works as always, just no Fixed function pipeline and you need to put the entity on a V1_LEGACY renderqueue group
Lax
Hobgoblin
Posts: 505
Joined: Mon Aug 06, 2007 12:53 pm
Location: Saarland, Germany
x 32

Re: [2.1] Easier way to communicate with hlms shader?

Post by Lax »

Hi,

I have a question, will this fog code snippets mentioned above be implemented as part of Ogre2.x?
Because if you update the ogre repository and have your own Fog implementation, this could become a mess quickly...

Regards
Lax
Image
http://www.lukas-kalinowski.com/Homepage/?page_id=1631
Please support Second Earth Technic Base built of Lego bricks for Lego ideas: https://ideas.lego.com/projects/81b9bd1 ... b97b79be62
Image
al2950
OGRE Expert User
OGRE Expert User
Posts: 1226
Joined: Thu Dec 11, 2008 7:56 pm
Location: Bristol, UK
x 157

Re: [2.1] Easier way to communicate with hlms shader?

Post by al2950 »

Lax wrote: Wed Mar 28, 2018 12:00 pm Because if you update the ogre repository and have your own Fog implementation, this could become a mess quickly...
I dont see this is as the case, in fact it is not the case! The fog implementation uses the standard interfaces for customizing a HLMS implementation. So you should face no difficulties in updating Ogre using your custom fog functions.
Lax
Hobgoblin
Posts: 505
Joined: Mon Aug 06, 2007 12:53 pm
Location: Saarland, Germany
x 32

Re: [2.1] Easier way to communicate with hlms shader?

Post by Lax »

Hi al2950,

ok thanks, I will try it out.

Regards
Lax
Image
http://www.lukas-kalinowski.com/Homepage/?page_id=1631
Please support Second Earth Technic Base built of Lego bricks for Lego ideas: https://ideas.lego.com/projects/81b9bd1 ... b97b79be62
Image
Lax
Hobgoblin
Posts: 505
Joined: Mon Aug 06, 2007 12:53 pm
Location: Saarland, Germany
x 32

Re: [2.1] Easier way to communicate with hlms shader?

Post by Lax »

Hi,

I'm looking at the fog implementation and the following came in my mind:

Code: Select all

hlmsPbs->setListener(mySomeClass);
Sets an active listener for hlms pbs, but if terra is used, there is already a listener "Ogre::HlmsPbsTerraShadows" set, so one listener would overwrite the other one.

I'm wondering, why several listeners cannot be registered like:

Code: Select all

hlmsPbs->addListener(mySomeClass);
Regards
Lax
Image
http://www.lukas-kalinowski.com/Homepage/?page_id=1631
Please support Second Earth Technic Base built of Lego bricks for Lego ideas: https://ideas.lego.com/projects/81b9bd1 ... b97b79be62
Image
hedphelym
Gremlin
Posts: 178
Joined: Tue Nov 25, 2008 10:58 am
Location: Kristiansand, Norway
x 20
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by hedphelym »

Hlsl version:

CustomPassBufferPiece_vs_piece_ps.hlsl

Code: Select all

@piece( custom_passBuffer )
	float4 fogParams;
	float4 fogColor;
@end
CustomPost_piece_ps.hlsl

Code: Select all

@piece( custom_ps_posExecution )
@property( !hlms_shadowcaster )
	float fogDistance =  length( inPs.pos );
	float fogFactor = clamp((passBuf.fogParams.y - fogDistance) / (passBuf.fogParams.y - passBuf.fogParams.x),0.0,1.0);
	outPs.colour0.rgb  = lerp(passBuf.fogColor.rgb,finalColour, fogFactor );
@end
@end
hedphelym
Gremlin
Posts: 178
Joined: Tue Nov 25, 2008 10:58 am
Location: Kristiansand, Norway
x 20
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by hedphelym »

When I run the fog with directx in debug mode it keeps hitting this assert in 'OgreHlmsPbs.cpp' :

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
Has anyone of you tested this? Or know why that happens?
I have used the code exactly as mentioned in the first post, with the HLSL code I posted right above this post.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4665
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 997
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by dark_sylinc »

Check that both getPassBufferSize and preparePassBuffer from your listener are getting called.

The listener's getPassBufferSize is supposed to tell Ogre that the buffer must be 32 bytes bigger. And preparePassBuffer should return a pointer that is advanced by 32 bytes (hence the post's code performs *passBufferPtr++ eight times).

The assert is there to check that everything that had to be sent to GPU got written, and that we didn't go past the bounds.
If the assert is failing, either getPassBufferSize/preparePassBuffer are not getting called, they are in disagreement, or something very weird is going on.

Watchout for alignToNextMultiple calls. I can't remember if they affect listener code, but it has bitten a few people a few times. Most variables have to be aligned to 16 bytes and this is painful to enforce when we allow arbitrary code to be inserted.
hedphelym
Gremlin
Posts: 178
Joined: Tue Nov 25, 2008 10:58 am
Location: Kristiansand, Norway
x 20
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by hedphelym »

The solution was very simple when I found it.
The getPassBufferSize function need to be const for it to be called.

Code: Select all

Ogre::uint32 PbsFog::getPassBufferSize( const Ogre::CompositorShadowNode *shadowNode, bool casterPass,
                                      bool dualParaboloid, Ogre::SceneManager *sceneManager ) const
                                      {
                                      	.....
                                      }
                                      
What happened was that It was not defined as const, thereby not being called, and ogre got '0' instead of '32'.
PBY
Halfling
Posts: 47
Joined: Tue Jul 16, 2013 12:57 pm
x 1

Re: [2.1] Easier way to communicate with hlms shader?

Post by PBY »

As this seems to be the only place where this simple way of adding small shader programs, and the documentation is useless, here is one point missing above:

Not only should your custom file end with the correct extension (like .hlsl), and declare the wanted @piece name, but the filename is also important.
OgreHlms will enumerate all files from the Hlms archive, but will only take file ending (before file extension) with one the following strings:
  • piece_vs
  • piece_ps
  • piece_gs
  • piece_hs
  • piece_ds
It will use the following list for finding the corresponding files:

Code: Select all

PieceFilePatterns[] = { "piece_vs", "piece_ps", "piece_gs", "piece_hs", "piece_ds" };
You should also use the correct ending, as it will take only one of the strings for each type, i.e. for creating the vertex shader, it will only search for piece_vs.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4665
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 997
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by dark_sylinc »

A few details to what you just said:
  1. You can concatenate these suffixes. MyFile_piece_vs_piece_ps.glsl means it will be parsed both during vertex and pixel shader stage.
  2. Use piece_all means it will be parsed in all shader stages (a shortcut to avoid concatenating all piece_*s suffixes)
  3. You can use the .any extension for files that should be parsed by any RenderSystem, ie. it gets parsed regardless of whether glsl/hlsl/metal is the currently active shader languague. You can also use @property( syntax == glsl )@end or @property( syntax != hlsl )@else @end to handle the minor differences
Lax
Hobgoblin
Posts: 505
Joined: Mon Aug 06, 2007 12:53 pm
Location: Saarland, Germany
x 32

Re: [2.1] Easier way to communicate with hlms shader?

Post by Lax »

Hi dark_sylinc,

I have an request for a new functionality for Hlms. At this time its only possible to set one listener, but I need two listeners. Because I use terra and hence for hlms the terra shadows listener is set. Now I want also set a custom fog listener. Which does work, but after that, terra does not work anymore, because its shadow listener has been overwritten.

Would it be possible to introduction a new function e.g.

Code: Select all

addListener
and

Code: Select all

removeListener
. So that custom several listener could be added for hlms?

That would be really great! What do you think?

Best Regards
Lax
Image
http://www.lukas-kalinowski.com/Homepage/?page_id=1631
Please support Second Earth Technic Base built of Lego bricks for Lego ideas: https://ideas.lego.com/projects/81b9bd1 ... b97b79be62
Image
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4665
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 997
Contact:

Re: [2.1] Easier way to communicate with hlms shader?

Post by dark_sylinc »

Can it be solved with a meta listener that calls the other listeners; and register them to your listener (instead of directly to Hlms)?
Lax
Hobgoblin
Posts: 505
Joined: Mon Aug 06, 2007 12:53 pm
Location: Saarland, Germany
x 32

Re: [2.1] Easier way to communicate with hlms shader?

Post by Lax »

hm, do you have an idea, how this could work in detail?
Because its required, that the HlmsPbs calls all virtual functions of all listeners...

Do you mean:
I use a base listener, which is registered to HlmsPbs and in this base listener, I call other custom listeners in all virtual functions?
Image
http://www.lukas-kalinowski.com/Homepage/?page_id=1631
Please support Second Earth Technic Base built of Lego bricks for Lego ideas: https://ideas.lego.com/projects/81b9bd1 ... b97b79be62
Image
User avatar
Zonder
Ogre Magi
Posts: 1155
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 66

Re: [2.1] Easier way to communicate with hlms shader?

Post by Zonder »

Lax wrote: Tue Feb 23, 2021 12:46 pm Do you mean:
I use a base listener, which is registered to HlmsPbs and in this base listener, I call other custom listeners in all virtual functions?
Yes I am pretty sure thats what he means.

I don't think we would want the extra complexity on the HLMS why he says this.
There are 10 types of people in the world: Those who understand binary, and those who don't...
Lax
Hobgoblin
Posts: 505
Joined: Mon Aug 06, 2007 12:53 pm
Location: Saarland, Germany
x 32

Re: [2.1] Easier way to communicate with hlms shader?

Post by Lax »

ok, I did as proposed, but its no more working. I always get an assert:

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
I now have a base listener, which gets registered directly to HlmsPbs and this listener has a vector list of contrete Listeners. Which are added.

Code: Select all

std::vector<Ogre::HlmsListener*> concreteListeners;
I tested it with a fog listener:

Code: Select all


	Ogre::uint32 HlmsFogListener::getPassBufferSize(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager) const
	{
		Ogre::uint32 size = 0;

		if (!casterPass)
		{
		        // size = 32
			size = sizeof(float) * 4 * 2; // float4 + float4 in bytes
		}
		return size;
	}
	
	float* HlmsFogListener::preparePassBuffer(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager, float* passBufferPtr)
	{
		if (!casterPass)
		{
			//Linear Fog parameters
			*passBufferPtr++ = sceneManager->getFogStart();
			*passBufferPtr++ = sceneManager->getFogEnd();
			*passBufferPtr++ = 0.0f;
			*passBufferPtr++ = 0.0f;

			*passBufferPtr++ = sceneManager->getFogColour().r;
			*passBufferPtr++ = sceneManager->getFogColour().g;
			*passBufferPtr++ = sceneManager->getFogColour().b;
			*passBufferPtr++ = 1.0f;
		}
		return passBufferPtr;
	}
And in the base listener, the function 'getPassBufferSize' for all listeners is called. In my example just the fog listener:

Code: Select all

Ogre::uint32 HlmsBaseListenerContainer::getPassBufferSize(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager) const
	{
		Ogre::uint32 size = 0;

		for (auto& it = this->concreteListeners.cbegin(); it != this->concreteListeners.cend(); ++it)
		{
			size += (*it)->getPassBufferSize(shadowNode, casterPass, dualParaboloid, sceneManager);
		}
		return size;
	}
	
	float* HlmsBaseListenerContainer::preparePassBuffer(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager, float* passBufferPtr)
	{
		for (auto& it = this->concreteListeners.cbegin(); it != this->concreteListeners.cend(); ++it)
		{
			(*it)->preparePassBuffer(shadowNode, casterPass, dualParaboloid, sceneManager, passBufferPtr);
		}
		
		return passBufferPtr;
	}
The size is incremented by the outcome of each concrete Listener. But somehow, Ogre does not accept the size anymore. The size is 256, but the assert is thrown:

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
--> 224 != 256
--> Somehow 32 bytes are missing

The strange thing is, that it did work before, when I registered directly the fog listener. So the question is, what is now different?? It should behave the same as before...

When I use the fog listener directly:

Code: Select all

assert( (size_t)(passBufferPtr - startupPtr) * 4u == mapSize );
--> 256 == 256
[Edit] Its because 'passBufferPtr' will become different, when its used in HlmsBaseListenerContainer! But I have no clue how to solve this...

[Edit2]: Ok I've got it! 'pussBufferPtr' must become a reference to pointer (in/out) variable, so that the adress is always the same. See:

Code: Select all

float* HlmsBaseListenerContainer::preparePassBuffer(const Ogre::CompositorShadowNode* shadowNode, bool casterPass, bool dualParaboloid, Ogre::SceneManager* sceneManager, float*& passBufferPtr)
@dark_sylinc: Would It be possible, that you add the '&' reference? It will not harm the pbs implementation and make it possible to use meta listeners!

Best Regards
Lax
Image
http://www.lukas-kalinowski.com/Homepage/?page_id=1631
Please support Second Earth Technic Base built of Lego bricks for Lego ideas: https://ideas.lego.com/projects/81b9bd1 ... b97b79be62
Image
Post Reply