Decal Shader Crash with Light Forward Mode

[Lax]

Hi dark_sylinc,

I ran into what looks like a shader generation issue with decals when Forward+ (hlms_forwardplus) is enabled.

Summary
When I enable Forward+ and place a decal, the generated D3D11 pixel shader fails to compile with:

Code: Select all

OGRE EXCEPTION(-2147467259:RenderingAPIException): Cannot compile D3D11 high-level shader ... Errors:
...PixelShader_ps.hlsl(...): error X3018: invalid subscript 'xyz'

If I set forward mode to none (i.e. disable Forward+), decals work fine.

What fails
The error happens inside the HLMS piece:

Code: Select all

@property( hlms_forwardplus )
@property( hlms_enable_decals )
@piece( forwardPlusDoDecals )
@insertpiece( forward3dHeader )

float3 posDdx = OGRE_ddx( inPs.pos.xyz );
float3 posDdy = OGRE_ddy( inPs.pos.xyz );

...
decalUvDdx.x = dot( invWorldView0.xyz, posDdx.xyz );
...


@end

The D3D compiler complains about an invalid subscript 'xyz'. That implies that in the generated permutation

Code: Select all

inPs.pos

(or one of the other inputs being swizzled) ends up as a scalar (or otherwise not a float3/float4) even though this piece expects it to be a vector.

Repro (minimal)

• Enable hlms_forwardplus (Forward+).
• Enable decals (hlms_enable_decals).
• Run a scene, place/create at least one decal.
• D3D11 shader compilation fails with X3018 invalid subscript 'xyz'.
• Disable Forward+ (forward mode = none) -> decals compile and work.

Hypothesis
It looks like in this Forward+ + Decals permutation, the PS input struct doesn’t always provide

Code: Select all

inPs.pos

as float3/float4 (world position), but the decals piece assumes it does.

So either:

• the PS input

  Code: Select all

  pos

  varying is conditionally omitted/typed differently in some permutations, or
• there’s a property/define mismatch where decals are enabled but the “world pos interpolant” is not emitted.

Possible fix direction
Ensure that when Forward+ decals are active, the PS input always contains

Code: Select all

inPs.pos

as float3/float4 (world position) and the VS writes it. In other words, make the “world position interpolant” mandatory for the permutations that include forwardPlusDoDecals.

Extra tiny guard fix suggestion (unrelated, but might help stability)
I also had an early crash with Ocean when there is no Terra yet, because

Code: Select all

mCurrentPassBuffer

can be 0 at a specific time during startup. Adding this guard avoided the crash:

Code: Select all

uint32 HlmsPbs::fillBuffersFor( const HlmsCache *cache, const QueuedRenderable &queuedRenderable,
bool casterPass, uint32 lastCacheHash, CommandBuffer *commandBuffer,
bool isV1 )
{
assert( dynamic_cast<const HlmsPbsDatablock *>( queuedRenderable.renderable->getDatablock() ) );
const HlmsPbsDatablock *datablock =
static_cast<const HlmsPbsDatablock *>( queuedRenderable.renderable->getDatablock() );

if( OGRE_EXTRACT_HLMS_TYPE_FROM_CACHE_HASH( lastCacheHash ) != mType )
{
    if( mCurrentPassBuffer == 0 )
        return 0;

...
}


}

Best Regards
Lax