MSAA error Topic is solved

[jwwalker]

I'm trying to get MSAA working in a Mac program with Metal rendering. In the createRenderWindow call, I added a parameter pair "FSAA", "4x". In my compositor, I added the "msaa_auto" option to all the "texture" lines. The first time I try to render, I get a fatal assertion failure inside Metal, which logs the message:

Code: Select all

-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5446: failed assertion `Draw Errors Validation
Fragment Function(main_metal): incorrect type of texture (MTLTextureType2DMultisample) bound at texture binding at index 0 (expect MTLTextureType2D) for depthMap[0].

So, Metal was expecting a non-multisample texture on the depth buffer? Was I not supposed to say "msaa_auto" on the depth textures?

[dark_sylinc]

It looks like something tried to bind the depth buffer for sampler. This is likely a pass_quad, but could be the HlmsPbs in certain cases (SSR, Refractions).

In such cases either the MSAA depth buffer must be downsampled into a regular buffer (eg. like Ogre/Resolve/1xFP32_Subsample0, SSR sample does this in ScreenSpaceReflectionsMsaaDepthResolve in ScreenSpaceReflections.compositor) or the pixel shader must be edited so that it can account for MSAA (the pixel shader must declare texture2d_msaa instead of texture2d; I can't remember the exact Metal syntax).

[jwwalker]

Thanks, it looks like the problem is a pass_quad at the end of my compositor, which copies the image from a texture to the window. I changed the shader to declare its inputs as texture2d_ms and depth2d_ms. Then I got a compile error on the unsampled reads I was using to get data from the textures. I can fix the compile error by adding a parameter that reads just one sample. I assume that looking at just one sample is missing the whole point of multisampled rendering, but I'll deal with that later. Anyway, here is the revised shader:

Code: Select all

#include <metal_stdlib>
#include <metal_math>
using namespace metal;


struct PS_OUTPUT
{
	float4 color [[color(0)]];
	float depth [[depth(any)]];
};


fragment PS_OUTPUT main_metal
(
	depth2d_ms<float, access::read>	depthMap  [[texture(0)]],
	texture2d_ms<float, access::read>	colorMap [[texture(1)]],
	float4 gl_FragCoord [[position]]
)
{
	PS_OUTPUT result;
	
	int2 center = (int2) gl_FragCoord.xy;
	
	float3 colorHere = colorMap.read( uint2( center ), 0 ).rgb;
	result.color.rgb = colorHere;
	result.color.a = 1.0;
	
	float depthHere = depthMap.read( uint2( center ), 0 );
	result.depth = depthHere;
	
	return result;
}

Now I get a runtime error that is kind of the opposite of the original one:

Code: Select all

-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5446: failed assertion `Draw Errors Validation
Fragment Function(main_metal): incorrect type of texture (MTLTextureType2D) bound at texture binding at index 0 (expect MTLTextureType2DMultisample) for depthMap[0].
Fragment Function(main_metal): incorrect type of texture (MTLTextureType2D) bound at texture binding at index 1 (expect MTLTextureType2DMultisample) for colorMap[0].

Any idea where a non-multisample texture is coming from now?

[dark_sylinc]

If that pass_quad is used in multiple places, now you could have the opposite problem after modifying it!

All I can suggest is that when XCode breaks into that error (IIRC, XCode allows breaking into Metal validation errors), just move up the callstack until you see the CompositorPassQuad and print the contents of mMaterial->mName or mDefinition->mMaterialName.

If XCode can't break you into the error, just fprintf( stderr, "MATERIAL: %s\n", mDefinition->mMaterialName.c_str() ); and see what was the last message before the error.

[jwwalker]

OK, that makes sense, there are situations where I deliberately set up a window with antialiasing turned off. So it seems like whenever I use a quad pass, I must have two versions of each shader, and two versions of the material, and two versions of the compositor node. But why don't the sample programs run into that problem, given that the startup dialog gives a choice of FSAA being 1x or something else?

[jwwalker]

More specifically, I'm looking at the postprocessing sample. Its initial compositor node declares two working textures:

Code: Select all

texture rt0 target_width target_height PFG_RGBA8_UNORM_SRGB msaa_auto
texture rt1 target_width target_height PFG_RGBA8_UNORM_SRGB msaa_auto

As I understand it, that msaa_auto means that maybe they're multisampled and maybe they're not, depending on the FSAA option chosen by the user at program startup.

To take one example, the Metal shader for the Postprocess/Posterize material declares its texture input as texture2d<float>. Why doesn't it get in trouble for not saying texture2d_ms if the texture is multisampled?

[jwwalker]

I thought maybe the problem was that I was reading rather than sampling from the textures, so I tried changing my shader to sample. But I still get the error about seeing a multisample texture when a non-multisample texture is expected.

[jwwalker]

It occurred to me that I don't really need to input or output depth when doing the final copy to the window. So I changed my shader to be the same code as Ogre's Copyback_4xFP32_ps.metal, and changed my material and compositor node to just pass in the color texture, not the depth, to the texture. Then the errors go away, but I have a black screen. Progress??? If I remove the msaa_auto option from the texture, it fixes the black screen, but presumably I'm not getting any antialiasing.

[dark_sylinc]

Hi!

Regarding your questions:

1. Some samples are simple enough that it doesn't matter. It just doesn't become a problem.
2. Other samples (like HDR, SSR) need a special path for MSAA. Often the best solution is that the MSAA compositor is very similar to the non-MSAA, but connects a node that workarounds the differences. This is what HDR sample does, which inserts a "HdrMsaaResolve" node while everything else is the same. SSR sample does the same thing, which inserts a "ScreenSpaceReflectionsMsaaDepthResolve" node.
3. The reason Postprocessing does not have this problem is because for colour targets (unless you request explicit_resolve) OgreNext will keep two textures as one: the MSAA texture and the resolve texture (which is a regular 2D texture). The compositor will render to the MSAA surface, resolve to the resolve surface; and the resolve surface is the one sampled by materials. However depth buffers can't use this trick. Your problem is caused by depth buffers, and the Postprocessing sample happens to not have any effect involving the depth buffer.

I strongly suggest you read Advanced MSAA section of the manual which explains what MSAA is.

Cheers

[jwwalker]

I read Advanced MSAA, but I'm still having a problem. As I understand it, if a rendering pass says store {all store_or_resolve}, then an MSAA color texture will be resolved, but the depth will remain multisampled. So if I start with a node like this,

Code: Select all

compositor_node JWRenderBase
{
    texture colorTx target_width target_height target_format msaa_auto
    texture depthTx target_width target_height PFG_D32_FLOAT msaa_auto
    
    rtv colorTx
    {
        depth depthTx
    }

    target colorTx
    {
        pass render_scene
        {
            load { all clear }
            store
            {
                colour store_or_resolve
                depth store
                stencil dont_care
            }
            rq_first    0
            rq_last     210
        }
        
    }
    
    out 0 colorTx
    out 1 depthTx
}

then the output color texture should be single-sample but the output depth will be multisample. If I then connect that up to this node

Code: Select all

compositor_node JWDepthResolve
{
    in 0 colorTx
    in 1 msaaDepth
    
    texture resolvedDepth target_width target_height PFG_D32_FLOAT

    target resolvedDepth
    {
        pass render_quad
        {
            load { all dont_care }
            material Ogre/Resolve/1xFP32_Subsample0
            input 0 msaaDepth
        }
    }
    
    out 0 colorTx
    out 1 resolvedDepth
}

then both outputs should be single-sample, no? But when I fed that output into another node that starts like this,

Code: Select all

compositor_node JWFinalNode
{
    in 2 ultimateDestination
    in 0 colorTx
    in 1 depthTx
    
    rtv colorTx2
    {
        colour colorTx
        depth depthTx
    }

    target colorTx2
    {
        pass render_scene

then I get an exception with this message:

Code: Select all

OGRE EXCEPTION(2:InvalidParametersException): Manually specified depth buffer 'resolvedDepth[Value 0x00000006]' is incompatible with colour RTT #0'colorTx[Value 0x00000004]
Colour: 448x448x1 4x MSAA PFG_BGRA8_UNORM_SRGB
Depth: 448x448x1 1x MSAA PFG_D32_FLOAT in RenderPassDescriptor::entriesModified

Where is Ogre getting those MSAA texture types?

[dark_sylinc]

Hi!

OgreNext's implicit MSAA textures means your colorTx texture is always internally 2 textures:

• MSAA when being used as a Render Target
• Regular texture when used for sampling

When at the final step you do:

Code: Select all

    target colorTx2
    {
        pass render_scene

You're asking OgreNext to render to the MSAA texture of colorTx again (which at this point contains undefined values that should be cleared or written over because the first operation resolved into the sampling texture, without keeping its contents).

Implicit MSAA textures are nice because when they work, they allow you to forget about having to deal with MSAA. But in more advanced examples, you may find more comfortable to just use explicit MSAA instead (and in fact some stuff is impossible to do with implicit MSAA).

The sample example with explicit MSAA textures would be like this:

Code: Select all

compositor_node JWRenderBase
{
    texture colorTxMsaa target_width target_height target_format msaa_auto explicit_resolve
    texture depthTxMsaa target_width target_height PFG_D32_FLOAT msaa_auto
    texture colorTx target_width target_height target_format
    
    rtv colorTxMsaa
    {
        depth	depthTxMsaa
        // Setup the RTV to resolve into colorTx.
        resolve	colorTx
    }

    target colorTxMsaa
    {
        pass render_scene
        {
            load { all clear }
            store
            {
                colour store_or_resolve
                depth store
                stencil dont_care
            }
            rq_first    0
            rq_last     210
        }
        
    }
    
    out 0 colorTx
    out 1 depthTxMsaa
}

compositor_node JWDepthResolve
{
    in 0 colorTx
    in 1 msaaDepth
    
    texture resolvedDepth target_width target_height PFG_D32_FLOAT

    target resolvedDepth
    {
        pass render_quad
        {
            load { all dont_care }
            material Ogre/Resolve/1xFP32_Subsample0
            input 0 msaaDepth
        }
    }
    
    out 0 colorTx
    out 1 resolvedDepth
}

compositor_node JWFinalNode
{
    in 2 ultimateDestination
    in 0 colorTx
    in 1 depthTx
    
    rtv colorTx2
    {
        colour colorTx
        depth depthTx
    }

    // WATCHOUT: At this point you're rendering WITHOUT MSAA. It seems this is what you intended.
    target colorTx2
    {
        pass render_scene

[dark_sylinc]

I forgot one tip: Because you probably want JWRenderBase to work with and without MSAA, it's better to define the textures in their own node:

Code: Select all

compositor_node JWTextureDefMsaa
{
    texture colorTxMsaa target_width target_height target_format msaa_auto explicit_resolve
    texture depthTxMsaa target_width target_height PFG_D32_FLOAT msaa_auto
    texture colorTx target_width target_height target_format

    out 0 colorTxMsaa
    out 1 depthTxMsaa
    out 2 colorTx
}

compositor_node JWTextureDef
{
    texture colorTx target_width target_height target_format
    texture depthTx target_width target_height PFG_D32_FLOAT

    out 0 colorTx
    out 1 depthTx
    out 2 colorTx // It's exported twice!
}
    
compositor_node JWRenderBase
{
    in 0 colorTxMsaa
    in 1 depthTxMsaa
    in 2 colorTx
    
    rtv colorTxMsaa
    {
        depth	depthTxMsaa
        // Setup the RTV to resolve into colorTx. If colorTxMsaa == colorTx then this is harmless.
        resolve	colorTx
    }

    target colorTxMsaa
    {
        pass render_scene
        {
            load { all clear }
            store
            {
                colour store_or_resolve
                depth store
                stencil dont_care
            }
            rq_first    0
            rq_last     210
        }
        
    }
    
    out 0 colorTx
    out 1 depthTxMsaa
}

This trick allows you to use JWRenderBase for both MSAA and non-MSAA without having to write it twice (which is important for complex nodes).

You'll need two workspaces, one that connects the MSAA will connect JWTextureDefMsaa to JWRenderBase; the non-MSAA will connect JWTextureDef to JWRenderBase.

[jwwalker]

Thank you very much for your help!

By the way, your latter tip did not quite work as stated, I got a warning saying RTV colorTxMsaa cannot be overriden because it comes from an input. Create a duplicate instead. So I changed the JWRenderBase node as follows:

Code: Select all

compositor_node JWRenderBase
{
    in 0 colorTxMsaa
    in 1 depthTxMsaa
    in 2 colorTx
    
    rtv colorRTV
    {
        depth   depthTxMsaa
        // Setup the RTV to resolve into colorTx. If colorTxMsaa == colorTx then this is harmless.
        colour 0 colorTxMsaa resolve colorTx
    }

    target colorRTV
    {
        pass render_scene
        {
            load { all clear }
            store
            {
                colour store_or_resolve
                depth store
                stencil dont_care
            }
            rq_first    0
            rq_last     210
        }
        
    }
    
    out 0 colorTx
    out 1 depthTxMsaa
}

Now everything compiles and runs, and I don't have a black screen. I'm not sure that the depth resolution is really working right, because later rendering passes look like they are receiving a clear depth buffer, but I'll leave that battle for another day.

[jwwalker]

I fixed my problem with depth resolution. Part of the problem was that the Ogre sample code was writing the resolved depth to a texture with pixel format PFG_R32_FLOAT, whereas I was writing to a texture with pixel format PFG_D32_FLOAT, so using a material with depth writing turned off was not going to work. I also had to change the Metal shader to declare the input multisampled depth as depth2d_ms rather than texture2d_ms.