[2.2] Porting my engine to D3D11

[xrgo]

Hello!
I am trying to port my engine to D3D11 (because since ever I always used OGL), for that I disabled almost everything, all my shader customization, made a BasicWorkspaceDef.. so I make sure I am not using any glsl stuffs and I am getting this crash:

D3D11 ERROR: ID3D11DeviceContext::CopySubresourceRegion: When offset by the destination coordinates, pSrcBox does not fit on the destination subresource. OffsetSrcBox = { left:0, top:0, front:16, right:16, bottom:16, back:17 }

Code: Select all

1   RaiseException                                    KERNELBASE                      0x7ffb3def9149 
2   CxxThrowException                                 VCRUNTIME140D                   0x7ffb2c6e7b60 
3   Ogre::ExceptionFactory::throwException            OgreException.h            277  0x7ff71d7aa781 
4   Ogre::D3D11StagingTexture::upload                 BaseProjectDev                  0x7ff71dcac765 
5   Ogre::ObjCmdBuffer::UploadFromStagingTex::execute OgreObjCmdBuffer.cpp       189  0x7ff71e58579d 
6   Ogre::ObjCmdBuffer::execute                       OgreObjCmdBuffer.cpp       109  0x7ff71e5851ac 
7   Ogre::TextureGpuManager::_update                  OgreTextureGpuManager.cpp  3105 0x7ff71dd55d03 
8   Ogre::RenderSystem::_update                       OgreRenderSystem.cpp       1348 0x7ff71e3c43e1 
9   Ogre::CompositorManager2::_updateImplementation   OgreCompositorManager2.cpp 758  0x7ff71e0c098b 
10  Ogre::RenderSystem::updateCompositorManager       OgreRenderSystem.cpp       1357 0x7ff71e3c4461 
11  Ogre::CompositorManager2::_update                 OgreCompositorManager2.cpp 655  0x7ff71e0c0308 
12  Ogre::Root::_updateAllRenderTargets               OgreRoot.cpp               1578 0x7ff71dcdc422 
13  Ogre::Root::renderOneFrame                        OgreRoot.cpp               1100 0x7ff71dcd9a84

I have no idea where to begin

Also I have this piece of code for selecting rendersystem:

Code: Select all

    Ogre::String buffer;
    if( yUtils::getConfigFileSetting("RenderSystem", buffer) && buffer == "D3D" ) {
        Ogre::D3D11Plugin* d3dPlugin = OGRE_NEW Ogre::D3D11Plugin();
        Ogre::Root::getSingleton().installPlugin(d3dPlugin);
    }
    else{
        //Install RenderSystem Plugin
        Ogre::GL3PlusPlugin* glPlugin = OGRE_NEW Ogre::GL3PlusPlugin();
        Ogre::Root::getSingleton().installPlugin(glPlugin);
    }

but I need to include this:

Code: Select all

#include "OgreGL3PlusPlugin.h"
#include "OgreGL3PlusRenderSystem.h"

#include "OgreD3D11Plugin.h"
#include "OgreD3D11RenderSystem.h"

but when I include both I get redefiniton errors

error: C2011: 'Ogre::FrameBufferDescKey': 'struct' type redefinition
error: C2011: 'Ogre::FrameBufferDescValue': 'struct' type redefinition

is there a trick to avoid that? although this is 0 important since I´ll probably just keep D3D11 if it gives me better performance

thanks!

[xrgo]

Hello! I am resuming this and I found that crash, its a 3D texture that's causing it, so for now I am just omitting it, now it runs!
Now I am having trouble porting my skydome shader, its based on this viewtopic.php?f=25&t=83763&p=519279#p519340 but its basically:
VS:

Code: Select all

#version 430
// Do not forget this line first. I had problems with the parser if it wasn't before the other layout vars. Moreover, it indicates some infos to the compiler (see Dark_Sylinc's answer)
layout (std140) uniform ;

// Input
layout (binding = 0) uniform passBuffer
{
	mat3 invViewMatCubemap;
	// has to be vec4 everything, even if not used entirely
	mat4 viewProj;
	mat4 viewProjRight;

	float timeOfDay;
	///some other pass vars
} passVars ;

// Inputs : those are key words for Ogre, i believe. You can't mess with it
in vec4 vertex ;
in vec2 uv0 ;
in vec2 uv1 ;
in vec2 uv2 ;

// The drawId is some info given by you in the HLMS to track which Ogre::Item is getting its draw.
in uint drawId ;

// We will only use the binding 2 here
layout (binding = 2) uniform entityBuffer
{
   // The array is here because our buffer will contain the datas for (max) 1024 Ogre::Item. The drawId is here to help you get the good index.
   // As a ConstBuffer will have a fixed size, if you change this uniform's structure, the array will be shorter (again, see Dark_Sylinc's answer). Be careful with the memory padding too.
   mat4 worldMat [1024] ;
} entityVars ;

// Output
out gl_PerVertex
{
   vec4 gl_Position ;
} ;

// Fragment shader's output
out block
{
	vec2 uv0;
	///some other outPs vars
} outPs ;

void main ()
{

///LOTS OF NON RELEVANT CODE.............

  outPs.uv0 = uv0;
	
	vec4 wpos = vertex * entityVars.worldMat[drawId];
  gl_Position = wpos * passVars.viewProj;

}

PS:

Code: Select all

#version 430

// Again, do not forget this line
layout (std140) uniform ;

// Input
layout (binding = 0) uniform passBuffer
{
	mat3 invViewMatCubemap;
	// has to be vec4 everything, even if not used entirely
	mat4 viewProj;
	mat4 viewProjRight;

	float timeOfDay;
	///some other pass vars
} passVars;

uniform sampler2D skyTone;
///some other samplers


// Datas from the Vertex shader
in block
{
	vec2 uv0;
	///some other inPs vars
} inPs;

// Output
out vec4 outColour ;

void main ()
{

///LOTS OF NON RELEVANT CODE.............

	vec2 uv0 = inPs.uv0;
	outColour = vec4(1,0,0,1);
}

I have no idea where to start (0 d3d knowledge), does anyone have a hlsl version of that custom hlms shader made by Kinslore? or there is another shader similar that I can reference?

thanks in advance!

[dark_sylinc]

Of all places, Microsoft has a good GLSL -> HLSL reference.

Code: Select all

//GLSL
layout (binding = 0) uniform passBuffer
{
//...
} passVars;

//HLSL
struct PassBufferStruct
{
//...
};
cbuffer passBuffer : register( b0 )
{
    PassBufferStruct passVars;
}

For vertex inputs you can actually change the names (we try to avoid the variable name 'vertex' because it's a reserved word in Metal, and most of HLSL and Metal code can be used interchangeably):

Code: Select all

//GLSL
// Inputs : those are key words for Ogre, i believe. You can't mess with it
in vec4 vertex ;
in vec2 uv0 ;
in vec2 uv1 ;
in vec2 uv2;

//HLSL
struct MyVertex
{
  float4 myPos : POSITION;
  float2 myUv0 : TEXCOORD0;
  float2 whateverYouWant : TEXCOORD1;
  float2 anotherUv2 : TEXCOORD2;
};

void main( MyVertex inVs )
{
}

HLSL splits samplers and textures, while GLSL has both of them combined. So:

Code: Select all

//GLSL
uniform sampler2D skyTone;
vec4 val = texture( skyTone, uv0.xy );

//Becomes:
Texture2D skyTone : register( t0 );
SamplerState skyToneSampler : register( s0 ); //Contains info such as point/bi-/trilinear/anisotrpic filtering, UV wrapping modes, border colour, etc

float4 val = skyTone.Sample( skyToneSampler,  uv0.xy );

Note that you can reuse a sampler for multiple textures, which is something GLSL can't do. For example you can do this:

Code: Select all

Texture2D skyTone : register( t0 );
Texture2D anotherTexture : register( t1 );

SamplerState trilinearSampler : register( s0 );

float4 val0 = skyTone.Sample( trilinearSampler,  uv0.xy );
float4 val1 = anotherTexture.Sample( trilinearSampler,  uv0.xy );

This is a performance optimization (since GLSL needs to allocate registers to store the other sampler used with "anotherTexture", while in HLSL you can share)

Other things to watch for:

1. In GLSL mat3 -> float3x3. However under correctly-performing drivers, the following:

Code: Select all

mat3 myMat;
float myVal;

Is the same (memory wise) in GLSL as this:

Code: Select all

vec4 row0_padding;
vec4 row1_padding;
vec4 row2_padding;
vec4 myVal_paddingX3;

However under HLSL rules, myVal gets packed together:

Code: Select all

float3x3 myMat;
float myVal;

Is the same (memory wise) as this:

Code: Select all

float4 row0_padding;
float4 row1_padding;
float4 row2_myVal;

See HLSL packing rules.

2. float3x4 vs float4x3 and mat3x4 vs mat4x3. They're hard to get right due to row major vs column major. Chances are you need to transpose or switch the order of multiplication.
In Ogre 2.2 I created a series of macros under the name "ogre_float4x3" to get it working consistently.

3. HLSL supports passing params from VS to PS both as single variables, or as a struct. In HLSL, declaration order is important and must be consistent between VS and PS.
Because SV_Position is often not used by the pixel shader (but is required to be an output by the vs), declare it always at the end. Otherwise if you declare it at the beginning, you must declare it in the pixel shader as well. This is because declaration order is important and although you can skip declaring variables in the PS, you cannot do so if you need the variable that comes after the one you skip (i.e. don't leave gaps):

Code: Select all

//VS HLSL
struct VS_OUTPUT
{
	float2 uv0			: TEXCOORD0;
	float4 gl_Position	: SV_Position;
};


//Pixel Shader HLSL:
struct VS_OUTPUT
{
	float2 uv0			: TEXCOORD0;
	//float4 gl_Position	: SV_Position; Not needed, won't use it. Commented out
};

4. Matrix multiplication in HLSL must be done via the intrinsic "mul( a, b )". There is no "*" operator like in GLSL.

For more examples, check the shaders under Samples/Media/2.0/scripts/materials/Common and see the differences between the HLSL and GLSL shaders.
Start with simple ones like Copyback_1xFP32_ps, Quad_vs and QuadCameraDir_vs.

Samples under Samples/Media/2.0/scripts/materials/HDR are also useful because they're a bit more advanced but still simple enough.


Once you're comfortable with HLSL, if you're using the Hlms you may find that @insertpiece( SetCrossPlatformSettings ) defined in CrossPlatformSettings_piece_all contains a lot of useful macros to port your shaders between GLSL and HLSL.

For example "vec4" can be written as "float4" in GLSL, mul(), lerp() and saturate() also work in GLSL, and uniform blocks can be written as:

Code: Select all

//GLSL
layout (binding = 0) uniform passBuffer
{
//...
} passVars;

//Portable HLSL and GLSL
CONST_BUFFER_STRUCT_BEGIN( passBuffer, 0 )
{
//...
} CONST_BUFFER_STRUCT_END( passVars );

Textures can be sampled via:

Code: Select all

//GLSL
uniform sampler2DArray myTexture;
float4 finalVal = texture( myTexture, uv, arrayIdx );

//HLSL
Texture2DArray myTexture : register( t0 );
SamplerState mySampler : register( s0 );

float4 finalVal = myTexture.Sample( mySampler, float3( uv, arrayIdx ) );

//Portable HLSL and GLSL (you still have to declare with language-specific syntax myTexture and mySampler in HLSL and myTexture in GLSL)
float4 finalVal = OGRE_SampleArray2D( myTexture, mySampler, uv, arrayIdx );

[xrgo]

Thank you so much for the detailed explanation!
its working now!!!!!

[xrgo]

Hello!, now I need that texture that's causing the crash

this is the error:

D3D11 ERROR: ID3D11DeviceContext::CopySubresourceRegion: When offset by the destination coordinates, pSrcBox does not fit on the destination subresource. OffsetSrcBox = { left:0, top:0, front:16, right:16, bottom:16, back:17 }. DstSubresource = { left:0, top:0, front:0, right:16, bottom:16, back:16 }. [ RESOURCE_MANIPULATION ERROR #280: COPYSUBRESOURCEREGION_INVALIDSOURCEBOX]

this is the texture (its a 3D texture):
https://www.dropbox.com/s/75crodirnat7i ... e.dds?dl=0

and this is how I am loading it:

Code: Select all

    mWindNoiseTexture = textureMgr->createOrRetrieveTexture(
                                    "windNoise.dds",
                                    Ogre::GpuPageOutStrategy::Discard,
                                    Ogre::TextureFlags::PrefersLoadingFromFileAsSRGB,
                                    Ogre::TextureTypes::Type3D,
                                    Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME );
    mWindNoiseTexture->scheduleTransitionTo( Ogre::GpuResidency::Resident );

it works fine with OpenGL, any ideas?

[xrgo]

and I have this other error with another texture (a 2D texture 4x4 full transparent)

D3D11 ERROR: ID3D11Device::CreateTexture2D: A Texture2D created with the following Format (0x4e, BC3_UNORM_SRGB) experiences aligment restrictions on the dimensions of the Resource. The dimensions, which are (Width: 2, Height: 2), must be multiples of (Width: 4, Height: 4). [ STATE_CREATION ERROR #101: CREATETEXTURE2D_INVALIDDIMENSIONS]
D3D11 ERROR: ID3D11Device::CreateTexture2D: Returning E_INVALIDARG, meaning invalid parameters were passed. [ STATE_CREATION ERROR #104: CREATETEXTURE2D_INVALIDARG_RETURN]

the texture:
https://www.dropbox.com/s/6oc3jrhy8cacn ... A.dds?dl=0

[xrgo]

I found another texture that's causing this error (this time a png):

D3D11 ERROR: ID3D11DeviceContext::CopySubresourceRegion: When offset by the destination coordinates, pSrcBox does not fit on the destination subresource. OffsetSrcBox = { left:0, top:0, front:8, right:8, bottom:8, back:9 }. DstSubresource = { left:0, top:0, front:0, right:8, bottom:8, back:8 }. [ RESOURCE_MANIPULATION ERROR #280: COPYSUBRESOURCEREGION_INVALIDSOURCEBOX]

but!!! this same texture is loading in another project, so the problem really might be in another place... strangely, I have almost not modified C++ code, most of it has been shader work.. I'll keep looking, any ideas on how to solve this would be appreciated =)

PS: I tested some scenes in my home computer (with an 8 years old mid-tier card) and an Oculus DK2, and in a simple scene with OGL I get 17fps and with DX11 I get 75fps (which is the max for the DK2)... and a more complex scene OGL 12fps, DX11 33fps. yay!!! I just hope that I also get this improvement in a modern VR Ready PC, I will probably test this tomorrow.

[dark_sylinc]

I found another texture that's causing this error (this time a png):

D3D11 ERROR: ID3D11DeviceContext::CopySubresourceRegion: When offset by the destination coordinates, pSrcBox does not fit on the destination subresource. OffsetSrcBox = { left:0, top:0, front:8, right:8, bottom:8, back:9 }. DstSubresource = { left:0, top:0, front:0, right:8, bottom:8, back:8 }. [ RESOURCE_MANIPULATION ERROR #280: COPYSUBRESOURCEREGION_INVALIDSOURCEBOX]

The majority of those errors looks like a bug on our part.
StagingTexture managing in D3D11 is a nightmare and we have to partition 2D regions of free unused space in a staging texture. It's possible this partition algorithm has a bug (i.e. marks a region as being free... outside the staging texture).
Another possible bug is that we're using TextureGpu::mInternalSliceStart incorrectly (only applicable for AutomaticBatching textures), such as adding it twice or forgetting to use it. Another possibility is just an off by 1 error.

The thing with D3D11 StagingTexture management is that it depends on texture loading order. Loading A.png then B.png works fine, but loading B.png then A.png triggers the bug.
This is like slicing a cake with pieces of different sizes. If you do it in order you get to eat the whole cake, do it wrong and some slice can't be big enough because you had been cutting random slices in the middle.

Use OGRE_FORCE_TEXTURE_STREAMING_ON_MAIN_THREAD macro to force deterministic load order, then try to repro this bug on a sample. TextureResidency can be of great help. Edit "const Ogre::String textureNames[]" with paths to your textures (which btw you can conveniently dump via TextureGpuManager::dumpMemoryUsage) until you've repro'd the crash with as few textures loaded as possible.

The actual contents of the textures don't matter, only matters their resolution and format.

The only thing that is not a bug is the BC3_UNORM_SRGB texture one. As the message from D3D11 API explains, BC3 textures of 2x2 are forbidden (despite mipmaps of 2x2 being allowed...). Why? I don't know. Certainly GPUs can do it, DX9 can do it, OpenGL can do it.... D3D11 decided to drop support for it.
The same happens for any BC1 and BC3 texture whose resolution is not multiple of 4.

[xrgo]

thank you! I am going to try to repro

it depends on texture loading order. Loading A.png then B.png works fine, but loading B.png then A.png triggers the bug

that explains the "randomness"

BC3 textures of 2x2 are forbidden

:S but the texture is 4x4 , I am going to try other sizes anyways, or maybe somehow I confused the texture, I'll look into it

Muchas gracias!!

[xrgo]

well, that was pretty easy to reproduce =)
I just added this:

Code: Select all

        Ogre::TextureGpu* ogreTexture = textureManager->createOrRetrieveTexture(
            "1.dds",
            Ogre::GpuPageOutStrategy::Discard,
            Ogre::TextureFlags::AutomaticBatching,
            Ogre::TextureTypes::Type2D,
            Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME);
        ogreTexture->scheduleTransitionTo(Ogre::GpuResidency::Resident);
        ogreTexture->waitForData();

        Ogre::TextureGpu* ogreTexture2 = textureManager->createOrRetrieveTexture(
            "2.dds",
            Ogre::GpuPageOutStrategy::Discard,
            Ogre::TextureFlags::PrefersLoadingFromFileAsSRGB,
            Ogre::TextureTypes::Type3D,
            Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME);
        ogreTexture2->scheduleTransitionTo(Ogre::GpuResidency::Resident);
        ogreTexture2->waitForData();

        Ogre::TextureGpu* ogreTexture3 = textureManager->createOrRetrieveTexture(
            "3.png",
            Ogre::GpuPageOutStrategy::Discard,
            Ogre::TextureFlags::AutomaticBatching,
            Ogre::TextureTypes::Type2D,
            Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME);
        ogreTexture3->scheduleTransitionTo(Ogre::GpuResidency::Resident);
        ogreTexture3->waitForData();

at the end of createScene01 on the PbsMaterials sample

this are the textures
https://www.dropbox.com/s/ki1xpl9hlnhuf ... s.zip?dl=0

Edit: mmmmmmmm it appears to also happens in OGL with this repro... I had never seen this error in my engine with OGL though

[dark_sylinc]

Yeah I just tried it in OpenGL and there's at the very least an off by one error on the 3D texture.

fullDstTextureBox.depth = 16; however dstBox.z = 16 (max value can be 15; since dstBox.z + dstBox.depth must be <= 16)

Update: Pushed fix. I only tested OpenGL (it's 2 am, I'm going to sleep now), but this bug should clearly affect D3D11 & Metal as well.
And it should've been affecting your engine as well in OGL. The only 2 ways you wouldn't see this bug is that somehow the mips were getting dropped early (is this possible? I don't remember), or you had asserts disabled.

Thanks for the repro!

[xrgo]

thank you!!! its working!!
I am going to post my next problem tomorrow (also late here)