Upgrading from 1.11.2 to 1.12.13 Topic is solved

[paroj]

I have no idea how to do that with git, as I just downloaded the entire source of Ogre (and have made many changes to it as well to meet my needs) and then compiled it, so I don't think it is connected in any way to git.

git bisect will not be fun then as it merely walks the commits (and thus discards your changes). You could "git stash" them and hope that they apply cleanly. What bisect does is basically just a binary search with the least amount of steps.

Looking at the file history I assume that the slowdown is due to a (seemingly) unrelated commit.

[rpgplayerrobin]

Ah...
So even with the old files used from 1.11.2 regarding all files starting with "OgreInstance", it still has only 52 FPS on that stress test scene.
That means that changes done to the instancing files has nothing to do with the performance difference.

Now I just wonder where that change may be in the code, and how to actually track it.
Normally, a bug is easy to find since it is reproducable easily, but only finding a bug that is causing a performance hit (and not that big either, only twice as slow as the previous version I used) might be very hard to do...

Has anything changed regarding rendering objects or something like that? I don't really know even where to start to solve this issue...

[paroj]

I have no idea how to do that with git, as I just downloaded the entire source of Ogre (and have made many changes to it as well to meet my needs) and then compiled it, so I don't think it is connected in any way to git.

git bisect will not be fun then as it merely walks the commits (and thus discards your changes). You could "git stash" them and hope that they apply cleanly. What bisect does is basically just a binary search with the least amount of steps.

Looking at the file history I assume that the slowdown is due to an unrelated commit.

edit: can you also take a look at the NewInstancing Sample whether it too exhibits that slowdown for you?

[rpgplayerrobin]

Yeah. Same issue in the sample browsers:

New (1.12.13) sample browser with 6000 entities (77*77):
108 FPS with instancing
35 FPS without instancing

Old (1.11.2) sample browser with 6000 entities (77*77):
308 FPS with instancing
50 FPS without instancing

So that kind of confirms that this has nothing to do with user-code.

By the way, maybe you missed my last post since it seems we posted at the exact same time : viewtopic.php?p=552875#p552875

[paroj]

might be the rendersystem. Do you also have this hit with GL3+? Besides that, you could capture a frame with renderdoc (D3D11, GL3+ only) and compare it between versions.

[rpgplayerrobin]

By the way, the first test up there in the new sample browser has the exact same results with the new or old versions of the "OgreInstance" files (tried by coping them directly from the 1.11.2 and making sure to compile it).
So the performance hit is somewhere else in Ogre.

When I try with Direct3D11 instead with 6000 objects using instancing in my game, these are the results I get:
1.11.2:
108 FPS with instancing, 18 FPS without instancing (compared to 107 and 42 on Direct3D9)

1.12.13:
110 FPS with instancing, 18 FPS without instancing (compared to 53 and 42 on Direct3D9)

So it seems that the problem with rendering with instancing in the newer version only has to do with Direct3D9, since Direct3D11 remain unchanged (though very poor at rendering many non-instancing objects for some reason , but that has at least not changed its performance between the new and the old version).

I do not have support in my game for GL3+ at the moment, only Direct3D9 and Direct3D11 currently.
I used CG for OpenGL, but that is deeply flawed since it only works on Nvidia cards, and it has many limitations regarding its compilation, so I need at a later stage fix OpenGL for my game without CG.

Since RenderDoc and graphics debugging in Visual Studio does not work for Direct3D9, how should I profile this?
Because the Direct3D11 works.

And I am not using DirectX9Ex if you are wondering.

[loath]

it seems like the quickest path is to use the bisect approach on a vanilla ogre and verify the performance change with the sample browser. there is a command line option to launch a particular test in the performance browser to minimize the UI steps to reproduce.

back when i used the dx9 render system i debugged with PIX which it comes with the june 2010 SDK. much faster to bisect though.
https://devblogs.microsoft.com/pix/download/

[paroj]

ah.. so we are talking about D3D9 here - in your first post you were saying D3D11. Then this might be a D3D9 specific regression.
From the top of my head, I would think of one of these:

• https://github.com/OGRECave/ogre/commit ... 82222bf21b
• https://github.com/OGRECave/ogre/commit ... 5a976995a3

[rpgplayerrobin]

I tried to get back to those versions, but it seems very hard since so much has changed.
The files have changed name and new files exists between the versions I am using, and diff:ing them in SVN shows an extreme amount of changes, which seems to be very hard to get back without just completely deleting the D3D9 system in the new version and then just using the old D3D9 system in the old version.
But that would not help other people, since the bug would not have been solved...

When I try to only change the most important places (like your code changes suggested as well), the FPS either goes down even more or it just crashes at startup.

And, I don't really understand these core functions for D3D9, it seems I would have to make a D3D9 system from scratch just trying to understand why this issue exists, but that would probably take weeks to do.

I tried downloading the samples for the latest version to see if the issue also exists there, but it seems the links are dead there (https://www.ogre3d.org/download/sdk/sdk-ogre) for the samples.
I managed to download the MSVC SDK file instead there and ran the sample browser, and it shows these stats:

New (13.3.4) sample browser with 6000 entities (77*77):
Direct3D9:
125 FPS with instancing
32 FPS without instancing
Direct3D11:
240 FPS with instancing
32 FPS without instancing

So it seems the issue still exists in the newest version of Ogre.
It should have 300 and 50 and FPS in Direct3D9, since it had that in the older version and in Direct3D11 (however, that figure changed for Direct3D11 in the newest version probably because of other changes, but it should at least be double in Direct3D9 than what it is there).

Again, when I say "instancing" I specifically mean HWInstancingBasic.

[rpgplayerrobin]

it seems like the quickest path is to use the bisect approach on a vanilla ogre and verify the performance change with the sample browser. there is a command line option to launch a particular test in the performance browser to minimize the UI steps to reproduce.

It seems this is the only way to do it... Though I have no experience with this at all.
I will have to research on what this exactly means and how to do it.

[rpgplayerrobin]

I found the version it stopped working in by downloading all prebuilt SDK:s in https://cloudsmith.io/_ogrecave/repos/ogre/packages/.

With instancing enabled for the different versions:
1.12.5: 245 FPS
1.12.6: 214 FPS
1.12.7: 217 FPS
1.12.8: 218 FPS
1.12.9: 98 FPS
1.12.10: 96 FPS

So the performance hit happened after version 1.12.8 and before or at version 1.12.9.

[rpgplayerrobin]

I downloaded both sources (1.12.8 and 1.12.9) and used SVN diff to check what changed.

With that I managed to fix it by copying over the content of the functions below from the 1.12.8 version:

OgreD3D9Mappings.cpp:
DWORD D3D9Mappings::get(HardwareBuffer::Usage usage)
DWORD D3D9Mappings::get(HardwareBuffer::LockOptions options, HardwareBuffer::Usage usage)

OgreD3D9HardwareBufferManager.cpp:
D3D9HardwareBufferManager::createVertexBuffer(size_t vertexSize, size_t numVerts, HardwareBuffer::Usage usage, bool useShadowBuffer)

OgreD3D9HardwareBuffer.cpp:
D3D9HardwareBuffer::D3D9HardwareBuffer(D3DFORMAT type, size_t sizeInBytes, Usage usage, bool useShadowBuffer)

I did not alter the constructor of the files, only the content of those 4 functions above.
I did however add "#define OGRE_D3D_MANAGE_BUFFERS 1" at the top of all those files I changed.

Now, I don't know exactly what the bug is, as soon as I change one of those functions back to the new version of it, the application crashes.
So in short, this is a fix for the issue, but I have no idea how to actually fix the issue with the newer changes of the 1.12.9+ versions.

But my game has now the exact same FPS in the stress test scene AND a in a normal game scene compared to the 1.11.2 version, so it works for sure.

If you manage to fix it for the latest version (13.3.4+) with this information I have supplied, let me know how you fixed it correctly instead!

[loath]

great work!

[paroj]

yeah, you reverted those commits I mentioned. Can you instead of doing that replace HBU_STATIC_WRITE_ONLY by HBU_DYNAMIC_WRITE_ONLY in OgreInstanceBatchHW.cpp?

[rpgplayerrobin]

yeah, you reverted those commits I mentioned. Can you instead of doing that replace HBU_STATIC_WRITE_ONLY by HBU_DYNAMIC_WRITE_ONLY in OgreInstanceBatchHW.cpp?

Yeah I also thought about that when I found the issue. When I tried those places before I think I changed too much instead, which made the game crash on startup.

And yeah, I changed those files back to the new (unmodified) versions and instead just changed HBU_STATIC_WRITE_ONLY to HBU_DYNAMIC_WRITE_ONLY in two places in that file, and it works as well as before!

So I guess that also helps for future versions of Ogre?

On towards the next bug now then!

[rpgplayerrobin]

The next bug is that grass in my game renders much slower than before in Direct3D9.
I realized it had to do with exactly the same thing as above, since it only happens in Direct3D9.

Then I saw that I again used HBU_STATIC_WRITE_ONLY there to create the vertex and index buffer, which would have to be switched to HBU_DYNAMIC_WRITE_ONLY to fix the issue like we did with InstanceBatchHW.
But... It did not fix the issue this time. I had to make the changes I made in the post viewtopic.php?p=552885#p552885 to make it work.
Now I can render grass at around twice the speed than before. So for some reason, the code in those functions in that post is extremely important in Direct3D9 for other stuff as well (possibly even for particles as well, not tested though).

[rpgplayerrobin]

I removed the change in the post viewtopic.php?p=552887#p552887 now completely, since that seems to be poison for Direct3D11 instead (75 FPS instead of 100 FPS when using instancing on the stress test map).
For Direct3D9, the changes in viewtopic.php?p=552885#p552885 makes those HBU_DYNAMIC_WRITE_ONLY changes not needed anyway, so it is perfect for both render systems by only changing the Direct3D9 files.

However, Direct3D11 seems to be very slow compared to Direct3D9 in a normal game scene (with or without that change above):

1.11.2: D3D9: 247 / D3D11: 237
1.12.13: D3D9: 250 / D3D11: 150

Keep in mind that that scene is exactly the same for both versions, and that both versions use almost exactly the same code for everything (apart from code changes needed for Ogre, exactly the same user-code).
The only thing that can have made this huge of a difference in FPS is that something might have changed in Direct3D11 between the versions...
This will have to wait for another day, it will probably take me 8 more hours to even understand why it happens...

[paroj]

To give some context first: Ogre used to mostly ignore what HBU_ you passed. Notably D3D9 with OGRE_D3D_MANAGE_BUFFERS. Therefore most code just passed arbitrary wrong HBU_ flags and got away with it. With recent releases, Ogre started to actually respect the HBU_ flags to select the buffer location. So the flags make a difference now and there are performance implications.
The good news is that you can improve performance by choosing the right HBU. The bad news is that old code with bad HBU flags will be slower.
Given that we have 7 RenderSystems now, there are probably bugs too.

So I guess that also helps for future versions of Ogre?

yes, HBU_DYNAMIC_WRITE_ONLY/ HBU_CPU_TO_GPU is the right usage, as the buffer is updated each frame. This also helps with other RenderSystems e.g. GL3+ gets 5% faster with this.

Then I saw that I again used HBU_STATIC_WRITE_ONLY there to create the vertex and index buffer, which would have to be switched to HBU_DYNAMIC_WRITE_ONLY to fix the issue like we did with InstanceBatchHW.

your index buffer probably should still be HBU_STATIC_WRITE_ONLY/ HBU_GPU_ONLY.
see

• https://ogrecave.github.io/ogre/api/lat ... ffer-Usage
• https://ogrecave.github.io/ogre/api/lat ... 36606dba64

since that seems to be poison for Direct3D11 instead (75 FPS instead of 100 FPS when using instancing on the stress test map).

I would assume that this is no longer the case with 13.x, where D3D11 buffer handling was updated again to support cbuffers. From your experiments D3D11 was significantly faster on 13.x than on 1.12.x.

[rpgplayerrobin]

To give some context first: Ogre used to mostly ignore what HBU_ you passed. Notably D3D9 with OGRE_D3D_MANAGE_BUFFERS. Therefore most code just passed arbitrary wrong HBU_ flags and got away with it. With recent releases, Ogre started to actually respect the HBU_ flags to select the buffer location. So the flags make a difference now and there are performance implications.
The good news is that you can improve performance by choosing the right HBU. The bad news is that old code with bad HBU flags will be slower.
Given that we have 7 RenderSystems now, there are probably bugs too.

Ok, so that means that I should really keep the OGRE_D3D_MANAGE_BUFFERS changes in my version, because otherwise it gets slowed down with a lot of different stuff, and the HBU_DYNAMIC_WRITE_ONLY change is a no-go because of the performance with Direct3D11 in my version, which is hopefully fixed in your (13.X) version.

I would assume that this is no longer the case with 13.x, where D3D11 buffer handling was updated again to support cbuffers. From your experiments D3D11 was significantly faster on 13.x than on 1.12.x.

It is hard to translate that though, since that was before the HBU_STATIC_WRITE_ONLY -> HBU_DYNAMIC_WRITE_ONLY change in the code.
After that change it showed the incorrect behaviour (the 25% less performance impact).
If 13.X has that change, you can try to make it back to HBU_STATIC_WRITE_ONLY and see if D3D11 is faster, because in my case it is for my version (1.12.13).

Anyway, I found the culprit regarding the Direct3D11 performance issue on a normal game scene (in 1.12.13).
It has to do with non-GPU animation (more specifically, skeletal vertex animation).
On a normal game scene with around 60 animated objects, like characters/plants/trees/etc, it has 500 FPS (my maximum limit in the game) with GPU animation enabled.
The animated objects are not rendered with instancing or anything like that, it is just normal shaders with GPU animation turned on (not RTSS).
But as soon as I turn off GPU animation, it goes down to 160 FPS.
When I turn off GPU animation in Direct3D9, it stays at the maximum FPS (500), and in Direct3D11 in the last version (1.11.2) it only goes down to 495 FPS.
When I turn off GPU animations, the code sets the new shaders in the materials by setting the vertex/fragment shaders for the normal rendering and for the shadow caster.
My code or shaders has not changed at all since the last version.
So in short, I get 160 FPS in this example scene in Direct3D11, but I expect closer to 500 (like it was in the last version).

What can cause non-GPU skeletal vertex animation in Direct3D11 to have gone down in performance so much?

[paroj]

this is still the HBU_ behavior, so likely

• https://github.com/OGRECave/ogre/commit ... 1ecb406ed5
• https://github.com/OGRECave/ogre/commit ... 78b7fee0f9

with software skeletal animation the GPU buffers are updated each frame. If HBL_DISCARD cannot be used and the buffer is HBU_GPU_ONLY, the update is slow but the pipeline must wait for the update, which is probably happening there.

[paroj]

for reference, with current master I get with 10k instances (all visible) and HW instancing (HBU_STATIC_WRITE_ONLY/ HBU_DYNAMIC_WRITE_ONLY)

• D3D9: 158/270
• GL3+: 260/280
• D3D11: 290/305

the large (2.5ms) gap with D3D9 is due to HBL_DISCARD not being available with HBU_STATIC_WRITE_ONLY.

[rpgplayerrobin]

for reference, with current master I get with 10k instances (all visible) and HW instancing (HBU_STATIC_WRITE_ONLY/ HBU_DYNAMIC_WRITE_ONLY)

D3D9: 158/270
GL3+: 260/280
D3D11: 290/305
the large (2.5ms) gap with D3D9 is due to HBL_DISCARD not being available with HBU_STATIC_WRITE_ONLY.

Well, at least it is good that it is fixed there now.

I have been trying for a while to get those merges you showed me into my code, but the code looks completely different from that one so I have no idea how to do this...
I also downloaded the source for 1.12.8 and 1.12.9 and diff:ed those files, but that made it just even more confusing since it does not even find all variables if I use those versions of the files. Also, it seems many variables have moved elsewhere.
For example, OgreD3D11HardwareVertexBuffer and OgreD3D11HardwareIndexBuffer has been merged into one class in my version (OgreD3D11HardwareBuffer), and many variables are different. Those were still split up in 1.12.8 and 1.12.9 and in the merges you showed me, so it is pretty hard to do this, but I will try.

It might have just been faster to edit my 1.11.2 version with exactly what I needed instead of trying to upgrade, there are so many bugs in 1.12.13.

However, I think this is the last big issue, so I basically only need to fix this Direct3D11 bug (somehow, not sure how) and I am almost done with the upgrade.

[rpgplayerrobin]

I think I finally managed to understand the code changes since then and made all of the fixes.
However, it did not help with the performance at all.
You can double check here if I actually did it correctly: https://fastupload.io/f42qjbGNDDmc506

If the code seems correct, there must be something else that makes the FPS go down so much, though I have no idea what in that case.

If the code is not correct, what did I do wrong?
If it is too complex to fix it, can I instead take the Direct3D11 files from somewhere else where it might be fixed?
I tried using all D3D11 files from the 13.3.4 version, but it would not compile (too many other changes, like inheritance stuff not existing).

[paroj]

I think I finally managed to understand the code changes since then and made all of the fixes.
However, it did not help with the performance at all.

great! this means that I did not break anything there in the first place

You can double check here if I actually did it correctly: https://fastupload.io/f42qjbGNDDmc506

If the code seems correct,

yes, from a quick glance it seems correct. However, it would really make my life easier if I could just look at a niceley formatted github commit instead of a zip file from a shady website..

to further investigate this, I would suggest that you step through Mesh::softwareVertexBlend. Particularly check whether:

• HBL_DISCARD can be used here https://github.com/OGRECave/ogre/blob/v ... .cpp#L2007
• There is no GPU>CPU transfer here (shadow buffer should be used): https://github.com/OGRECave/ogre/blob/1 ... .cpp#L1977
• on exit from the function, you should end up here: https://github.com/OGRECave/ogre/blob/1 ... r.cpp#L223 and the whole buffer should be discarded

[rpgplayerrobin]

I only tried it in release so far...
As soon as I tried to debug that function, I ran in debug and it asserts at:
OgreAssertDbg(!mUseTempStagingBuffer, "should be handled by _updateFromShadow");
in "D3D11HardwareBuffer::unlockImpl", so apparently the code is not correct after all (very hard to merge two different version when all files/functions are not 1:1).

yes, from a quick glance it seems correct. However, it would really make my life easier if I could just look at a niceley formatted github commit instead of a zip file from a shady website..

I now learned how to do it in github instead , here is a commit between the different versions:
https://github.com/rpgplayerrobin/TestO ... 7292a531a8
or just here if the commit link is strange (I am not used to github at all):
https://github.com/rpgplayerrobin/TestO ... mmits/main

So I guess there is something wrong there since I cannot even start it in debug.

I will revert the code back to the old version temporarily and debug the mesh code instead and see where that takes me.