[2.3] Vulkan Progress

Discussion area about developing with Ogre-Next (2.1, 2.2 and beyond)


rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

dark_sylinc wrote: Mon Sep 07, 2020 3:17 am IT LIVES!!!
Great!!!
BTW: Yesterday I've updated my dev machine's Ogre to yesterday's latest commit on v2-2-vulkan branch (the "Vulkan is runing on Anroid!" one) and used it for work for a whole day (D3D11 only. I don't have time to test Vulkan) without any problem. Of course my daily work doesn't involve too many Ogre features but at least the most common ones are still working.

I'll continue to use it for my daily work (probably update it again during the week if I have time) and probably test a bit of Vulkan next week if I have some time :)
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

dark_sylinc, In case you're interested, here is what I found yesterday, when upgrading Ogre:

1. in OgreHlms.cpp
It was:

Code: Select all

    #if OGRE_DEBUG_MODE >= OGRE_DEBUG_HIGH
        mDebugOutputProperties( true ),
    #else
        mDebugOutputProperties( false ),
    #endif
But now the first one also changed to false... is it intentional?

2. in some compositors, "keep_content" was added. what's that?

3. SMAA:
* In SMAA.hlsl's comment, #include changed to #//include, probably because #include is processed by Ogre now, and the preprocessor is not strong enough to realize it's inside a comment 8-)
* The texture format PFG_D24_UNORM_S8_UINT => PFG_D32_FLOAT_S8X24_UINT

4. Two places needs to be ported. They're legacy codes that I wrote a long time ago and it's trivial to change.
* _setTexture() added a bDepthOnly parameter
* one call to CompositorManager2::addWorkspace() needs changing because ResourceLayoutMap + ResourceAccessMap => ResourceStatusMap

5. I didn't try using Vulkan in our app but I did try to build Ogre's Vulkan RenderSystem and run the samples.

Debug:
* FindVulkan.cmake: "find_library(Vulkan_SHADERC_LIB_DBG"'s HINTS is missing some 4 lines, compared to Vulkan_SHADERC_LIB_REL. I had to add these lines myself because I don't want to copy VulkanSDK into ${OGRE_DEPENDENCIES_DIR}/lib
* Vulkan RenderSystem got a lot of linking errors... of the same kind:
2> shaderc_combined.lib(ShaderLang.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in OgreVulkanAsyncTextureTicket.obj

Release(RelWithDebInfo):
* Sample_PbsMaterials is running succesfully!
* Only fullscreen mode works. non-fullscreen mode gives white screen (no error in Ogre.log).
* "VSync=no" seems not working (still locked to FPS=60)
* FSAA=16 not working (seems like driver crash?), FSAA=8 is working. BTW: D3D11 doesn't have the option FSAA=16 now!

I'm using Win10, VulkanSDK 1.2.148.1, CPU&GPU is:

Code: Select all

21:37:16:  *   CPU ID: AuthenticAMD: AMD Ryzen 5 3600X 6-Core Processor
21:37:16:  *   Logical cores: 12
...
21:37:16: RenderSystem Name: Vulkan Rendering Subsystem
21:37:16: GPU Vendor: nvidia
21:37:16: Device Name: GeForce GTX 1660
21:37:16: Driver Version: 432.0.0.0
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

rujialiu wrote: Mon Sep 07, 2020 3:00 pm dark_sylinc, In case you're interested, here is what I found yesterday, when upgrading Ogre:

1. in OgreHlms.cpp
It was:

Code: Select all

    #if OGRE_DEBUG_MODE >= OGRE_DEBUG_HIGH
        mDebugOutputProperties( true ),
    #else
        mDebugOutputProperties( false ),
    #endif
But now the first one also changed to false... is it intentional?
I need to remember to toggle it on when I merge it to master. The properties were getting in the way of 'there is an error in line <X>' while porting GLSL shaders to Vulkan.
2. in some compositors, "keep_content" was added. what's that?
If you see 'Transitioning texture <Texture Name> from Undefined to a read-only layout. Perhaps you didn't want to set TextureFlags::DiscardableContent / aka keep_content in compositor?' messages, then you to add 'keep_content' to the compositor's declaration. e.g.

Old:

Code: Select all

texture prevFrameDepthBuffer	target_width target_height PFG_R32_FLOAT uav
New:

Code: Select all

texture prevFrameDepthBuffer	target_width target_height PFG_R32_FLOAT uav keep_content
This is normal for textures whose contents are meant to be carried over from the previous frame.
Debug:
* FindVulkan.cmake: "find_library(Vulkan_SHADERC_LIB_DBG"'s HINTS is missing some 4 lines, compared to Vulkan_SHADERC_LIB_REL. I had to add these lines myself because I don't want to copy VulkanSDK into ${OGRE_DEPENDENCIES_DIR}/lib
This is intentional. You need to pull ogre-next-deps.
* Vulkan RenderSystem got a lot of linking errors... of the same kind:
2> shaderc_combined.lib(ShaderLang.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in OgreVulkanAsyncTextureTicket.obj
This is why it's intentional :lol:
* Only fullscreen mode works. non-fullscreen mode gives white screen (no error in Ogre.log).
That's weird. Definitely a first.
* "VSync=no" seems not working (still locked to FPS=60)
I think on Windows it should be working but I'm not sure. Technically VSync=No can fail but I find it strange NVIDIA drivers would not expose this.

In the Ogre.log you should find this message:
'Trying presentMode = '

IMMEDIATE_KHR is VSync = off.
FIFO_KHR is VSync = on.
MAILBOX_KHR is VSync = on + VSync mode = Lowest Latency

If you see we tried IMMEDIATE_KHR and failed, it means the driver won't allow turning VSync off.
Perhaps NVIDIA wants us to use VSync = On + Lowest Latency VSync mode, which is very close to turning VSync off but without tearing.

If we never tried IMMEDIATE_KHR, then it's a silly argument parsing bug in our Win32 backend.
* FSAA=16 not working (seems like driver crash?), FSAA=8 is working. BTW: D3D11 doesn't have the option FSAA=16 now!
The FSAA values on Vulkan are currently hardcoded, regardless of support
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

Today I've updated my dev machine's Ogre to v2-2-vulkan ("Fix NotTexture flag being ignored") and pull ogre-next-deps to compile shaderc manually for Debug 8-)

When compiling shaderc, I got the following error with VS2019 (16.3.10)

Code: Select all

2>    CMake Error at C:/Program Files/CMake/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
2>      Could NOT find PythonInterp: Found unsuitable version "2.7.12", but
2>      required is at least "3" (found C:/Python27/python.exe)
2>    Call Stack (most recent call first):
2>      C:/Program Files/CMake/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:376 (_FPHSA_FAILURE_MESSAGE)
2>      C:/Program Files/CMake/share/cmake-3.15/Modules/FindPythonInterp.cmake:160 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
2>      cmake/setup_build.cmake:52 (find_package)
2>      CMakeLists.txt:59 (include)
I have both Python 3 and 2 installed on my dev machine, and python 3 takes precedence (typing python in cmd.exe enters python 3), however, FindPythonInterp.cmake finds python 2 first and stopped :(

Defining PYTHON_EXECUTABLE in cmake or environment variable doesn't help, so I manually changed all PythonInterp calls to Python3 (only 2 files are changed: shaderc/src/CMakeLists.txt and shaderc/src/cmake/setup_build.cmake)

However, I still got the following message:

Code: Select all

2>      Update build-version.inc in the Shaderc build directory (if necessary).
2>      Traceback (most recent call last):
2>        File "C:/ogremygui/ogredep19/src/vulkan/shaderc/src/utils/update_build_version.py", line 154, in <module>
2>          main()
2>        File "C:/ogremygui/ogredep19/src/vulkan/shaderc/src/utils/update_build_version.py", line 139, in main
2>          for (p, d) in zip(projects, sys.argv[1:])
2>        File "C:/ogremygui/ogredep19/src/vulkan/shaderc/src/utils/update_build_version.py", line 125, in get_version_string
2>          detailed_version_string_lst.append(deduce_software_version(directory))
2>        File "C:/ogremygui/ogredep19/src/vulkan/shaderc/src/utils/update_build_version.py", line 80, in deduce_software_version
2>          with open(changes_file, errors='replace') as f:
2>      TypeError: 'errors' is an invalid keyword argument for this function
It looks like update_build_version.py is still executed with python 2 somehow. I don't know why. I restarted cmd.exe, cmake and vs2019 but it still doesn't work.

Maybe restarting my computer will work, but after removing the "errors='replace'" part, the script can be executed with python 2 and the build is now successful!

When compiling the update Ogre, I got a compilation error but trivial to fix. In OgreD3D11TextureGpu.cpp:

Code: Select all

    D3D11TextureGpuRenderTarget::D3D11TextureGpuRenderTarget(
            GpuPageOutStrategy::GpuPageOutStrategy pageOutStrategy,
            VaoManager *vaoManager, IdString name, uint32 textureFlags,
            TextureTypes::TextureTypes initialType,
            TextureGpuManager *textureManager ) :
        D3D11TextureGpu( pageOutStrategy, vaoManager, name,
                         textureFlags, initialType, textureManager ),
        mDepthBufferPoolId( 1u ),
        mPreferDepthTexture( false ),
        mDesiredDepthBufferFormat( PFG_UNKNOWN ),
        mOrientationMode( msDefaultOrientationMode )
    {
Initialization of mOrientationMode should be wrapped #if/#endif.

The debug build is running!! Non-fullscreen mode still doesn't work, but this time I can see an assertion failed after some time! in OgreVulkanWindow.cpp:

Code: Select all

    VkSemaphore VulkanWindow::getImageAcquiredSemaphore( void )
    {
        OGRE_ASSERT_LOW( mSwapchainStatus != SwapchainReleased );
Then I spent a few minutes testing Vulkan with our app (just used another set of configs and set shader syntax/resource directory carefully) and got the following error:

Code: Select all

Error on shader HDR/DownScale03_SumLumEnd_ps_VK: params buffer slot must be a uniform buffer
I've changed the fp definition to (just replacing glsl to hlsl):

Code: Select all

fragment_program HDR/DownScale03_SumLumEnd_ps_VK hlslvk
{
	source DownScale03_SumLumEnd_ps.hlsl
}
We're modified quite a few post-processing shaders so I'd like to make hlslvk work instead of re-apply our changes to your updated glslvk.

Any suggestions? Are there any easy way to test hlslvk (postprocess + hlms) in some simple Ogre sample before tackling our app?
happyOgreRust
Gnoblar
Posts: 6
Joined: Mon Aug 10, 2020 3:31 am

Re: [2.3] Vulkan Progress

Post by happyOgreRust »

I am wondering can we do async compute in current vulkan implementation?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

happyOgreRust wrote: Fri Sep 11, 2020 3:16 pm I am wondering can we do async compute in current vulkan implementation?
Yes, in "limited form" because right now everything is just one queue (the Graphics queue) and only barriers determine what is done asynchronously. This is very coarse and if you execute, in this order:

1. Independent job A1
2. Dependent job A2
3. Independent job B1
4. Dependent job B2

Job A1 will execute, then both A2 and B1 executing concurrently, then B2. This is already possible with OpenGL (but most drivers tend to execute everything serially, except Mesa). If you launch them in this order:

1. Independent job A1
2. Independent job B1
3. Dependent job A2
4. Dependent job B2

Jobs A1 and B1 will execute concurrently, then A2 and B2 concurrently. This can be mentally stressing to group by hand, and we aren't currently reordering jobs automatically (we have the information to do so, though).

In the future I want to be able to move work between queues; so that pure-compute can be moved to its own queue without getting interrupted by rendering jobs. The main use case for this is to perform rendering, then postprocessing on another queue, while rendering begins working on the next frame already.

While this is possible to do with just one queue, it is extremely difficult to get right because it is very likely a barrier will sneak and block asynchronicity.
rujialiu wrote: Fri Sep 11, 2020 11:46 am
When compiling shaderc, I got the following error with VS2019 (16.3.10)

Code: Select all

...
I have both Python 3 and 2 installed on my dev machine, and python 3 takes precedence (typing python in cmd.exe enters python 3), however, FindPythonInterp.cmake finds python 2 first and stopped :(
Mmm this is from shaderc (not Ogre code), and 'it just worked for me', I'll stay alert on more reports like this. I have yet to see what Python versions I have installed on my machine.
rujialiu wrote: Fri Sep 11, 2020 11:46 am ation error but trivial to fix. In OgreD3D11TextureGpu.cpp:

Code: Select all

    D3D11TextureGpuRenderTarget::D3D11TextureGpuRenderTarget(
            GpuPageOutStrategy::GpuPageOutStrategy pageOutStrategy,
            VaoManager *vaoManager, IdString name, uint32 textureFlags,
            TextureTypes::TextureTypes initialType,
            TextureGpuManager *textureManager ) :
        D3D11TextureGpu( pageOutStrategy, vaoManager, name,
                         textureFlags, initialType, textureManager ),
        mDepthBufferPoolId( 1u ),
        mPreferDepthTexture( false ),
        mDesiredDepthBufferFormat( PFG_UNKNOWN ),
        mOrientationMode( msDefaultOrientationMode )
    {
Initialization of mOrientationMode should be wrapped #if/#endif.
Heh, I pushed this without checking anything, so I'm not surprised. Thanks!
rujialiu wrote: Fri Sep 11, 2020 11:46 am The debug build is running!!
Nice!
rujialiu wrote: Fri Sep 11, 2020 11:46 amNon-fullscreen mode still doesn't work
That is SO weird! I only tested one GPU vendor on Windows though. I have yet to test Intel. I can't currently test NVIDIA.
rujialiu wrote: Fri Sep 11, 2020 11:46 ambut this time I can see an assertion failed after some time! in OgreVulkanWindow.cpp:

Code: Select all

    VkSemaphore VulkanWindow::getImageAcquiredSemaphore( void )
    {
        OGRE_ASSERT_LOW( mSwapchainStatus != SwapchainReleased );
Mmmm.... this shouldn't happen but my subconscious has been telling me for weeks me there is something wrong with that code.

Does this happen after alt+tab? Or just randomly while you're flying through?

Code: Select all

Error on shader HDR/DownScale03_SumLumEnd_ps_VK: params buffer slot must be a uniform buffer
If you're using HLSL, I noticed that for glslang this:

Code: Select all

void main( uniform float myParam ) {}
is not the same as this:

Code: Select all

void main( uniform float myParam )
{
}
In D3D11 we support both variations (not both at once though), but it seems glslang only generates the right code for one of these variants.
We're modified quite a few post-processing shaders so I'd like to make hlslvk work instead of re-apply our changes to your updated glslvk.
This is around my objectives too (but admitedly very low priority right now). Everything I could check online strongly recommends to use DirectXCompiler instead of glslang though.
Any suggestions? Are there any easy way to test hlslvk (postprocess + hlms) in some simple Ogre sample before tackling our app?
I stopped looking into glslang's HLSL output so your guess is as good as mine. RenderDoc shows the disasm of spirv in human readable form (you may be able to spit it out tweaking SpvOptions) and compare a GLSL vs an HLSL equivalent to see what's going wrong.
happyOgreRust
Gnoblar
Posts: 6
Joined: Mon Aug 10, 2020 3:31 am

Re: [2.3] Vulkan Progress

Post by happyOgreRust »

That will be cool to support multi queues.

Another question is, I want to implement some gpu culling in ogre, is the indirect draw ok in vulkan implementation? I tried directx 11, but with some exception about "pBufferForArgs" , And also compute pass between render pass dependency is needed, cull results read back is needed. Hope for your suggestions.

Some like https://github.com/nvpro-samples/gl_occlusion_culling
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

dark_sylinc wrote: Fri Sep 11, 2020 5:34 pm Mmm this is from shaderc (not Ogre code), and 'it just worked for me', I'll stay alert on more reports like this. I have yet to see what Python versions I have installed on my machine.
Yeah... I know it's not Ogre code. I'm posting it here just because other people might run into this issue too :)
dark_sylinc wrote: Fri Sep 11, 2020 5:34 pm Mmmm.... this shouldn't happen but my subconscious has been telling me for weeks me there is something wrong with that code.
Does this happen after alt+tab? Or just randomly while you're flying through?
No alt-tab. Just waited a few seconds after seeing a white screen. I think I know the reason now by looking at Ogre.log:

Code: Select all

ERROR: [Validation] Code 0 : Validation Error: [ VUID-VkSwapchainCreateInfoKHR-imageExtent-01274 ] Object 0: handle = 0x1fca76da440, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x7cd0911d | vkCreateSwapchainKHR() called with imageExtent = (1440,900), which is outside the bounds returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR(): currentExtent = (1440,881), minImageExtent = (1440,881), maxImageExtent = (1440,881). The Vulkan spec states: imageExtent must be between minImageExtent and maxImageExtent, inclusive, where minImageExtent and maxImageExtent are members of the VkSurfaceCapabilitiesKHR structure returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR for the surface (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkSwapchainCreateInfoKHR-imageExtent-01274)
There is only one option in "Video Mode": 1440 x 900 so I just can't make other choices.
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

happyOgreRust wrote: Sat Sep 12, 2020 2:19 am That will be cool to support multi queues.

Another question is, I want to implement some gpu culling in ogre, is the indirect draw ok in vulkan implementation? I tried directx 11, but with some exception about "pBufferForArgs" , And also compute pass between render pass dependency is needed, cull results read back is needed. Hope for your suggestions.

Some like https://github.com/nvpro-samples/gl_occlusion_culling
Hello, happyOgreRust! Looking at your username, are you planning to use Ogre with Rust programming language in some way? 8-) Just curious...
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

rujialiu wrote: Sat Sep 12, 2020 3:54 am No alt-tab. Just waited a few seconds after seeing a white screen. I think I know the reason now by looking at Ogre.log:

Code: Select all

ERROR: [Validation] Code 0 : Validation Error: [ VUID-VkSwapchainCreateInfoKHR-imageExtent-01274 ] Object 0: handle = 0x1fca76da440, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x7cd0911d | vkCreateSwapchainKHR() called with imageExtent = (1440,900), which is outside the bounds returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR(): currentExtent = (1440,881), minImageExtent = (1440,881), maxImageExtent = (1440,881). The Vulkan spec states: imageExtent must be between minImageExtent and maxImageExtent, inclusive, where minImageExtent and maxImageExtent are members of the VkSurfaceCapabilitiesKHR structure returned by vkGetPhysicalDeviceSurfaceCapabilitiesKHR for the surface (https://vulkan.lunarg.com/doc/view/1.2.148.0/windows/1.2-extensions/vkspec.html#VUID-VkSwapchainCreateInfoKHR-imageExtent-01274)
There is only one option in "Video Mode": 1440 x 900 so I just can't make other choices.
Ahh I see! Hotshot5000 copied the resolution code from GL and it may need review.
Additionally windowMovedOrResized hasn't been implemented, so if the window is created to e.g. 1440x900 but then it shrinks because the title bar doesn't fit, it will cause swapchain problems.

I strongly suspect that's why you can't see in windowed mode
happyOgreRust
Gnoblar
Posts: 6
Joined: Mon Aug 10, 2020 3:31 am

Re: [2.3] Vulkan Progress

Post by happyOgreRust »

rujialiu wrote: Sat Sep 12, 2020 3:55 am
happyOgreRust wrote: Sat Sep 12, 2020 2:19 am That will be cool to support multi queues.

Another question is, I want to implement some gpu culling in ogre, is the indirect draw ok in vulkan implementation? I tried directx 11, but with some exception about "pBufferForArgs" , And also compute pass between render pass dependency is needed, cull results read back is needed. Hope for your suggestions.

Some like https://github.com/nvpro-samples/gl_occlusion_culling
Hello, happyOgreRust! Looking at your username, are you planning to use Ogre with Rust programming language in some way? 8-) Just curious...

Yes, my friend. Planning to expose to rust. Like rust + gdnative. But godot not suitable for make a render library.
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

happyOgreRust wrote: Sat Sep 12, 2020 9:44 am Yes, my friend. Planning to expose to rust. Like rust + gdnative. But godot not suitable for make a render library.
That's great!!! I'm still a rust beginner (only read half of "the book") but I alread liked it very much!
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

dark_sylinc wrote: Sat Sep 12, 2020 5:40 am I strongly suspect that's why you can't see in windowed mode
Me too :) Anyway, that's not a very big problem so you can do it later...

I've spent a bit of time investigating Vulkan+HLSL issue today. My plan is to change Ogre's HDR sample to use DownScale03_SumLumEnd_ps.hlsl for Vulkan, so I changed HDR.material

Code: Select all

fragment_program HDR/DownScale03_SumLumEnd_ps_VK hlslvk
{
	source DownScale03_SumLumEnd_ps.hlsl
}
The only change is glslvk -> hlslvk and .glsl -> .hlsl. No more changes.

Before debugging, I learnt a bit of "vulkan hlsl" by compiling the glsl -> spirv and use spirv-cross to get back an hlsl:

Code: Select all

glslc -fshader-stage=frag 1.glsl -o 1.spv
spirv-cross 1.spv --hlsl --output 1.hlsl --stage frag --shader-model 50
It looks like glsl's `layout(binding = x)` directly corresponds to hlsl's register(tx/sx/bx), but uniform parameters in the `Params` struct is translated into:

Code: Select all

cbuffer Params : register(b0)
{
    float3 _34_exposure : packoffset(c0);
    float _34_timeSinceLast : packoffset(c0.w);
    float4 _34_tex0Size : packoffset(c1);
};
instead of parameters of main(), as in current hlsl file:

Code: Select all

float4 main
(
	in float2 uv : TEXCOORD0,
	uniform float3 exposure,
	uniform float timeSinceLast,
	uniform float4 tex0Size
) : SV_Target
So I changed it accordingly. Then I got this:

Code: Select all

Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-InconsistentSpirv ] Object 0: handle = 0x12578cefc10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bbb14 | SPIR-V module not valid: In the Vulkan environment, the OriginLowerLeft execution mode must not be used.
  OpExecutionMode %main OriginLowerLeft
Luckily, I realized that it was discussed earlier in this thread (thanks Hotshot5000 and dark_sylinc!). However, Hotshot5000's solution no longer works because I'm getting this:

Code: Select all

Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-InconsistentSpirv ] Object 0: handle = 0x151617f4ad0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bbb14 | SPIR-V module not valid: Invalid SPIR-V binary version 1.1 for target environment SPIR-V 1.0 (under Vulkan 1.0 semantics).
Luckily, I found I can "partially" apply his changes to remove the validation error by only adding these two lines:

Code: Select all

            glslang::EShTargetClientVersion VulkanClientVersion = glslang::EShTargetVulkan_1_1;
            shader.setEnvClient( glslang::EShClientVulkan, VulkanClientVersion );
Then I run into this assertion failed in VulkanProgram.cpp

Code: Select all

        if( type != VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER )
        {
            OGRE_EXCEPT( Exception::ERR_INVALIDPARAMS,
                         "Error on shader " + mName + ": params buffer slot must be a uniform buffer",
                         "VulkanProgram::buildConstantDefinitions" );
        }
After a bit of debugging, I found the reason: Ogre's expecting a VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER at binding=1, but there are multiple `SpvReflectDescriptorBinding`s with binding=1.
findBinding() finds another one with binding=1 first! I've manually checked the spirv compiled with hlsl by spirv-dis and confirmed that registers t0, s0, b0 will all compiled to binding=0! (I don't know whether this is really valid, though...)

Anywany, I changed `VulkanProgram::findBinding` to accept a descriptor_type and then it successfully finds the descriptor with VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER.

BTW: I have to hard-code the register to b1 (`cbuffer Params : register(b1)`) because ogre_P0 expands to `c1` instead of `b1`.

However, I got a couple of other validation error (and then crashed):

Code: Select all

Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-DescriptorTypeMismatch ] Object 0: handle = 0x22a68770f00, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x790fd336 | Type mismatch on descriptor slot 0.0 (expected `VK_DESCRIPTOR_TYPE_SAMPLER, VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER`) but descriptor of type VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER
Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-DescriptorTypeMismatch ] Object 0: handle = 0x22a68770f00, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x790fd336 | Type mismatch on descriptor slot 0.0 (expected `VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE`) but descriptor of type VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER
Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-DescriptorTypeMismatch ] Object 0: handle = 0x22a68770f00, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x790fd336 | Type mismatch on descriptor slot 0.1 (expected `VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE`) but descriptor of type VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER
Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-DescriptorTypeMismatch ] Object 0: handle = 0x22a68770f00, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x790fd336 | Type mismatch on descriptor slot 0.1 (expected `VK_DESCRIPTOR_TYPE_SAMPLER, VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER`) but descriptor of type VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER
Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-InputNotProduced ] Object 0: handle = 0x7dceed0000000068, name = HDR/DownScale03_SumLumEnd_ps_VK, type = VK_OBJECT_TYPE_SHADER_MODULE; | MessageID = 0x23e43bb7 | fragment shader consumes input location 0.0 which is not written by vertex shader
I'm running out of time today so I stopped. It would be nice if dark_sylinc can go a little bit further and make this simple change (only use DownScale03_SumLumEnd_ps.hlsl) work. And maybe you're right, we probably should go with DXC instead, but what does it mean really? Does that mean our clients must upgrade their graphics card drivers to support VK1.1 or even VK1.2? I hope not, and I have an impression that DXC supports VK1.0
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

First a small disclaimer: my primary goal is getting Ogre working very well on Android (not perfect because that's almost impossible due to driver bugs, but as best as possible) and this goal seems to have been achieved (I'm cleaning up the repo right now, preparing docs on how to compile for Android).

Therefore while HLSL compilation is very desirable, it is not in my top priority list right now. I won't be working in HLSL in the foreseeable future.
rujialiu wrote: Sun Sep 13, 2020 11:21 am Does that mean our clients must upgrade their graphics card drivers to support VK1.1 or even VK1.2?
It means the client must upgrade their drivers, but not their graphics cards (unless the vendor has stopped issuing updates for that card, but AFAIK any Vulkan-capable card hasn't been EOL'ed yet, except for Intel).
rujialiu wrote: Sun Sep 13, 2020 11:21 am and I have an impression that DXC supports VK1.0
I have that impression too, after reading their docs. But I'm unsure if they meant VK1.0 + extension, or just bare VK1.0.

Note: In OpenGL, extensions are enabled automatically whether you want it or not. In Vulkan these extensions must be requested explicitly. Usually extensions are later gathered and upgraded to a major version e.g. the extensions form part of base VK1.1
Extensions are requested in either VulkanRenderSystem::_createRenderWindow (device extensions) or in VulkanRenderSystem::initializeVkInstance (instance extensions). Most Vulkan extensions are device ones.

For Desktop this isn't much of a problem, for Android this is a major one because most phones don't ever get a GPU driver update.

rujialiu wrote: Sun Sep 13, 2020 11:21 am
dark_sylinc wrote: Sat Sep 12, 2020 5:40 am I strongly suspect that's why you can't see in windowed mode
Me too :) Anyway, that's not a very big problem so you can do it later...
Actually this should've been taken care of (hopefully) in the latest commits. Hopefully it fixes most of your issues. I have yet to try Intel for myself.
Before debugging, I learnt a bit of "vulkan hlsl" by compiling the glsl -> spirv and use spirv-cross to get back an hlsl:
That functionality is actually thought to be able to use GLSL with D3D11, rather than with Vulkan.
The generated code is meant to be used with D3D11 (or even D3D9 if the right parameters are set). The goal is write code once, use it everywhere.
instead of parameters of main(), as in current hlsl file:

Code: Select all

float4 main
(
	in float2 uv : TEXCOORD0,
	uniform float3 exposure,
	uniform float timeSinceLast,
	uniform float4 tex0Size
) : SV_Target
So I changed it accordingly. Then I got this:

Code: Select all

Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-InconsistentSpirv ] Object 0: handle = 0x151617f4ad0, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bbb14 | SPIR-V module not valid: Invalid SPIR-V binary version 1.1 for target environment SPIR-V 1.0 (under Vulkan 1.0 semantics).
Luckily, I found I can "partially" apply his changes to remove the validation error by only adding these two lines:

Code: Select all

            glslang::EShTargetClientVersion VulkanClientVersion = glslang::EShTargetVulkan_1_1;
            shader.setEnvClient( glslang::EShClientVulkan, VulkanClientVersion );
In OgreVulkanDevice.cpp you should also change VK_MAKE_VERSION( 1, 0, 2 ) to VK_MAKE_VERSION( 1, 1, 0 ).
Then I run into this assertion failed in VulkanProgram.cpp

Code: Select all

        if( type != VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER )
        {
            OGRE_EXCEPT( Exception::ERR_INVALIDPARAMS,
                         "Error on shader " + mName + ": params buffer slot must be a uniform buffer",
                         "VulkanProgram::buildConstantDefinitions" );
        }
Like I said in a post before, putting uniform params like arguments to a function i.e. void main( uniform float myParam ); makes glslang to miscompile the program. Put them as globals instead.

glslang will otherwise generate code that expects a vertex buffer to be bound to myParam's slot; which is NOT what we want (it should be an uniform buffer)
findBinding() finds another one with binding=1 first! I've manually checked the spirv compiled with hlsl by spirv-dis and confirmed that registers t0, s0, b0 will all compiled to binding=0! (I don't know whether this is really valid, though...)
Yeah that is invalid. The registers in Vulkan must be assigned by hand however, which probably your spirv-cross isn't doing.

In Ogre we tackle this problem with the ogre_xN macros. I strongly urge you to read OgreRootLayout.h documentation. Once you're done reading it, read it again :)

It will explain how slots are assigned to shaders in Vulkan, and managed from C++ as well.
However, I got a couple of other validation error (and then crashed):

I'm running out of time today so I stopped. It would be nice if dark_sylinc can go a little bit further and make this simple change (only use DownScale03_SumLumEnd_ps.hlsl) work. And maybe you're right, we probably should go with DXC instead, but what does it mean really? Does that mean our clients must upgrade their graphics card drivers to support VK1.1 or even VK1.2? I hope not, and I have an impression that DXC supports VK1.0
Everything seems to suggest that way. Please note DXC has some extensions for assigning Vulkan slots which are very similar to GLSL's counterpart
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

I managed to track down the regressions! Unit tests are passing. Merging Vulkan branch into master is now back in the menu.
rujialiu
Goblin
Posts: 296
Joined: Mon May 09, 2016 8:21 am
x 35

Re: [2.3] Vulkan Progress

Post by rujialiu »

dark_sylinc wrote: Sun Sep 13, 2020 5:28 pm First a small disclaimer: my primary goal is getting Ogre working very well on Android (not perfect because that's almost impossible due to driver bugs, but as best as possible) and this goal seems to have been achieved (I'm cleaning up the repo right now, preparing docs on how to compile for Android).

Therefore while HLSL compilation is very desirable, it is not in my top priority list right now. I won't be working in HLSL in the foreseeable future.
I understand that... but it looks like I have to give up because integrating DXC seems non-trivial and I don't really have time for it.

Anyway, I spent a bit of time today again.
First, Vulkan+non-fullscreen mode works with the lastest commit ("Fix D3D11 regression in Compute Shaders"), thanks!

Then I've carefully read OgreRootLayout documentation and thinks I understand it now :) According to your suggestion earlier in this thread, I changed

Code: Select all

    static const char c_HLSLBufferTypesMap[] = "ccuttsuu";
to

Code: Select all

    static const char c_HLSLBufferTypesMap[] = "bbuttsuu";
Otherwise I'll get assertion failed at `GpuProgramParameters::_findNamedConstantDefinition()`:

Code: Select all

                OGRE_EXCEPT(Exception::ERR_ITEM_NOT_FOUND,
				"Parameter called " + name + " does not exist. " + knownNames,
                            "GpuProgramParameters::_findNamedConstantDefinition");
Like dark_sylinc said, the ONLY way (that I've ever found) to make it (DownScale03_SumLumEnd_ps.hlsl) work is to declare uv as a "global variable" rather than main()'s parameter. Otherwise I'll get this (however, the app does NOT crash! It's running):

Code: Select all

Ogre: ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-CoreValidation-Shader-InputNotProduced ] Object 0: handle = 0x7dceed0000000068, name = HDR/DownScale03_SumLumEnd_ps_VK, type = VK_OBJECT_TYPE_SHADER_MODULE; | MessageID = 0x23e43bb7 | fragment shader consumes input location 0.0 which is not written by vertex shader
Ogre: Validation error:
...
Also, I've added a `[[vk::location(0)]]` annotation before it (I've even confirmed with spirv-dis that it really affects the generated spirv code by changing 0 to 1), but it looks like it defaults to location = 0 anyway, and adding `:TEXCOORD0` doesn't seem to change anything.

The final working code is:

Code: Select all

static const float2 c_offsets[4] =
{
	float2( -1.0, -1.0 ), float2( 1.0, -1.0 ),
	float2( -1.0,  1.0 ), float2( 1.0,  1.0 )
};

Texture2D<float> lumRt		: register(ogre_t0);
SamplerState samplerBilinear: register(ogre_s0);

Texture2D<float> oldLumRt	: register(ogre_t1);
SamplerState samplerPoint	: register(ogre_s1);

cbuffer Params : register(ogre_P0)
{
    float3 exposure;
    float timeSinceLast;
    float4 tex0Size;
};

[[vk::location(0)]] float2 uv;

float4 main() : SV_Target
{
	float fLumAvg = lumRt.Sample( samplerBilinear, uv + c_offsets[0] * tex0Size.zw ).x;

	for( int i=1; i<4; ++i )
		fLumAvg += lumRt.Sample( samplerBilinear, uv + c_offsets[i] * tex0Size.zw ).x;

	fLumAvg *= 0.25f; // /= 4.0f;

	float newLum = exposure.x / exp( clamp( fLumAvg, exposure.y, exposure.z ) );
	float oldLum = oldLumRt.Sample( samplerPoint, float( 0.0 ).xx ).x;

	//Adapt luminicense based 75% per second.
	return lerp( newLum, oldLum, pow( 0.25f, timeSinceLast ) );
}
Then I tried to use hlsl for Ogre/Compositor/Quad_vs_VK and got 8 copies of this:

Code: Select all

Ogre: PERFORMANCE WARNING: [Validation] Code 0 : Validation Performance Warning: [ UNASSIGNED-CoreValidation-Shader-OutputNotConsumed ] Object 0: handle = 0xcb56470000000042, name = Ogre/Compositor/Quad_vs_VK, type = VK_OBJECT_TYPE_SHADER_MODULE; | MessageID = 0x609a13b | vertex shader writes to output location 0.0 which is not consumed by fragment shader
Then I'm clueless :(
Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 226
Joined: Thu Oct 14, 2010 12:30 pm
x 56

Re: [2.3] Vulkan Progress

Post by Hotshot5000 »

I'm finally back with a bit of a delay since I'm working on proof of concept for an application (not related to graphics or Ogre) so I will have even less time than before. I understand that vulkan has been merged into master? I will look into it and try to get it working again locally. I have an OnePlus 5T android phone so I can probably test the android stuff on it and report issues that I encounter.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

Hotshot5000 wrote: Tue Sep 15, 2020 8:04 pm I understand that vulkan has been merged into master?
Not yet but soon.
Hotshot5000 wrote: Tue Sep 15, 2020 8:04 pm I will look into it and try to get it working again locally. I have an OnePlus 5T android phone so I can probably test the android stuff on it and report issues that I encounter.
I suggest you ditch your branch now, since the "v2-2-vulkan" is robust now; and work from there on.

I compile Ogre (grab v2-2-vulkan branch) and ogre-next-deps (grab master branch) for Android this way:

Code: Select all

mkdir -P build/Android/Debug
cd build/Android/Debug

cmake \
    -DCMAKE_TOOLCHAIN_FILE=/home/matias/Android/Sdk/ndk/21.0.6113669/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_NATIVE_API_LEVEL=24 \
    -DOGRE_BUILD_PLATFORM_ANDROID=1 \
    -DOGRE_DEPENDENCIES_DIR=/home/matias/Projects/SDK/OgreVulkan/AndroidDependencies \
    -DOGRE_SIMD_NEON=OFF \
    -DOGRE_SIMD_SSE2=OFF \
    ../../../
make -j9
Obviously change /home/matias/Android/Sdk/ndk/21.0.6113669/ to wherever your NDK is.
If you want Debug, you also need -DCMAKE_BUILD_TYPE=Debug

AndroidDependencies is a symlink which points to /home/matias/Projects/SDK/ogre-next-deps/build/android-test which is where I built ogre-next-deps with the same cmake command (you can use Release dependencies with an Ogre Debug build)

Once it's built, go to Samples/2.0/AndroidAppTemplate and run python3 GenTemplate.py

This will create lots of Android Studio projects, one for each sample; that link to the libraries stored in build/Android/Debug

I think you may have to modify Samples/2.0/AndroidAppTemplate/Template/local.properties first to point to your SDK's path first though. You can run python3 GenTemplate.py multiple times to broadcast your changes done to the template to all the sample projects

All Android Studio does is link all libraries together (OgreMain+RenderSystem+Hlms+Sample's OgreCommon+Sample) and package the APK. It compiles a few C++ but not many.

Note: Watch out Android Studio likes to copy lib files to temp folders, creating duplicates. Thus compiling ALL samples can take A LOT of disk space (in the several hundred GBs if you build all of them).

Note 2: If debugging Ogre code gives you messed up callstacks or breakpoints aren't getting hit, in Android Studio select "Build->Rebuild Project"

If you find an issue while building you're smart to figure it out; hopefully it should be quite a smooth build experience with only a few rough edges.

Btw. some samples like Forward3D and Decals aren't working right due to broken calculation with orientation support (unless you force-disable OGRE_CONFIG_ENABLE_VIEWPORT_ORIENTATIONMODE)

LocalCubemaps crashes on me but I suspect it's due to out of memory. I think either the cubemaps are too big for my device, or we queue up too many rendering commands (rendering 3 cubemaps means rendering 3*6 render passes, plus PSSM shadows. It can easily reach 72 render passes at once)

Note 3: The validation layers may report false positives in Android because the driver incorrectly reports something as unsupported or reports very low limits.

Note 4: Samples using D32 shadow maps may have very visible shadow map corruption if your Adreno driver is < 512.440; I have yet to add a workaround for that. Just edit the .compositor file to use D24 or D16 instead.
dermont
Bugbear
Posts: 812
Joined: Thu Dec 09, 2004 2:51 am
x 42

Re: [2.3] Vulkan Progress

Post by dermont »

NVidia driver 450.66
Kernel 5.4.0-47-generic
Ubuntu 20.04.1 LTS
GeForce GTX 650 Ti BOOST
Vulkan Render System
Hi,

I understand your priority at the moment is with the Android build. I'm hoping that maybe you could give some pointers on how to resolve these issue on linux desktop.

================================
Sample_PbsMaterials
================================

On initial run Sample_PbsMaterials the shows slight artififacts/tearing. If I run the demo again the artififact/tearing and/or missing geometry is really noticable. Same for both Debug/Release builds.

https://ibb.co/rfXRRDc

Deleting hlmsDiskCache*.bin and textureMetadataCache.json as per initial run.

================================
Sample_Meshv2
================================

Sample_Meshv2, initially runs OK, on subsequent runs similar to Sample_PbsMaterials.

https://ibb.co/XpjfFH0

Deleting hlmsDiskCache*.bin and textureMetadataCache.json as per initial run.

I have tried running:
- with/without vsync and the various vsync options;
- various vulkan sdks;
- patched ogre to build against the version NVidia driver uses 1.2.131 (i.e add flags for VK_ACCESS_ACCELERATION_STRUCTURE_READ_BIT_KHR etc).

On the android build is armeabi-v7a supported? The build fails:

Code: Select all

/media/sdb11/ANDROID/android-sdk/ndk/21.3.6528147/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/vulkan/vulkan_core.h:4927:35: error: typedef redefinition with different types ('uint64_t' (aka 'unsigned long long') vs 'struct VkSwapchainKHR_T *')
VK_DEFINE_NON_DISPATCHABLE_HANDLE(VkSwapchainKHR)
                                  ^
/media/sdb5/Libraries/OGRE/src/v2-2-vulkan-android/RenderSystems/Vulkan/include/OgreVulkanPrerequisites.h:49:34: note: previous definition is here
typedef struct VkSwapchainKHR_T *VkSwapchainKHR;

Code: Select all

            cmake \
            -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake \
            -DANDROID_ABI=${BUILD} \
            -DOGRE_SIMD_NEON=OFF \
            -DOGRE_SIMD_SSE2=OFF \
            -DANDROID_NATIVE_API_LEVEL=29 \
            -DANDROID_PLATFORM=29 \
            -DOGRE_BUILD_PLATFORM_ANDROID=1 \
            -DOGRE_DEPENDENCIES_DIR=${DEPS_DIR}/${BUILD}-sdk \
            -DCMAKE_PREFIX_PATH=${DEPS_DIR}/${BUILD}-sdk \
            -DOGRE_DEPENDENCIES_DIR=${DEPS_DIR}/${BUILD}-sdk ..
            make -j 6 || exit $?
            make install || exit $?
Nice job with the Samples/Templates, really classy.

Thanks.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

Ouch!

This is the first time we got such errors. AMD (both drivers), Intel (both drivers), NVIDIA (rujia's card), and Adreno 505 & 506 have all behaved well so far.

A quick look online shows your GPU exposes 8 identical memory types, which means there must be some strict requirements we may not be conforming to (i.e. textures can only live in X type of memory type, buffers in X type of memory type, etc).
We may be ignoring unusually high strict alignment for buffers (I hope that's the reason, since it's an easy one).

It's not surprising since the GTX 600 series is the first gen of NVIDIA cards to get Vulkan support. The truth is NV cards were at a serious HW disadvantage with Vulkan (they weren't flexible enough).
dermont wrote: Wed Sep 16, 2020 4:22 am On initial run Sample_PbsMaterials the shows slight artififacts/tearing. If I run the demo again the artififact/tearing and/or missing geometry is really noticable. Same for both Debug/Release builds.

https://ibb.co/rfXRRDc

Deleting hlmsDiskCache*.bin and textureMetadataCache.json as per initial run.

================================
Sample_Meshv2
================================

Sample_Meshv2, initially runs OK, on subsequent runs similar to Sample_PbsMaterials.
I suspect a possible race condition, or missing barrier (or NV ignored our barrier). If the vertex buffers aren't loaded correctly, this would happen.

From your PbsMaterials pic, it would seem both the cube model had a bad vertex, and the sphere model had a couple bad vertex. However if you see a sphere or cube that is looking perfect, then it's a driver bug; because it's always the same mesh model.

Do the validation layers say something?

To get the validation layers working on Linux, you must set the following env variables:

Code: Select all

VULKAN_SDK=/home/matias/Projects/vulkan/1.1.121.1/x86_64
PATH=$PATH:/home/matias/Projects/vulkan/1.1.121.1/x86_64/bin
LD_LIBRARY_PATH=/home/matias/Projects/vulkan/1.1.121.1/x86_64/lib
VK_LAYER_PATH=/home/matias/Projects/vulkan/1.1.121.1/x86_64/etc/vulkan/explicit_layer.d
/home/matias/Projects/vulkan/1.1.121.1/ is where I installed the Vulkan SDK. Please upload your Ogre.log after launching it with validation layer.

You can also go to OgreVulkanRenderSystem.cpp and place a breakpoint in dbgFunc:

Code: Select all

} else if (msgFlags & VK_DEBUG_REPORT_ERROR_BIT_EXT) {
            sprintf(message, "ERROR: [%s] Code %d : %s", pLayerPrefix, msgCode, pMsg); //-- breakpoint here --
            static_cast<VulkanRenderSystem*>( pUserData )->debugCallback();
To collect some callstacks to see what's triggering which validation errors.

I just hope it's not this bug:
After deallocation, delay block reinsertion to free blocks for N frames. During allocation, if an allocation cannot be covered but a delayed block can, insert a heavy memory barrier
Basically after the VaoManager deallocates memory, I theorized we should put a few barriers (or wait N frames until it's safe to recycle without barriers) because if the memory gets recycled and we issue writes commands to this memory region, it's not impossible for the write commands to begin before the previous commands (before recycling) have ended.

So far I never saw this happening.
dermont wrote: Wed Sep 16, 2020 4:22 am On the android build is armeabi-v7a supported? The build fails:
I didn't try, but there is no major reason it wouldn't work. Judging from the error, our forward decls in OgreVulkanPrerequisites.h should be wrapped around:

Code: Select all

#if defined(__LP64__) || defined(_WIN64) || (defined(__x86_64__) && !defined(__ILP32__) ) || defined(_M_X64) || defined(__ia64) || defined (_M_IA64) || defined(__aarch64__) || defined(__powerpc64__)
 // typedef everything to struct ptrs
#else
 // typedef everything to uint64_t
#endif
Cheers!
Matias
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

I just noticed we never take bufferImageGranularity into account.

I would make sense that if a texture ends up aliasing to a vertex buffer, then uploading to this texture would partially overwrite the vertex buffers' contents.

The validation layers should detect this. Let's cross fingers this is the problem

Update: I'm not sure validation layers would detect this bug. To test this theory you can go to VulkanVaoManager::allocateTexture and change memReq.alignment to Math::lcm( memReq.alignment, mDevice->mDeviceProperties.limits.bufferImageGranularity ).

It's a bold workaround (will produce more wasted memory) but it would test the theory.
dermont
Bugbear
Posts: 812
Joined: Thu Dec 09, 2004 2:51 am
x 42

Re: [2.3] Vulkan Progress

Post by dermont »

dark_sylinc wrote: Wed Sep 16, 2020 5:20 am I just noticed we never take bufferImageGranularity into account.

I would make sense that if a texture ends up aliasing to a vertex buffer, then uploading to this texture would partially overwrite the vertex buffers' contents.

The validation layers should detect this. Let's cross fingers this is the problem

Update: I'm not sure validation layers would detect this bug. To test this theory you can go to VulkanVaoManager::allocateTexture and change memReq.alignment to Math::lcm( memReq.alignment, mDevice->mDeviceProperties.limits.bufferImageGranularity ).

It's a bold workaround (will produce more wasted memory) but it would test the theory.
Yes, it worked, thank you so much for your assistance on this.

Edit:
The only thing I found reported before applying the above change was in Sample_SceneFormat. I'll include it here in case if it is of any help.

Code: Select all

ERROR: [Validation] Code 0 : Validation Error: [ VUID-vkBindBufferMemory-size-01037 ] Object 0: handle = 0x2e200000002e2, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0x8f6ed3d0 | vkBindBufferMemory(): memory size minus memoryOffset is 0x1830180 but must be at least as large as VkMemoryRequirements::size value 0x1830200, returned from a call to vkGetBufferMemoryRequirements with buffer. The Vulkan spec states: The size member of the VkMemoryRequirements structure returned from a call to vkGetBufferMemoryRequirements with buffer must be less than or equal to the size of memory minus memoryOffset (https://vulkan.lunarg.com/doc/view/1.2.141.0/linux/1.2-extensions/vkspec.html#VUID-vkBindBufferMemory-size-01037)
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

Thanks!

It's a relief it was something so simple to fix. I pushed a proper fix for the problem. Let me know if it fixes your problem too or if it still happens (or something else happens).
dermont
Bugbear
Posts: 812
Joined: Thu Dec 09, 2004 2:51 am
x 42

Re: [2.3] Vulkan Progress

Post by dermont »

Sorry doesn't fix problem, all samples:

Code: Select all

ERROR: [Validation] Code 0 : Validation Error: [ UNASSIGNED-GeneralParameterError-RequiredParameter ] Object 0: handle = 0x561b71d8e610, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x99fb7dfd | vkBindBufferMemory: required parameter buffer specified as VK_NULL_HANDLE
ERROR: [Validation] Code 0 : Validation Error: [ VUID-vkBindBufferMemory-buffer-parameter ] Object 0: handle = 0x561b7192bdf0, type = VK_OBJECT_TYPE_INSTANCE; | MessageID = 0x6ba09117 | Invalid VkBuffer Object 0x0. The Vulkan spec states: buffer must be a valid VkBuffer handle (https://vulkan.lunarg.com/doc/view/1.2.141.0/linux/1.2-extensions/vkspec.html#VUID-vkBindBufferMemory-buffer-parameter)
gdb

Code: Select all

0x00007ffff1cc8bb1 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.450.66
(gdb) bt
#0  0x00007ffff1cc8bb1 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.450.66
#1  0x00007ffff1cca823 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.450.66
#2  0x00007ffff2acca94 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
#3  0x00007fffef394d41 in vulkan_layer_chassis::BindBufferMemory (
    device=0x555555ef9770, buffer=0x0, memory=0x50000000005, memoryOffset=0)
    at /media/sdb11/VULKAN/VULKANSDK/1.2.141.0/source/Vulkan-ValidationLayers/layers/generated/chassis.cpp:1582
#4  0x00007ffff00ccb6c in Ogre::VulkanVaoManager::allocateVbo (
    this=0x555555e26130, sizeBytes=4194304, alignment=4, 
    vboFlag=Ogre::VulkanVaoManager::CPU_WRITE_PERSISTENT_COHERENT, 
    textureMemTypeBits=256, outVboIdx=@0x7fffffffceb0: 0, 
    outBufferOffset=@0x7fffffffcea8: 0)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/RenderSystems/Vulkan/src/Vao/OgreVulkanVaoManager.cpp:789
#5  0x00007ffff00cd5e2 in Ogre::VulkanVaoManager::allocateRawBuffer (
    this=0x555555e26130, 
    vboFlag=Ogre::VulkanVaoManager::CPU_WRITE_PERSISTENT_COHERENT, 
    sizeBytes=4194304, alignment=4)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/RenderSystems/Vulkan/src/Vao/OgreVulkanVaoManager.cpp:902
--Type <RET> for more, q to quit, c to continue without paging--
#6  0x00007ffff00568f4 in Ogre::VulkanDiscardBufferManager::VulkanDiscardBufferManager (this=0x555555839590, device=0x555555e0e7b0, vaoManager=0x555555e26130)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/RenderSystems/Vulkan/src/OgreVulkanDiscardBufferManager.cpp:45
#7  0x00007ffff0063bc9 in Ogre::v1::VulkanHardwareBufferManagerBase::VulkanHardwareBufferManagerBase (this=0x555556095f70, device=0x555555e0e7b0, 
    vaoManager=0x555555e26130)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/RenderSystems/Vulkan/src/OgreVulkanHardwareBufferManager.cpp:42
#8  0x00007ffff0096b7e in Ogre::v1::VulkanHardwareBufferManager::VulkanHardwareBufferManager (this=0x555556098560, device=0x555555e0e7b0, 
    vaoManager=0x555555e26130)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/RenderSystems/Vulkan/include/OgreVulkanHardwareBufferManager.h:67
#9  0x00007ffff008dde4 in Ogre::VulkanRenderSystem::_createRenderWindow (
    this=0x5555557006e0, name="PBS Materials Sample", width=1024, height=768, 
    fullScreen=false, miscParams=0x7fffffffd370)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/RenderSystems/Vulkan/src/OgreVulkanRenderSystem.cpp:883
#10 0x00007ffff76e3dd9 in Ogre::Root::createRenderWindow (this=0x555555ab90b0, 
    name="PBS Materials Sample", width=1024, height=768, fullScreen=false, 
    miscParams=0x7fffffffd370)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/OgreMain/src/OgreRoot.cpp:1379--Type <RET> for more, q to quit, c to continue without paging--

#11 0x00005555555a1dda in Demo::GraphicsSystem::initialize (
    this=0x55555561d070, windowTitle="PBS Materials Sample")
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/Samples/2.0/Common/src/GraphicsSystem.cpp:319
#12 0x00005555555b2bfe in Demo::MainEntryPoints::mainAppSingleThreaded (
    argc=1, argv=0x7fffffffd778)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/Samples/2.0/Common/src/System/Desktop/MainLoopSingleThreaded.cpp:78
#13 0x000055555559b213 in mainApp (argc=1, argv=0x7fffffffd778)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/Samples/2.0/Showcase/PbsMaterials/PbsMaterials.cpp:25
#14 0x000055555559b146 in main (argc=1, argv=0x7fffffffd778)
    at /media/sdb5/Libraries/OGRE/src/v2-2-VULKAN/Samples/2.0/Common/include/MainEntryPointHelper.h:40
(gdb) 


User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5299
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1279
Contact:

Re: [2.3] Vulkan Progress

Post by dark_sylinc »

Ahhh sorry. I messed up.

I didn't test the code because I couldn't when I pushed. I was able to repro that crash on my end too.

It should be fixed now.
Post Reply