[News] Vulkan Progress Report

News, announcements and important global threads from the Ogre3D Team.
Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

I've been reading the Metal docs and MetalRenderSystem about encoders and I see the idea implemented in VulkanQueue. But I noticed something in this article it says:
A common misconception I see is that _READ flags are passed into srcAccessMask, but this is redundant. It does not make sense to make reads available, i.e. you don’t flush caches when you’re done reading data.
But in VulkanQueue::prepareForUpload I see:

Code: Select all

if( buffer )
        {
            BufferPackedDownloadMap::iterator it = mCopyDownloadBuffers.find( buffer );

            if( it == mCopyDownloadBuffers.end() )
                bufferAccessFlags = VulkanMappings::get( buffer->getBufferPackedType() );
            else
            {
                if( !it->second )
                {
                    // bufferAccessFlags = VK_ACCESS_TRANSFER_WRITE_BIT;
                    // We assume consecutive writes means we're writing to non-overlapping areas
                    // Do not wait for previous transfers.
                    bufferAccessFlags = 0;
                }
                else
                    bufferAccessFlags = VK_ACCESS_TRANSFER_READ_BIT;
            }

            mCopyDownloadBuffers[buffer] = false;

            mCopyEndReadSrcBufferFlags |= VK_ACCESS_TRANSFER_WRITE_BIT;
        }
So when there are no matching mCopyDownloadBuffers it does bufferAccessFlags = VulkanMappings::get( buffer->getBufferPackedType() ); which initializes to a _READ flag.

I am not familiar enough yet with the code to know if this is OK, so I wanted to ask you about it.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

Hi!

I don't doubt it could be true that srcAccessMask with _READ flags are unnecessary (I didn't know, thanks). But that only covers cache flushes. We need to be wary of execution dependencies too.

The barriers are:
  • VulkanQueue::prepareForUpload
    • We must wait until previous commands are done reading from/writing to dst before we can proceed with a copy command what writes to dst.
      That includes previous copy commands, but we don't because we assume two consecutive uploads to the same buffer (or textures) are always done to non-overlapping regions.
    • At endCopyEncoder: we must ensure future GPU commands wait for our copy command to end writing to dst before they can start.
  • VulkanQueue::prepareForDownload
    • We must wait until previous commands are done writing to src before we can proceed with a copy command that reads from src. Some buffers/textures are guaranteed to be read-only, in this case we can avoid this barrier. Textures may still need to transition their layout though.
    • At endCopyEncoder: we must ensure future commands don't write to src before we're done reading from src. Again, buffers/textures which are guaranteed to be read-only don't need this. Textures may still need to transition their layout though
Update: Pushed this to VulkanQueue's documentation.

Update 2: The particular case you're mentioning (bufferAccessFlags = VK_ACCESS_TRANSFER_READ_BIT, which ends up in srcAccessMask) is for the case when we just issued a download from the buffer, and now we want to upload to the buffer.
We want the download (read) to finish before the upload (write) can start. Even if the bufferAccessFlags = VK_ACCESS_TRANSFER_READ_BIT is unnecessary, the execution dependency is still necessary.

It is quite possible we'll have to review these unnecessary _READ flags in srcAccessMask in the future, but I wouldn't say it's urgent.

Since some of our logic depends on this e.g. bNeedsBufferBarrier = bufferAccessFlags & VK_ACCESS_TRANSFER_READ_BIT is actually necessary so that bNeedsBufferBarrier becomes true in order to generate an execution barrier; then quite possibly the obvious solution is to perform:

Code: Select all

memBarrier.srcAccessMask = bufferAccessFlags & maskAllReadFlagsAway;

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

Yay! I'm fixing more and more bugs and it feels like it's coming together :)

The D3D11/Metal-style layer should be done now. I'll now focus on DescriptorSetTexture and co.

Root Layouts have changed slightly:

Code: Select all

/*
## ROOT LAYOUT BEGIN
{
	"0" :
	{
		"has_params" : ["vs", "ps"],
		"samplers" : [0, 1],
		"textures" : [0, 1]
	}
}
## ROOT LAYOUT END
*/

layout(ogre_P0) uniform Params
{
	mat4 worldViewProj;
};
layout(OGRE_POSITION) in vec3 position;
layout(OGRE_TEXCOORD0) in vec2 inUv0;
layout(location = 0) out vec2 uv0;
has_params is no longer a boolean, but rather must specify which stages will use ogre_P0 (i.e. parameters updated via the old Material interface). One can specify each stage or use "all" (not recommended) for all stages (including geometry, hull and domain shaders; or compute in case of compute)

I'm thinking of probably defining conservatively small default root layouts for low level material to avoid having to deal with this (unless e.g. you use too many textures).

Another issue I detected is that we have calls to vkDestroySampler/vkDestroyImage/vkDestroy* which are unsafe. According to spec any command submitted to the GPU using that handle must have completed by the time vkDestroy* is called.

We need to schedule/queue the destruction of those handles for later (i.e. N frames later). Fortunately this is easy to do.
Note that we don't have to delay the memory management.

For example when we do:

Code: Select all

void VulkanTextureGpu::destroyInternalResourcesImpl( void )
{
  vkDestroyImage( device->mDevice, mFinalTextureName, 0 ); // Needs to be delayed
  vaoManager->deallocateTexture( mTexMemIdx, mVboPoolIdx, mInternalBufferStart,
                                                 memRequirements.size ); // Does not have to be delayed
}
The reason for this is that even if the memory range released by deallocateTexture gets immediately recycled into another buffer/texture; we won't immediately change its content (not even its layout). We will schedule commands to operate on it.
Thus by the time those commands get executed, the GPU is already finished working on that memory range when it was originally a texture.

The only exception are dynamic buffers (BT_DYNAMIC_*), staging buffers and staging textures; as they can immediately be overwritten from CPU after being recycled while the GPU could still be using that memory region.
However these 3 are already delayed (by VaoManager and TextureGpuManager) so we don't have to worry about those.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

I'm afraid I beat you to the endEncoder call stuff.
I was actively avoiding it... but today it got in my way (it was causing crashes, so I fixed it)

It lives!



Now I have to fix a deadlock caused by VulkanQueue resetting fences too early

Update: Deadlock fixed.

Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

Don't worry I didn't actually start implementing anything in VulkanQueue :). Mostly just read documentation and specs.
Also I have some problems with my laptop: buzzing sound coming from right speaker when under load. Investigated and the solution is to change the motherboard. It's a design defect that Acer doesn't want to admit so they don't change our laptops (there is a long thread about this on their community forums). I have an Acer Predator 17 G9-793 that is 3 years old now (out of warranty) and the reparation cost is around $1000. At which point I might simply start thinking about buying a new laptop and throw this one in the dump where it belongs. Or I could simply remove the speaker connector from the inside but then I don't have any sound except if I use headphones.

What I am trying to say is: Acer is garbage don't buy their products :). I paid around $3000 for this POS and in the last 3 years I only had headaches because of it.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

Ouch! That sucks.

There is one thing I don't understand: is the buzzing sound an auditive annoyance or does it also block the laptop from working / going to get worse?
Can the problem go away by silencing the speaker or unplugging it?
I paid around $3000 for this POS
Whoa! That's a lot. You reminded me of this recent thread about a 1099€ Ryzen laptop.

Ryzen laptops are taken the laptop market by surprise. They're very powerful. Their iGPU are also surprisingly decent (it can run all Ogre demos at > 60 fps) but obviously won't compare to a dGPU.
If you're sticking to iGPUs, prefer the Vega 11 over Vega 8.

Regarding Vulkan. I've been thinking on what you could do next. It's been hard for me because we're figuring some stuff along the way and there would be a lot of overlap with little isolation.

But I will list a few things that don't seem to overlap with my efforts:
  • glslang supports compiling HLSL to SPIRV but I didn't look into it. I've been thinking HLSL may actually be more useful than GLSL for postprocessing shaders and other low level materials because unlike our OpenGL GLSL shaders, HLSL separates textures and samplers and thus they could be reused more easily.
  • Alternatively you could look into how to port GLSL postprocessing shaders under Samples/Media/2.0/scripts/materials/Common (and a few more outside the Common folder).
    For example Copyback_4xFP32_ps.glsl should look like this to support both GL and Vulkan at the same time:

    Code: Select all

    #version ogre_glsl_version( 450, 330 )
    
    vulkan_layout( ogre_t0 ) uniform texture2D tex;
    #if VULKAN
    	vulkan_layout( ogre_s0 ) uniform sampler mySampler;
    #endif
    
    vulkan_layout( location = 0 ) in block
    {
    	vec2 uv0;
    } inPs;
    
    out vec4 fragColour;
    
    void main()
    {
    	fragColour = texture( ogre_sampler2D( tex, mySampler ), inPs.uv0 );
    }
    • ogre_glsl_version( vkVersion, glslVersion ) will return the former or the latter depending on which RenderSystem we're using (I didn't implement this yet)
    • vulkan_layout translates to layout in Vulkan, translates to nothing in GL
    • ogre_sampler2D(a,b) translates to sampler2D(a,b) in Vulkan, to (a) in GL
    I'm thinking these differences are too verbose (too much noise preventing us to look at what the code is doing), which is why HLSL may be preferrable. HLSL would probably need some macros to strip away the Vulkan stuff when compiled in D3D11, but I'm guessing it would be less noisy.

    Alternatively (opt 2) we could port the shaders to use the texture sampling macros from CrossPlatformSettings_piece_all.glsl so that we use OGRE_Sample instead of texture()

    Alternatively (opt 3) we could add support for combined sampler/texture to our root layouts. This could significantly reduce the noise in GLSL postprocessing shaders to manageable levels:

    Code: Select all

    #version ogre_glsl_version( 450, 330 )
    
    vulkan_layout( ogre_t0 ) uniform texture2D tex;
    
    vulkan_layout( location = 0 ) in block
    {
    	vec2 uv0;
    } inPs;
    
    out vec4 fragColour;
    
    void main()
    {
    	fragColour = texture( tex, inPs.uv0 );
    }
  • Look into SPIRV-Tools optimizer (these tools come with the SDK, they should be in your SDK). SPIRV tools support optimization, even though it's not guaranteed to actually make any difference (drivers optimize SPIRV too). However I suspect we'll need it for 3 reasons:
    1. Mobile drivers may need it. I suspect mobile drivers don't optimize as aggressively as Desktop does.
    2. 'Inline all function calls exhaustively' is very interesting. I've heard of mobile driver bugs (particularly early Android 7.x) where sampler arguments passed into functions don't work. Hence aggressive inlining would automatically fix this. Not sure how much this is needed because ultimately Roblox got so tired of Vulkan Android bugs in Android 7.x they just blacklisted 7.x from using Vulkan.
    3. 'Stripping debug info': Again, certain Android driver would break if the debug info names between vertex and pixel shaders didn't match, even though it is by explicit by spec that debug info should not change the behavior of the program. Sigh.

Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

The buzzing starts at a low volume but it grows louder and louder. If I stop doing intensive stuff then it goes away eventually. I can live with it if I am wearing headphones but I don't like wearing headphones for the whole day. The forums say that it goes louder and louder with months of use so I can expect that at some point not even headphones will work (some user said he could hear it from the next floor on his house with the doors closed). It seems to be an insulation problem and I hear the ground hum in the speaker. It doesn't matter if I am switching to headphones and the sound channel goes to headphones because the interference comes from the motherboard directly. The suggested solution is to remove the speaker cords but then you don't get any sound in the laptop and it's not really a solution as the fact that the noise gets worse over time suggests that there is something going worse with the motherboard.

The laptop seems to be working fine apart from that. And apart from the other issues with it that I've grown accustomed to (Not resuming from suspend; when the screen turns off it doesn't go back to on and the system gets unresponsive about once a month and the solution is to remove the battery cover since not even on off button works anymore; the touchpad buttons rattle when I listen to music and it hits certain frequencies; the space bar makes a squeaking noise; Screen flickering on certain applications as if I'm running WhatsApp at 20 fps etc. etc). Sigh... I get angry only remembering the list of issues and thinking that this was supposed to be a high end powerful stable machine.

But you are right. I might get a ryzen laptop which should be good enough for working on Ogre and my game (a mobile game) and keep the buzzing Acer Predator 4,5 Kg beast only for gaming. The loud fans already forced me to use headphones in games so why does it matter that now it also buzzes from the right speaker?

I will find a workaround or buy a new laptop and throw this one into fire :)

Anyway back to Ogre. I will merge the latest commits from 2-2-vulkan into my fork and test it out. Then I'll check HLSL out (I have not worked with DX since DX 8 days so I am not familiar with HLSL) and if it looks cleaner then I have no problem switching to it from GLSL. At the first glance HLSL looked better it was just that I thought that since GLSL and Vulkan are made by the same guys I was hoping that GLSL was kind of first party support so less chance of issues. But with Vulkan 1.2 they say that HLSL is considered at the same level of support as GLSL so I think/hope it will be fine.
Last edited by Hotshot5000 on Sun Aug 02, 2020 4:51 pm, edited 1 time in total.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

OOUCHH!!! That sounds like a nightmare.

From what you're describing it sounds like the motherboard has improper insulation which gets worse as the capacitors age. It may be possible to fix it by replacing all the caps which is inexpensive (but dangerous. Improperly desoldering a single capacitor will brick the motherboard).

This is just a guess. My guess comes from the spacebar making a squeaking noise and the touchpad with certain music; both sound like coil whine (example 1, example 2, example 3) which in a notebook is either caused by capacitors or inductors.

You may want to try that after you get a new laptop (if that's the problem it DEFINITELY doesn't cost $1000 to repair. That's more like $5-20 at best in materials*) and give the repaired one to a relative.

*If they charge you $1000 it may be because they're trying to cover from the risk of bricking the motherboard (unless they don't guarantee that; in which case they're scamming you). Find somebody with decent soldering skills who knows that you don't care if the laptop gets broken by accident.

Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

The squeaking noise on space bar only happens when I press it on the left side (which is where I normally press with my left thumb) and the touchpad buttons just resonate with the music if certain frequencies are hit. I mean if I touch the button I feel it vibrating and I can thus stop the noise. I don't think it's coil whine in this case. Of course the laptop also has coil whine but I got used to that a long time ago and it's from other component/s :)

I don't think that they are scamming me as the guys from Acer have offered this fix to replace the motherboard to everybody on the forums (not the guys from the repair shop) and a new motherboard costs around $700-800.

UPDATE: So is it safe to get rid of my mShaderProfile == "glsl-vulkan" with specific vulkan glsl shader folders and just use the normal one from GLSL (which have been presumably adapted with the new #if VULKAN stuff) ?

Answer: It looks like so.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

Hotshot5000 wrote:
Sun Aug 02, 2020 4:55 pm
UPDATE: So is it safe to get rid of my mShaderProfile == "glsl-vulkan" with specific vulkan glsl shader folders and just use the normal one from GLSL (which have been presumably adapted with the new #if VULKAN stuff) ?

Answer: It looks like so.
Yes-ish.

A couple remarks:
  • We use "glslvk" to specify glsl that should only be built with Vulkan (it's the same as glsl-vulkan... but shorter). Thus Hlms uses @property( syntax == glslvk ) to identify Vulkan-specific code
  • We will support for the user to optionally allow "glsl" to be compiled with Vulkan too (to avoid code duplication. It can be turned off because you get lots of errors in the log if your shaders weren't ported to support the Vulkan path)
  • Shaders have #define VULKAN so they can differentiate when being compiling under vulkan
  • I already explained vulkan_layout macro
  • ogre_t0 and co. I already explained them earlier. They come from the root layout (which macros are defined depends on which slots the root layout specifies that will be used). The nomenclature is the same as HLSL (e.g. t0 for textures, s0 for samplers, B0 for const buffers) but additionally we use UPPERCASE for buffers and lowercase for textures; and P0 is always for the parameters buffer (i.e. the UBO where we place the params from Low level materials)
  • You can see the exposed vertex semantics like vulkan_layout( OGRE_POSITION ) in OgreVulkanProgram.cpp (look for SemanticMacro)
Thus in short:
  • Users can optionally treat glsl the same as glslvk. Turned off for now (see HighLevelGpuProgramManager::getSingleton().addFactory( mVulkanProgramFactory1 ); commented out)
  • With a couple macros most GLSL shader source file can be adopted to be used by both GL and Vulkan

Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

So I merged you changes and just as I started compiling well what do you know? The damn buzzing has started again AAAARRGHHH! I want to smash this laptop so bad... But I don't have a backup right now. Actually now that I am thinking I do have another laptop: Toshiba Qosmio from 2013 with Intel 4700mq 24 GB ram and nvidia 770m. But that also has some problems (keyboard's upper rows don't work) and the DGPU only supports Vulkan 1.1 so if we want to try 1.2 stuff I'm out of luck.

Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

Hmm something is not working right on my end. So I see the system is loading the 'Popular' resources and calls HlmsPbs::loadLtcMatrix(void) that creates 3 textures but the createInternalResourcesImpl() is never called and it goes directly to void VulkanQueue::prepareForUpload() which calls VkImageMemoryBarrier VulkanTextureGpu::getImageMemoryBarrier( void ) which returns the struct with imageMemBarrier.image == NULL. This gives a Vulkan error in vkCmdPipelineBarrier() and then a crash in void VulkanTextureGpu::notifyDataIsReady( void ) as assert( mFinalTextureName || mPixelFormat == PFG_NULL ); fails as mFinalTextureName is still NULL.

EDIT: It looks like for these 3 textures it has automaticBatching enabled and goes through

Code: Select all

//Ask the manager for the internal resource.
mTextureManager->_reserveSlotForTexture( this );
Also when you have time please merge my changes in VulkanUtils::findDepthFormat and maybe also the VulkanWin32Window class. It seems that I have an issue at line 306 with the message logging. It just doesn't compile anymore on my side even though I didn't change anything there and it worked just fine the last week when I tested this.

Now I get:

Code: Select all

ogre\ogre-next\ogremain\include\OgreLog.h(224): error C2678: binary '<<': no operator found which takes a left-hand operand of type 'Ogre::Log::Stream::BaseStream' (or there is no acceptable conversion)

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

I commented the loadLtcMatrix call in my machine because it was crashing and I forgot to test it with all the latest changes.


Hotshot5000 wrote:
Sun Aug 02, 2020 6:37 pm
EDIT: It looks like for these 3 textures it has automaticBatching enabled and goes through

Code: Select all

//Ask the manager for the internal resource.
mTextureManager->_reserveSlotForTexture( this );
Just in case you don't know, AutomaticBatching textures (i.e. the majority of regular textures) are textures which don't really own their own API handle.

They are also not 2D textures. They just request a single array slice from a "master pool" texture, which is the actual owner of all the API resources.

Externally the TextureGpu pretends to be and act like a 2D texture (e.g. even copy operations pretend there is only one slice).

Internally it's a 2D array texture. For AutomaticBatching textures the following variables become relevant:
  1. mInternalSliceStart: The slice in the 2D array where this texture is located
  2. mTexturePool: the pool owner of our API handles
  3. getInternalTextureType(): returns Type2DArray when AutomaticBatching is set
Hotshot5000 wrote:
Sun Aug 02, 2020 6:37 pm
Also when you have time please merge my changes in VulkanUtils::findDepthFormat and maybe also the VulkanWin32Window class. It seems that I have an issue at line 306 with the message logging. It just doesn't compile anymore on my side even though I didn't change anything there and it worked just fine the last week when I tested this.

Now I get:

Code: Select all

ogre\ogre-next\ogremain\include\OgreLog.h(224): error C2678: binary '<<': no operator found which takes a left-hand operand of type 'Ogre::Log::Stream::BaseStream' (or there is no acceptable conversion)
It seems to be a side effect of merging with master.

In master we improved build times by getting rid of many header dependencies; and #include <sstream> (a big fat header) was being included with LogManager as we exposed the std::basic_stringstream class directly.

So we workarounded with a helper class, which probably for some reason cannot handle your case. Or maybe you just need to add #include <sstream>

Hotshot5000
OGRE Contributor
OGRE Contributor
Posts: 197
Joined: Thu Oct 14, 2010 12:30 pm
x 45

Re: [News] Vulkan Progress Report

Post by Hotshot5000 »

Yep <sstream> did it. Now it compiles with logging enabled. Thanks for the explanation about AutomaticBatching as I am not familiar at all with this (it's probably added more recently and related to streaming). I will look over it to gain some understanding of this and the texture streaming stuff. What I used in my game is on the 2.1 Ogre branch so it's 1-2 years old and I haven't kept up to date with the 2.2 branch.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4438
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 905
Contact:

Re: [News] Vulkan Progress Report

Post by dark_sylinc »

I'm continuing the thread in another post.

This post has very little visibility.

Post Reply