spir-v shaders

[bishopnator]

Hi, I am considering for my project to implement only one set of shaders - like HLSL and generate GLSL shaders on the fly using spir-v compiler. There is compiler glslc (https://github.com/google/shaderc) which translates GLSL or HLSL to spir-v form and then there is spirv-cross (https://github.com/KhronosGroup/SPIRV-Cross) which is able to translate spir-v representation to GLSL or HLSL. Does anybody play with it? Should it be feasible?

I am considering such representation of shaders to simplify the coding and maintaining only one form (like HLSL). In general it should be doable as there is e.g. Diligent Engine (https://github.com/DiligentGraphics/DiligentEngine) which claims that it uses only HLSL shaders for all back-ends (dx11, dx12, gl, vulkan). However the Diligent Engine uses its own translator from HLSL to GLSL (which is also open source).

[dark_sylinc]

Hi!

Two things about this:

1. The GLSL we use in Vulkan relies on macros set from OgreNext to setup the proper bindings based on the RootLayout.
e.g. OgreNext creates:

Code: Select all

#define ogre_t0 set = 0, binding = 5

So that "layout( ogre_t0 )" translates to valid GLSL code, where "t0" corresponds to texture unit 0 (i.e. coming from HLSL t0 will sound very familiar to you).
However whether t0 maps to binding 5 or binding "whatever" depends on the RootLayout. The scheme is explained here. You can find more docs about RootLayouts here. I also wrote further doxygen documentation in a branch that hasn't been merged into master yet here.
It's not hard at all to understand, but what I'm trying to say is that the GLSL generated by Spirv-Cross can't be fed directly to our Vulkan RenderSystem. You'd have to patch up the bindings (perhaps Spirv-Cross already supports this?) so that they say ogre_t0 for textures, ogre_T0 for texture buffers, ogre_b0 for const buffers, etc. But SPIRV-Cross may "just work" for the OpenGL backend.

2. The DXC(*) compiler supports generating SPIR-V from HLSL. This would be a full native solution(**). You'd have to implement a backend for it though. I don't know if extending VulkanProgram or creating a clone of it (e.g. called VulkanProgramDXC / VulkanProgramHLSL) would be best. I suspect the latter would be best. Please don't be confused by the fact that you'll find references to HLSL in VulkanProgram. This is because glslang supports compiling HLSL into SPIR-V but it was quite unstable and subpar. The approach was ultimately abandoned (even glslang devs recommend to use DXC instead).
I never bothered with DXC because the need for extensions meant mobile was out and me and my client was deeply interested in mobile support.

(*) DXC is the new compiler from Microsoft to compile HLSL for DX12 (and later Vulkan). DX11 uses the older FXC compiler.

(**) It requires to enable some Vulkan extensions during initialization that are present in almost all desktop setups (if it's not, it's just one driver update away), but not on mobile.

[bishopnator]

I am still studying the code and it is almost impossible to inject shaders with their own extension due to implementation of Ogre::Hlms.

Code: Select all

            // Prefer glslvk over hlslvk over glsl, and glsl over glsles
            const String shaderProfiles[6] = { "hlsl", "glsles", "glsl", "hlslvk", "glslvk", "metal" };
            const RenderSystemCapabilities *capabilities = mRenderSystem->getCapabilities();

        for( size_t i = 0; i < 6; ++i )
        {
            if( capabilities->isShaderProfileSupported( shaderProfiles[i] ) )
            {
                mShaderProfile = shaderProfiles[i];
                mShaderSyntax = shaderProfiles[i];
            }
        }

Here actually it is hard-coded the order of the shaders. I would expect that if I give Hlms a dataFolder which contains VertexShader_vs.xxx and PixelShader_ps.xxx then the 'xxx' is used as mShaderProfile. If the render system doesn't support 'xxx', then exception should be thrown. Why is there hard-coded order of the preference for the shader profiles?

For example if I manage to implement a good conversion of HLSL To GLSL throuh spir-v, I would like to keep the extension of the shaders simply 'hlsl' and allow to load HLSL shaders by Ogre::Hlms with GL3Plus RS. There are however also quite a lot of checks which syntax the shaders use - in case of GLSL there are some extra code snippets which I am not sure at the moment whether will be needed when shaders are generated from HLSL. In this case the mShaderProfile will be HLSL, but maybe it would be needed to mimic it as GLSL - like mSourceShaderProfile, mTargetShaderProfile. I am aware of the member variables mShaderProfile, mShaderSyntax and mShaderFileExt, but in the current implementation they are redundant as _changeRenderSystem sets then somehow to the same values with some minor expections as hlslvk and glslvk, but it doesn't resolve the described problem.

The mShaderFileExt is not possible to set it independently from nShaderProfile. It is possible to overwrite _changeRenderSystem, but it doesn't sound good. Better would be to have auto-detection of the files from dataFolder provided in the constructor which files does it contain:

Code: Select all

//////////////////////////////////////////////////////////////////////////
bool HlmsExt::AutodetectShaderProfile()
{
	mShaderProfile = "unset!";
	mShaderFileExt = "unset!";
	mShaderSyntax = "unset!";

if (mRenderSystem == nullptr)
{
	LogManager::getSingleton().logMessage(LML_NORMAL, "Shader profile auto-detection must be called only with valid RenderSystem.");
	return nullptr;
}

// Get the file(s) for the vertex shaders - it is sufficient to check only one.
const auto pFiles = mDataFolder->find("*_vs.*", false, false);
if (pFiles == nullptr || pFiles->empty())
{
	LogManager::getSingleton().logMessage(LML_NORMAL, "HLSM data folder doesn't contain *_vs.* file(s).");
	return false;
}

// Simple helper functor to extract file's extension.
const auto getFileExtension = [](const String& file) {
	const auto ix = file.rfind('.');
	return ix != String::npos ? file.substr(ix + 1) : String();
};

String shaderProfile;
for (const auto& file : *pFiles)
{
	// Get the extension.
	const auto ext = getFileExtension(file);
	
	// Check whether the shader profile is support by RenderSystem.
	if (mRenderSystem->getCapabilities()->isShaderProfileSupported(ext))
	{
		// Shader profile is support by the RenderSystem.
		shaderProfile = ext;
		break;
	}
}

if (shaderProfile.empty())
{
	LogManager::getSingleton().logMessage(LML_NORMAL, "HLSM data folder doesn't contain shaders supported by current RenderSystem.");
	return false;
}

mShaderProfile = shaderProfile;
mShaderSyntax = shaderProfile;
mShaderFileExt = "." + shaderProfile;
return true;
}

Is there another way how to inject shaders with custom extension/handling? What do you think?

Actually I would like to try to write a plugin for Ogre which adds support for HLSL shaders to GL3Plus RS. At this moment I am not quite sure about all the details, but the idea as that after RS is initialized, the plugin is loaded and it checks whether it has support for HLSL. If not (like the case of GL3Plus), it updates the RS capabilities (injects HLSL profile there), add new factory for creating the HLSL programs which will convert internally the shaders to GLSL. The HLMS then will provide only folders with HLSL shaders. Current HLMS implementation however selects the "best" shader profile according to RS capabilities and there is no way to use HLSL shaders with GL3Plus (not saying even about using another extension for the files as mentioned at the beginning of the post).

[dark_sylinc]

Hi!

I fail to see what's stopping you / preventing you from continuing

• _changeRenderSystem is virtual. So you can override this behavior.

• mShaderSyntax is what we tell the Hlms parser. e.g. @property( syntax == hlsl ). (If you're using Compute jobs it also affects its behavior see entries for getShaderSyntax() in the source code).

• mShaderProfile is what we tell the RenderSystem to compile as (e.g. hlsl, hlslvk, glsl, glslvk, metal etc).

• mShaderFileExt is the filename extension we look for (i.e. right now it's *.hlsl, *.glsl and *.metal).

• mShaderTargets[] is used by D3D11 / HLSL to target specific versions (e.g. vs_4_0 vs vs_5_0).

• Additionally HLMS has hardcoded to look for any file with *.any extension for parsing.

So if you set mShaderFileExt to glsl we will look for *.glsl files, and if you set mShaderProfile to hlsl, we will try to compile the resulting shader as HLSL despite ending in *.glsl.

What did I miss? What are you trying exactly to do?

Better would be to have auto-detection of the files from dataFolder provided in the constructor which files does it contain

It is perfectly possible to put both *.hlsl and *.glsl files in the same folder, e.g. PixelShader_ps.glsl & PixelShader_ps.hlsl. Which would make such auto detection ambiguous.

PBS and Unlit are complex enough that we're keeping them in separate folders for better organization. But for simple implementations it just makes more sense to keep them together.

In this case the mShaderProfile will be HLSL, but maybe it would be needed to mimic it as GLSL - like mSourceShaderProfile, mTargetShaderProfile

I have a much bigger question: where are you placing the Spir-V step? In the Hlms? In the RenderSystem?
When are you converting the code from HLSL to GLSL?

I feel like it would make the most sense to put the Spir-V step in the RenderSystem as its own profile, e.g. "hlsl-spirv". I don't know if it's possible to do this without modifying OgreNext.

However it may be just possible to do it using HighLevelGpuProgramManager::getSingleton().addFactory( mSpirVProgramFactory ); where mSpirVProgramFactory is your custom implementation, and you call RenderSystemCapabilities::addShaderProfile to add your profile as supported (AFAIK you'd have to const cast RenderSystemCapabilities).

[bishopnator]

Yes, I would like to implement it without modifying ogre-next (including Ogre::Hlms and Ogre::RenderSystem). The ideal step to convert the shaders would be in Ogre::Hlms::compileShaderCode and Ogre::Hlms::_compileShaderFromPreprocessedSource where I got HLSL shader as input, convert it to GLSL and then compile the standard way by GL3Plus RS.

Otherwise I am looking also in the injection of this factory which you mentioned and adding the new capability for the shader profile recognition in the RS.

The Ogre::Hlms::_changeRenderSystem also creates new default block - I didn't look there whether it also executes parsing of the shader templates and compile then - if yes, overriding of _changeRenderSystem would be problematic - I have to copy whole implementation from ogre-next and in my custom overriden method I cannot call the parent method - I have to verify this step (probably you will reply me directly how it is).

[bishopnator]

It seems that it is necessary a lot of customization. The idea with custom HighLevelGpuProgramFactory won't work properly as the result of Hlms::compileShaderCode is assigned to ShaderCodeCache and further assigned to HlmsPso. The RenderSystem is notified then about created HlmsPso and in GL3PlusRenderSystem::_hlmsPipelineStateObjectCreated the shaders are directly casted to GLSLShader which is final class and not possible to inherit further.

With that in mind, it seems the best to modify ogre-next and make Hlms::compileShaderCode virtual so I can convert the input source from HLSL to GLSL and then call parent implementation. I just wanted to avoid direct modification of ogre-next source code due to updates in the future to newer versions And I am just at the beginning of my project and when I need to modify source ogre already at this stage, I am afraid that the neccessity of modifications will just grow and grow during my implementation phase.

Have you any ideas how to inject such shader conversion without modifying ogre-next source code?

[bishopnator]

I forgot to react to one of your statement:

It is perfectly possible to put both *.hlsl and *.glsl files in the same folder, e.g. PixelShader_ps.glsl & PixelShader_ps.hlsl. Which would make such auto detection ambiguous.

PBS and Unlit are complex enough that we're keeping them in separate folders for better organization. But for simple implementations it just makes more sense to keep them together.

If you check my code snippet, it will work also with a data folder containing both glsl and hlsl shaders - I iterate through all the files _vs. and check whether the shader type is support by the render system. The D3D11 will select HLSL and GL3Plus will select GLSL. If a render system supports both HLSL and GLSL, it will select the shader profile according to which file comes first from the archive. Current implementation of Hlms will select GLSL as it is placed later in the hard-code order of shader profiles.

[bishopnator]

I managed to get running my CustomHlms with HLSL shaders with GL3Plus RS:

I only needed to update OgreHmls.h from ogre-next (added virtual keyword):

Code: Select all

        virtual HighLevelGpuProgramPtr compileShaderCode( const String &source,
                                                  const String &debugFilenameOutput, uint32 finalHash,
                                                  ShaderType shaderType ); // ADDED virtual KEYWORD TO THE METHOD

Here is the conversion function to convert HLSL to GLSL:

Code: Select all

// shaderc
#include <shaderc/shaderc.hpp>

// spirv-cross
#include <spirv_cross/spirv_glsl.hpp>

//////////////////////////////////////////////////////////////////////////
Ogre::String Ogre::ConvertHlsl2Glsl(const String& hlsl, const String& debugFilenameOutput, ShaderType shaderType)
{
	shaderc::Compiler compiler;
	if (!compiler.IsValid())
		return {};

shaderc::CompileOptions compilerOptions;
compilerOptions.SetSourceLanguage(shaderc_source_language_hlsl);
compilerOptions.SetTargetEnvironment(shaderc_target_env_opengl, shaderc_env_version_opengl_4_5);
compilerOptions.SetIncluder(std::make_unique<HlslIncluder>());

// Compile HLSL to SPIR-V (function has name GlslToSpv but it supports also HLSL as input through the compilerOptions).
const auto result = compiler.CompileGlslToSpv(hlsl, cShaderTypeToShaderKind[shaderType], debugFilenameOutput.c_str(), "main", compilerOptions);
if (result.GetCompilationStatus() != shaderc_compilation_status_success)
	return {};

// Get the SPIR-V code.
std::vector<uint32_t> spirv(result.begin(), result.end());

// Convert SPIR-V to GLSL.
spirv_cross::CompilerGLSL compilerGLSL(std::move(spirv));

// Remap drawId to location 15 as it is set by GLSLProgram::bindFixedAttributes
if (shaderType == VertexShader)
{
	spirv_cross::ShaderResources resources = compilerGLSL.get_shader_resources();
	for (auto& resource : resources.stage_inputs)
	{
		if (resource.name == "input.drawId")
			compilerGLSL.set_decoration(resource.id, spv::DecorationLocation, 15);
	}
}

compilerGLSL.add_header_line("layout(std140) uniform;\n"); // only to avoid assertions in GLSLProgram::extractLayoutQualifiers()
auto glsl = compilerGLSL.compile();
return glsl;
}

(both shaderc and spriv-cross are part of vcpkg so I took them from there)

The spirv-cross has also a nice reflection so it is possible to overwrite the location of drawId to 15 as it is set by GLSLProgram.

But as proof of concept I think it is good - I think it is necessary to always keep an eye on the converted shaders to ensure that in given context the conversion is correct.

Note that the 'add_header_line" is only to avoid assertions in GLSLProgram::extractLayoutQualifiers() for which I complained in another thread.

[bishopnator]

Just for anyone who has interest in the topic, here is my current implementation which resolves also the binding locations according to the GLSLProgram::bindFixedAttributes and also with the implementation of includer for the conversion of HLSL->SpirV.

Code: Select all

#include "OgreHlsl2Glsl.h"

// OGRE
#include <OgreDataStream.h>
#include <OgreResourceGroupManager.h>
#include <vao/OgreVertexElements.h>

// shaderc
#include <shaderc/shaderc.hpp>

// spirv-cross
#include <spirv_cross/spirv_glsl.hpp>

// std
#include <array>

namespace
{
	//////////////////////////////////////////////////////////////////////////
	class HlslIncluder : public shaderc::CompileOptions::IncluderInterface
	{
	public:
		struct IncludeResult
		{
			std::string m_FileName;
			std::string m_FileContent;
			shaderc_include_result m_Data;
		};

public:
	HlslIncluder() = default;
	~HlslIncluder() override = default;

	shaderc_include_result* GetInclude(const char* requested_source, shaderc_include_type /*type*/, const char* /*requesting_source*/, size_t /*include_depth*/) override
	{
		auto found = m_Includes.find(requested_source);
		if (found == m_Includes.end())
		{
			// Load the include file.
			Ogre::DataStreamPtr pStream = Ogre::ResourceGroupManager::getSingleton().openResource(Ogre::String(requested_source));

			auto pIncludeResult = std::make_unique<IncludeResult>();
			pIncludeResult->m_FileName = requested_source;
			pIncludeResult->m_FileContent = pStream != nullptr ? pStream->getAsString() : Ogre::String();
			if (pIncludeResult->m_FileContent.empty())
			{
				// error
				pIncludeResult->m_FileName = Ogre::String();
				pIncludeResult->m_FileContent = Ogre::String("Cannot load the file ") + requested_source;
			}

			pIncludeResult->m_Data.source_name = pIncludeResult->m_FileName.c_str();
			pIncludeResult->m_Data.source_name_length = pIncludeResult->m_FileName.size();

			pIncludeResult->m_Data.content = pIncludeResult->m_FileContent.c_str();
			pIncludeResult->m_Data.content_length = pIncludeResult->m_FileContent.size();

			pIncludeResult->m_Data.user_data = nullptr;

			found = m_Includes.try_emplace(pIncludeResult->m_FileName, std::move(pIncludeResult)).first;
		}

		return &found->second->m_Data;
	}

	void ReleaseInclude(shaderc_include_result* data) override
	{
		m_Includes.erase(data->source_name);
	}

private:
	std::unordered_map<Ogre::String, std::unique_ptr<IncludeResult>> m_Includes;
};

//////////////////////////////////////////////////////////////////////////
/// Remap ShaderType to shaderc_shader_kind
static_assert(Ogre::NumShaderTypes == 5, "ShaderType enum has been changed! Update the mapping.");
constexpr std::array<shaderc_shader_kind, Ogre::NumShaderTypes> cShaderTypeToShaderKind = {
	shaderc_vertex_shader,         // Ogre::VertexShader
	shaderc_fragment_shader,       // Ogre::PixelShader
	shaderc_geometry_shader,       // Ogre::GeometryShader
	shaderc_tess_control_shader,   // Ogre::HullShader
	shaderc_tess_evaluation_shader // Ogre::DomainShader
};

//////////////////////////////////////////////////////////////////////////
/// Remap VertexElementSemantic to the binding locations.
/// note: The mapped values are copied from GL3PlusVaoManager::getAttributeIndexFor to avoid linking against RenderSystem_GL3Plus module.
const std::unordered_map<std::string, uint32_t> cVertexElementSementicToLocation = {
	{"DRAWID", 15}, // DRAWID
	{"POSITION", 0}, // VES_POSITION
	{"BLENDWEIGHT", 3}, // VES_BLEND_WEIGHTS
	{"BLENDINDICES", 4}, // VES_BLEND_INDICES
	{"NORMAL", 1}, // VES_NORMAL
	{"COLOR", 5}, // VES_DIFFUSE
	{"COLOR0", 5}, // VES_DIFFUSE (just an alias in HLSL if 2 colors are used)
	{"COLOR1", 6}, // VES_SPECULAR
	{"TEXCOORD0", 7}, // VES_TEXTURE_COORDINATES + 0
	{"TEXCOORD1", 8}, // VES_TEXTURE_COORDINATES + 1
	{"TEXCOORD2", 9}, // VES_TEXTURE_COORDINATES + 2
	{"TEXCOORD3", 10}, // VES_TEXTURE_COORDINATES + 3
	{"TEXCOORD4", 11}, // VES_TEXTURE_COORDINATES + 4
	{"TEXCOORD5", 12}, // VES_TEXTURE_COORDINATES + 5
	{"TEXCOORD6", 13}, // VES_TEXTURE_COORDINATES + 6
	{"TEXCOORD7", 14}, // VES_TEXTURE_COORDINATES + 7
	{"BINORMAL", 16}, // VES_BINORMAL
	{"TANGENT", 2} // VES_TANGENT
};
}

//////////////////////////////////////////////////////////////////////////
Ogre::String Ogre::ConvertHlsl2Glsl(const String& hlsl, const String& debugFilenameOutput, ShaderType shaderType)
{
	shaderc::Compiler compiler;
	if (!compiler.IsValid())
	{
		OGRE_EXCEPT(Exception::ERR_RENDERINGAPI_ERROR, "Out of memory!", "Ogre::ConvertHlsl2Glsl");
	}

shaderc::CompileOptions compilerOptions;
compilerOptions.SetSourceLanguage(shaderc_source_language_hlsl);
compilerOptions.SetTargetEnvironment(shaderc_target_env_opengl, shaderc_env_version_opengl_4_5);
compilerOptions.SetIncluder(std::make_unique<HlslIncluder>());
compilerOptions.SetHlslFunctionality1(true); // to access semantic information for the variables in spirv

// Compile HLSL to SPIR-V (function has name GlslToSpv but it supports also HLSL as input through the compilerOptions).
const auto result = compiler.CompileGlslToSpv(hlsl, cShaderTypeToShaderKind[shaderType], debugFilenameOutput.c_str(), "main", compilerOptions);
if (result.GetCompilationStatus() != shaderc_compilation_status_success)
{
	OGRE_EXCEPT(Exception::ERR_RENDERINGAPI_ERROR, 
		"HLSL to Spir-V compilation error!\nFile: " + debugFilenameOutput + "\nOutput: " + result.GetErrorMessage(),
		"Ogre::ConvertHlsl2Glsl");
}

// Get the SPIR-V code.
std::vector<uint32_t> spirv(result.begin(), result.end());

// Convert SPIR-V to GLSL.
spirv_cross::CompilerGLSL compilerGLSL(std::move(spirv));

// Remap input locations to reflect the settings from GLSLProgram::bindFixedAttributes
if (shaderType == VertexShader)
{
	spirv_cross::ShaderResources resources = compilerGLSL.get_shader_resources();
	for (auto& resource : resources.stage_inputs)
	{
		const auto& semantic = compilerGLSL.get_decoration_string(resource.id, spv::DecorationHlslSemanticGOOGLE);
		const auto found = cVertexElementSementicToLocation.find(semantic);
		if (found == cVertexElementSementicToLocation.end())
		{
			// Unknown semantic.
			OGRE_EXCEPT(Exception::ERR_RENDERINGAPI_ERROR,
				"Spir-V parsing error!\nFile: " + debugFilenameOutput + "\nOutput: Unknown semantic " + semantic,
				"Ogre::ConvertHlsl2Glsl");
		}

		compilerGLSL.set_decoration(resource.id, spv::DecorationLocation, found->second);
	}
}

try
{
	compilerGLSL.add_header_line("layout(std140) uniform;\n"); // only to avoid assertions in GLSLProgram::extractLayoutQualifiers()
	auto glsl = compilerGLSL.compile();
	return glsl;
}
catch (const std::exception& e)
{
	OGRE_EXCEPT(Exception::ERR_RENDERINGAPI_ERROR,
		"Spir-V to GLSL compilation error!\nFile: " + debugFilenameOutput + "\nOutput: " + e.what(),
		"Ogre::ConvertHlsl2Glsl");
}
}

I debugged the compilation inside the shaderc library as the locations would be definitely better to fix during HLSL->SpirV conversion and not during SpirV->GLSL, but there is missing implementation of a feature from glslang to provide custom implementation of TIoMapResolver. Inside the shaderc, the function which accepts this interface, is called with just nullptr without carrying about even giving the opportunity to the user to update the mappings. Using reflections from the spirv-cross seems sufficient, but it was necessary to enable SPV_GOOGLE_hlsl_functionality1 through the compiler options. For my simple shaders it doesn't have any impact on the generated GLSL shaders, but hard to say what it does with more complex shaders.