Can we add "pre process" support into material scr

[iloseall]

Can we add #define #undefine #ifdef #if #else #endif to ogre material script sytem?

for example :
I make a shader with define.
I think make a material template use the shader with define too.
juet like CGFX or ms effect FX.
Now we make this with create a lot of program and material .

[sinbad]

Can you explain why this would be useful?

[iloseall]

My English is very poor!
Let me try to make a full example to explain this.
Sorry for my fuzzy post.


It is Chinese New Year now.
Happy new year everybody.

I shall do this as soon as possible.

[MartinBean]

Maybey something like this:

Code: Select all

#if HOST_GPU_NVIDIA
   #define some_shader nvidia_shader
#elseif HOST_GPU_ATI
   #define some_shader ati_shader
#endif

material foo
{
   technique
   {
      pass
      {
         vertex_program_ref some_shader
      }
   }
}

?

[iloseall]

Thanks MartinBean;

To MartinBean:
We can use another method to do this when only for shader with ATI or NVIDIA or another user config.
Juse like provider global pre-process macro for all shader:
In my every shader file :

Code: Select all

#include "GlobalDefine.inc"

vertex_.....()
{

#ifdef ATI
	fetch4....
#else
	tex2D....
#endif
} 

then
you create the GlobalDefine.inc with user config information
when game run.

c++ code:

Code: Select all

create_global_define_inc_resource()
{
	if(is_ati())
	{
		global_define.add_define("ATI",1);
	}
	if(is_Nvidia())
	{
		................
	}
	if(user.UseNormal())
                {
                         global_define.add_define("UseNormal",1);
                }
                if(user.SoftParticle() )
                {
                         global_define.add_define("SoftParticle",1);
                }
} 

Or :
we can add a "global define" attribute to ProgramManager with a patch,
every program should add the global define into it's pre-define params when program loading.

-------------
But we could not use the method for material because material can't use pre-process (#ifdef,#include) ;
for example "use or not use normal texture unit in pass","use or not use dot-product texture instead pow function (for specular)","what rtt-format is use in composite",etc..
Now we need create a lot of program and material-technique for this and with scheme to switch.
permutation and combination all options in ".os" file.
I think: you can use scheme when you only provider "low medium high";
but if you want provider a lot of options , the program,material ( technique) or composite explodes.
----------------

[iloseall]

another fictitious material script with pre_prcess macro:

-----------------------------

Code: Select all

AAA.os:

abstract material AAAA
{
	//default value
	set $Scroll_Anim "1 1" 
	set $Bump_Scale 1
	
	T
	{
		Pass
		{
			scene_blend $Scene_Blend
			
			vertex_program_ref ....
			{
				//*** material's pre_define macro will be pass to program define too.
				use_material_macro true
				
#if NORMAL_MAP
				param_named float bump_scale $Bump_Scale
#endif

#if GLOSSINESS_MAP
			param_named ....
#endif

		...

			}
			
					
			texture_unit 
			{
				texture $Diffuse_Map
			}


#if NORMAL_MAP
			texture_unit 
			{
				texture $Normal_Map // normal.xyz=>agb,height=>r
			}
#endif

#if GLOSSINESS_MAP
		texture_unit 
		{
			texture $Glossiness_Level_Map
		}
#endif

#if SPECULAR_LEVEL_MAP
		texture_unit 
		{
			texture $Specular_Level_Map
		}
#endif
	
#if CUBE_MAP
			texture_unit 
			{
				texture $Cube_Map
			}
#endif

#if AMBIENT_OCCLUSION
			texture_unit 
			{
				texture $Ambient_Occlusion
			}
#endif
			
#if FLASH_MAP
			texture_unit 
			{
				texture $Flash_Map
				alpha_op ....
				color_op ....		
				texture_coord 2
				scroll_anim $Scroll_Anim
			}
#endif			
			
		}
	}
	
}

------------------

Code: Select all

BBB_01.os

import * from AAA.os "FLASH_MAP=1;NORMAL_MAP=1;CUBE_MAP=1"

/*
or: 
#define FLASH_MAP 1
#define NORMAL_MAP 1
#define CUBE_MAP 1
#include AAA.os
*/

material BBB_01 : AAA
{
	set $Scene_Blend "add"
	set $Bump_Scale 1.4 
	set $Scroll_Anim "2 3"
	set $Diffuse_Map "bbb_diffuse.dds"
	set $Normal_Map	"bbb_normal_height.dds"
	set $Cube_Map "cube_01.dds"
	set $Flash_Map "flash_01.dds"
}

[Tenttu]

I agree that this is a feature found and commonly used with cgfx and microsoft fx, but one of the last features ogre material scripting still lacks. However this is better for the fx frameworks as they contain the shaders within the same files as the "material script" part. For the approach iloseall suggested to be even more useful the same definitions would need to be automaticly passed into shader compiler.

[Praetor]

All of this is possible. The definitions, the setting of macros, passing macros through to referenced shader definitions and even into the gpu programs themselves.

I think what we need to know is 1) Is it useful enough to be added? 2) is it useful enough to be added to Shoggoth? 3) How do we do it which gives us the most usable results with the fewest dependencies?

Writing a C preprocessor from scratch is not on my list of things I'd like to do for fun. What do we say to each of these 3 questions?

[pekar]

1) I would say yes, I've been using a scheme like this for shader a couple of weeks ago, it's not a bad way to work, even if it is ugly. Suppose you have an incompatibility in your CG shader between DirectX and OpenGL, it would be clearer than having 2 definitions of the shader, material.
2) If it is decided that this could be Shoggoth, I wouldn't mind making the required changes, if no one else feels the call...
3) Boost has one, with all related consequences. http://www.boost-consulting.com/tmpbook ... essor.html. I also have my own, but it would require some work to be robust.

[Praetor]

I know about boost's, and it is good. But we decided to keep boost out of Ogre's dependencies for now. If you are really willing to show us your preprocessor, then that would be very interesting.

[pekar]

I'll show you mine if you show me yours

This is the very basic one that I've been using. It doesn't support macro's or #else, and is only tested on windows.

Code: Select all

#include <string>
#include <sstream>
#include <iostream>
#include <map>

typedef std::map<std::string,std::string> DefineMap;

using namespace std;

class Preprocessor
{
protected:
	template <class T>
	static void stringOf(const T& object, std::string & s)
	{
		std::ostringstream os;
		os << object;
		s = os.str();
	}
	static string replace(string* t, string find, string replace)
	{
		if (!t)
			return *t;

		size_t start = 0;

		if (*t == find)
		{
			return *t;
		}
		
		size_t position = position = t->find(find);

		if (position==string::npos)
			return *t;
		
		ostringstream os;
		os << "";
		while(position!=string::npos)
		{
			os << t->substr(start, (position - start)) << replace;
			start = position + find.size();
			position = t->find(find, t->find(' ',position + 1));
		}
		os << t->substr(start);

		return os.str();
	}
public:
	static string* preprocess( istream* input , ostream* output )
	{
		char buffer[512];

		const string sDefine = "define";
		const string sUndefine = "undef";
		const string sIfdef = "ifdef";
		const string sIfndef = "ifndef";
		const string sEndif = "endif";
		
		
		DefineMap* defineMap = new DefineMap();

		bool ifFlag = true;
		size_t ifDepth = 0;
		size_t lastTrueDepth = 0;
		size_t errorLine = 0;
		string* errorString = 0;
 
		while(!input->eof())
		{
			input->getline(buffer,512);
			string* ptemp = new string(buffer);
			string temp = *ptemp;
			size_t pound = temp.find('#');
			if (pound == string::npos&&ifFlag)
			{
				size_t definePosition = string::npos;
				for (DefineMap::const_iterator iter = defineMap->begin(); iter != defineMap->end(); iter++)
				{
					string retval = replace(ptemp, iter->first, iter->second);
					delete ptemp;
					ptemp = new string(retval);
				}
				*output << *ptemp;
			}
			else
			{
				size_t space = temp.find(' ', pound);
				string directive;
				if (space == string::npos)
					directive = temp.substr(pound + 1, temp.size() - (pound + 2));
				else
					directive = temp.substr(pound+1, space - (pound+1));

				if (directive == sDefine&&ifFlag)
				{
					size_t endIdentifier = temp.find(' ', space+1);
					string identifier;
					if (endIdentifier!=string::npos)
					{
						identifier = temp.substr(space+1, endIdentifier - (space+1));
						(*defineMap)[identifier] = temp.substr(temp.find_first_not_of(" \t",endIdentifier ));
					}
					else
					{
						identifier = temp.substr(space+1, temp.size() - (space+2) );
						(*defineMap)[identifier] = "1";
					}
				}
				else if (directive == sUndefine&&ifFlag)
				{
					size_t endIdentifier = temp.find(' ', space+1);
					string identifier = temp.substr(space);
					defineMap->erase(identifier);
				}
				else if (directive == sIfdef || directive == sIfndef)
				{
					bool condition;

					size_t endIdentifier = temp.find(' ', space+2);
					string identifier;

					if (endIdentifier==string::npos)
						identifier = temp.substr(space+1, (temp.size() - (space + 2)));
					else
						identifier = temp.substr(space+1, endIdentifier - (space+1));

					condition = defineMap->find(identifier) != defineMap->end();
					if (directive == sIfndef)
						condition = !condition;
					
					ifFlag = condition ? ifFlag : false;
					if (ifFlag)
						lastTrueDepth = ifDepth;
					ifDepth++;
				}
				else if (directive == sEndif)
				{
					ifDepth--;
					if (ifDepth == lastTrueDepth + 1 )
						ifFlag = true;
				}
			}
			errorLine++;
			if (errorString)
				break;
			delete ptemp;
			//*output << "Line : " << errorLine << " depth : " << ifDepth << " condition : " << ifFlag << endl;
		}
		delete defineMap;
		if (errorString)
		{
			errorString->append( " on line " );
			string number;
			stringOf<size_t>(errorLine,number);
			errorString->append(number);
		}
		return errorString;
	}

};

[iloseall]

I send a patch https://sourceforge.net/tracker/index.p ... tid=302997
so I can change my material content in callback:

Code: Select all

 Ogre::DataStreamPtr resourceLoaded(const Ogre::String &name, const
Ogre::String &group, Ogre::Resource *resource,Ogre::DataStreamPtr
dataStream); 

I use boost::wave library to make pre-process on the call back method.
en, it work fine.
So ,I think ,pre-process could not be the core function.

code in ogre call back method

Code: Select all

Ogre::DataStreamPtr FXResourceLoadingListener::resourceLoaded(const Ogre::String &name, const Ogre::String &group, Ogre::Resource *resource,const Ogre::DataStreamPtr dataStream)
{
	//Ogre::String resouce_type=resource->getCreator()->getResourceType();

	if(Ogre::StringUtil::endsWith(name,".template.os"))
	{
		size_t size=dataStream->size();
		size_t pos1=dataStream->tell();
		Ogre::String s=dataStream->getAsString();
		std::stringstream* ss=new std::stringstream();
		material_preprocess(name.c_str(),s,*ss);

		Ogre::StdStreamDataStream* stream=new Ogre::StdStreamDataStream(ss,true);

		
		return Ogre::DataStreamPtr(stream);
	}
	return dataStream;
}

preprocess function:

Code: Select all

void material_preprocess(const char* file_name,const std::string& instring,std::ostream& os)
{
	//std::ofstream fs("e:\\export\\Test.material.os");
	//std::ostream& os=fs;
    
	//current file position is saved for exception handling
	boost::wave::util::file_position_type current_position;
    try
	{
        typedef boost::wave::cpplexer::lex_token<> token_type;
    
		typedef boost::wave::cpplexer::lex_iterator<token_type> lex_iterator_type;
        
		typedef boost::wave::context
				<
					std::string::const_iterator, 
					lex_iterator_type,LoadFileResouceToString,
					boost::wave::context_policies::eat_whitespace<typename lex_iterator_type::token_type>
	//boost::wave::context_policies::default_preprocessing_hooks
				>
            context_type;
		
		
		context_type ctx (instring.begin(), instring.end(),file_name);

                                //boost wave check include file exists use fs::exists method,so we must put the template file on file sytem.
		ctx.add_sysinclude_path("..\\GameData\\Template\\");

	
		context_type::iterator_type first = ctx.begin();
		context_type::iterator_type last = ctx.end();
        
    
		bool is_pp_line=false;//for format beautify
		bool is_new_line=false;//for format beautify
		int layer=0; //for format beautify
        
		while (first != last) 
		{
            current_position = (*first).get_position();
			
			const boost::wave::cpplexer::lex_token<>::string_type line=(*first).get_value();
			const char* text=line.c_str();
			boost::wave::token_id token=boost::wave::token_id(*first);
			unsigned int id= ID_FROM_TOKEN(token);
			if(token==boost::wave::T_PP_LINE)
			{
				is_pp_line=true;

				++first;
				continue;
			}

			if(is_pp_line)
			{
				if(token==boost::wave::T_NEWLINE)
				{
					is_pp_line=false;
					is_new_line=true;
				}
				
				++first;
				continue;
			}
			
			
			if(token==boost::wave::T_NEWLINE)
			{
				is_new_line=true;
				++first;
				os<<"\n";
				continue;
			}
		

			if(id=='\\')
			{
				{
					context_type::iterator_type t=first;
					++t;
					if(t!=last)
					{
						std::string t2=(*t).get_value().c_str();
						if(t2=="n")
						{
							os<<"\n";
							++first;
							++first;

							is_new_line=true;
							continue;
						}
					}
				}
			}
			
			if(token==boost::wave::T_RIGHTBRACE)
			{
				layer--;
			}
			
			if(is_new_line && token==boost::wave::T_SPACE)
			{
				++first;
				is_new_line=true;
				continue;
			}

			if(is_new_line)
			{
				for(int i=0;i<layer;++i)
				{
					os<<"\t";
				}
			}

			if(token==boost::wave::T_LEFTBRACE)
			{
				layer++;
			}


			//std::cout<< "["<<id<<"]";
			os << text;

			is_new_line=false;

            ++first;
		}
    }
    catch (boost::wave::cpp_exception const& e) 
	{
		// some preprocessing error
		std::cerr
			<< e.file_name() << "(" << e.line_no() << "): "
            << e.description() << std::endl;
    }
    catch (std::exception const& e) 
	{
		// use last recognized token to retrieve the error position
        std::cerr 
            << current_position.get_file() 
            << "(" << current_position.get_line() << "): "
            << "exception caught: " << e.what()
            << std::endl;
    }
    catch (...) 
	{
		// use last recognized token to retrieve the error position
        std::cerr 
            << current_position.get_file() 
            << "(" << current_position.get_line() << "): "
            << "unexpected exception caught." << std::endl;
    }
}

the function can build new material script from a material script with macro. en, I make some code to do format beautify for readable by human.


in material template ,i can define programname , program , material , render state, texture unit ect with macro.
I make a rule to build program's name so:
if a program with same name in some mateiral file,so the program is same too.
Now , I will get a collision on program.
The greate ScriptCompilerListener::createObject method is used to resovle collision.

Code: Select all

Ogre::Any FXScriptCompilerListener::createObject(Ogre::ScriptCompiler *compiler, const Ogre::String &type, const std::vector<Ogre::Any> &args)
{
	if(type=="HighLevelGpuProgram")
	{
		const Ogre::String name=Ogre::any_cast<Ogre::String>(args[1]);

		if(Ogre::StringUtil::startsWith(name,"Dynamic",false))
		{
			Ogre::HighLevelGpuProgramPtr ptr=Ogre::HighLevelGpuProgramManager::getSingletonPtr()->getByName(name);
			if(!ptr.isNull())
			{
				return Ogre::Any(ptr.getPointer());
			}
		}
	}

	return __super::createObject(compiler,type,args);
}

[FrameFever]

I know about boost's, and it is good. But we decided to keep boost out of Ogre's dependencies for now. If you are really willing to show us your preprocessor, then that would be very interesting.

why will you not using boost for Ogre?

[nullsquared]

I know about boost's, and it is good. But we decided to keep boost out of Ogre's dependencies for now. If you are really willing to show us your preprocessor, then that would be very interesting.

why will you not using boost for Ogre?

It's an extra dependency, one at which is not always 1-click process to get up and running properly and fully.

[FrameFever]

are you kidding?
boost is known for is simplicity and portability, so there is in fact only just one reason do not use it, it's another dependency. not more

[Praetor]

Yes, the reason is that at the time of writing the compilers I was using boost's spirit parser. We were not planning on using any other parts from boost then, so we decided not to use it as a dependency. It is, as you must be aware, a rather large library.