Advanced compilers for Ogre's scripts

[Praetor]

I'm working today still on the pass options. Wow there are a lot. I'm restructuring a bit and splitting out each material property compilation into its own function. No performance or operational benefit, but code readability alone. It was getting messy. When I finish the pass I think I'll have enough functionality to start creating a companion test demo. I will probably just create my own demo instead of duplicating a previously existing one. I have enough experience with materials that I'll know if what I'm seeing is correct (as opposed to particle systems, with which I have very little experience). Hope that's done over the weekend.

As far as the proposed generative gpu system addon discussion, if you want to continue it I'll move it into Developer's section. Otherwise, I'm fine with leaving it here. I don't think it detracts from this discussion at all.

Last thing is a general question: In terms of future endeavors I'm thinking of two documentation-type tasks. I'll create 1 or two wiki articles about customizing the compilers (the listeners and how to use them) and about the new language features like variables. Along with that perhaps 2 final demos: 1 on customization, and 1 on general usage and the new script features. Can anyone think of any other documentation things you would like (even comments, please tell me if and where you think needs some more explanation)?

[Kencho]

For the new scripts features I suggest something like a material framework, like defining template materials (cel-shaded with a single texture, for instance) and then paint a whole scene with different materials, all of them using the same base material (the material textures might be defined in variables, so they can be shared among different -fallback?- techniques). Something in that fashion would probably showcase pretty well the power of what can be done with those features Something like defining a... uniform visual style for the app

Besides... would be too much to do as well some pdf or downloadable tutorials as well as the wiki version? Not now, obviously, but if you don't like the idea I will do them myself; just that it will probably take longer.

I'm excited about this new system. I'm planning to start using it for Caelum as soon as this is officially supported by the core Ogre. Definitely that will let me do some great stuff (like ease to add Caelum's haze to materials!)

[Praetor]

Well, I'll have the actual demo code lying around from my testing. I could comment them up and make them available. Perhaps the demo you suggested could be something which makes its way into the set of demos and samples that come with Ogre.

[Kencho]

Fine here Was just an idea I had

[Praetor]

So I sort of got done what I wanted to get done over the weekend. I have a simple material compiler test demo, and all the pass properties have been coded. I have no idea if they work, and no doubt some of them like the iteration option is completely messed up, but the demo should help iron that out.

For the demo I was thinking of using the combination of technique schemes and new script features to define a set of different "themes." Something like glossy, stone, etc that you can switch between. Then I'll use as many features as I can to write as little actual script code to accomplish it. Hopefully there will be close to zero code reproduction thanks to the new features.

[Praetor]

I'ev taken a step back from the compiler for a few days, but July marches on and I wanted to get the material compiler done before August. I still need to test to make sure the pass options work. Putting in the texture_unit and gpu program stuff should be quicker now. I'll just need to see how default parameters are handled now. I used to know, but I'm getting rusty! Beginning of August I'll get to work on a material demo. Then come compositor, then documentation. I figure even if the summer ends and documentation isn't done, I can just sit down with a cup of coffee for a few nights and pound it out. Same with the overlay and font compilers.

[jacmoe]

I am really looking forward to using it!
This is going to revolutionize scripting in Ogre!

[sinbad]

Default parameters are handled just like regular parameters, the GpuProgramParameters instance is just held on the GpuProgram (getDefaultParameters()) instead of the pass.

You're right that proving the parsers with materials and compositors is the primary goal, although given the complexity of some of the new features you've added, docs will be essential too. Looking forward to seeing it all slot together!

[Praetor]

Do I need to call anything to propagate the default parameters held in the GpuProgram into the final parameters structure assigned to the pass, or is this handled automatically?

[sinbad]

This is done automatically in the GpuProgram::createParameters method - essentially the new params object comes pre-initialised with the defaults, ready to be updated / added to.

[Praetor]

That's easy. Great.

[Praetor]

Took a bit longer than I expected to get this far. I'm just digging into the gpu program stuff. Hopefully I can get a good amount done early in August.

I'm going to ditch my custom test for now. I'm going to start actually integrating the compilers into the core. To do this there will be a #define in OgreConfig that will switch usage of the new compilers on. When the new compilers are switched on the MaterialManager will have a new member variable of the new compiler. It will use it for parseScript and any other calls to initialize the materials. This will let me use the current demos as tests for the compiler, in a wider range of situations than I can invent. Of course, the new features won't be stressed, but I can do that myself. My most important goal is making current scripts compile first.

Of course, by default the switch will be off. You'll need to switch it in in OgreConfig.h and recompile. The hope is, before Shoggoth is released the compilers will be ready and the switch can be on by default, or even removed completely.

[sinbad]

Sounds good. Would perhaps be worth inviting people to bombard you with their scripts for some quick and easy regression testing - I know our examples don't stress some aspects as much as full-blown projects.

[Praetor]

It appears there are now 3 competing compilers in the MaterialManager, the serializer, the new compiler, and my new new compiler. By default, mine was set up to use the old serializer, which seems strange. Is that because the newer compiler never quite passed performance requirements?

Either way I think it is begging for some cleanup. The parseScript method is now choked with #ifs and #elses.

[Lioric]

Yes, if i remember correctly, it was because the execution time and the memory reqs were not optimized yet

[Praetor]

There is such a bulk in code for compiling materials that I'm stuck fixing all of my typos and stupid errors. But, integrating the compiler into the material manager is making testing a lot easier. From what I can tell after I get it compiling the old scripts I need to seriously look into the performance.

[Praetor]

So I've been having trouble getting the examples to work with my compiler, and I finally figured out why. There is a problem all the way back in the parser. The rules are mismatching when trying to parse the texture type ("2d", "3d", etc.). The number rule matches the 2, then the word rule the d, splitting the token into two. What I need is to manufacture the word rule to match first. However, if that happened, the word's tolerance right now would allow it to match ALL numbers. I suppose I could just make the word match the general token type and manually check if a token is a number type during registration. Thinking about it, that seems like the best choice now, and the easiest. I want to get this thing back on track, because I'm getting the impression that I will have performance issues to work out before the deadline.

[sinbad]

Yes, I've been updating the MaterialScriptCompiler as I add new features even though it's not used just in case we ever decided to make it mainstream. Because of the lower performance and higher memory requirements it was never made the default choice.

Given the enhancements I doubt that you'll be able to make your compiler as fast as the original MaterialSerializer simply because of its simplicity, but obviously the aim should be to get as close as possible. There are two factors here to mitigate any final differences between the two:

- Extra features which were not available on the simpler compiler
- Ability to precompile scripts to faster bytecode or similar

Your compiler should have more weight on the former than the MaterialScriptCompiler did.

The latter was something we mooted for the MaterialScriptCompiler but it never got done. Saving the output of the lexer as a 'binary' material file and having that as a potential input for final dsitribution is an interesting option - you wouldn't do it during development, but to speed up the loading of the final product and to make it harder for people to tamper with the materials it would be a very useful extension.

[sinbad]

On that note, having looked at the code for the first time in a while, is there any reason why you're using string comparisons in the compiler rather than token IDs? Things like this:

Code: Select all

if((*j)->token == "lod_distances")
    compileLodDistances(j, (*i)->children.end());

I would have expected "lod_distances" to have already been lexed into a token ID by this stage, as in MaterialScriptCompiler - that would certainly lend itself to bytecode later. There appears to be lots of this in the code - am I missing something?

[edit]Could this also be linked to the '2d' word / number problem too? If the parser already knew how to manage '2d' as an identified token rule would that eliminate the confusion? I may be completely off the mark here.

I'm not asking you to change anything like this at this stage (way too late in the day as regards SoC, although I'm hoping you'll be staying around afterwards anyway), just trying to understand the choices right now.

[Kencho]

On that note, having looked at the code for the first time in a while, is there any reason why you're using string comparisons in the compiler rather than token IDs?

I haven't looked into this, but I suppose it's a con of the compiler extensibility. If it's true that plugins will be able to easily extend the scripting language without having to change the compiler itself, then you can't assign token ids beforehand.

[Praetor]

One of the performance things I need to profile. If string comparisons are the main culprit I have an idea how to fix it. The main issue with that is that basically I wanted the lexer to be flexible enough to lex any of the scripts. Either it had to be completely agnostic to specific keywords (as it is now) or it had to know ALL of the possible keywords, which seemed not too elegant. My idea will be that each end compiler will create a map of string->tokenID, which can be passed in to lexer. This map will be queried as the AST is built and a data variable of the token struct set. This would ensure only one string comparison would be needed for each keyword.

On the lexer rule problem, my idea worked as far as I can tell. I didn't get the examples fully working, because I believe there are a still a few bugs in handling a few keywords which is causing a hang. I'm tracking them down quickly now though. I should be able to get this onto my desktop to profile soon.

[Praetor]

While I'm poking around the parser already I might as well make the change to using tokenIDs. It's a "big" change, so it will cut into other progress but it has some advantages. Besides regular performance, string tokens will only be stored when the tokenID cannot be determined, saving memory. In the compiler, stupid spelling mistakes will be caught by the compiler. Extensibility will be maintained by adding methods to the listeners which can override the string->tokenID map before the compiler's input it fed to the parse function.

About pre-compilation. It will help, especially if the need to store strings is removed. Scripts could be condensed down to a very small package. What I think we can do is eventually change the MaterialSerializer to a general ScriptSerializer, which calls parse with the correct tokenID map, then takes the generated AST and saves it to disk. The compilers already have a compile override which accepts an existing AST, so compiling these binary scripts will be as easy as serializing the file into a ScriptNodeList and feeding it in. Without profiling I can't say how much time would be saved though. Obviously not doing the parsing at runtime is nice, but traditionally the bulk of a compiler's time is not spent in parsing, but in the "code generation" phase. That would be when we transform an AST into the actual Material or ParticleSystem.

[Kencho]

What I think we can do is eventually change the MaterialSerializer to a general ScriptSerializer, which calls parse with the correct tokenID map, then takes the generated AST and saves it to disk.

Then you could only serialize resources created through scripts, don't you? You would need to add a way to create ASTs from code-generated resources (something like reverse engineering).

I suggest to leave this for the post-SoC phase

[Praetor]

I suggest to leave this for the post-SoC phase

Without a doubt yes.

[Praetor]

I've been poking and prodding the grammar for some time now. I plan on pounding out a new revision of the system from parser to final compilers this weekend. Or as much as I can with whatever illness I've come down with.

I'm redesigning the grammar from scratch, and using more advanced spirit techniques to make a lighter-weight, faster parser. The drawback is that if you looked at the code in OgreScriptParser.cpp your eyes might just bleed. If you want to see the hell that template metaprogramming can create, just look at the concrete type of the parser declared in script_grammar.

Anyway, the benefit of this revision is that the AST generated is much "nicer" (well-formed). It will provide a lot more information just by its basic structure during later compilation. It shouldn't be too hard to change the basic code handling imports, inheritance, and variables. Those concepts remain the same. I'll post back soon with better examples of what I'm talking about when I finish the new grammar.