Advanced compilers for Ogre's scripts

[Praetor]

Time for input folks!

I'm tackling variables now, and I'm pretty confident how it will work. Scoping rules are lexical. So, works like C/C++. Each scope can have 1 (1!!!) assignment of a variable. Multiple assignments just overwrite each other. So for instance:

Code: Select all

...
set $diffuseColor "1 1 1 0"
diffuse $diffuseColor
set $diffuseColor "1 0 0 1"
specular $diffuseColor
...

Both diffuse and specular get set to "1 0 0 1". This is due to the order of operations. All variable assignments are processed first, then variable accesses are done (this is due the fact that we cannot guarantee proper ordering, especially when inheritance is involved. If we imposed strict ordering, you're likely to get a lot of uninitialized variable use).

Above I used the set $varName <value> form. What do you all think about that? It seems reasonable to me for variable assignment, though I'm sure there is a more elegant format. Then again, this is very explicit and I'm a fan of that. I was thinking maybe we should distinguish the set keyword a little more. Variables start with '$' and no other keyword can start with that character. Maybe "set" can become "%set". Since this is almost like a pre-processing phase during compilation, I don't want this processing to "get in the way" of things like properties which may have the word "set" in them somewhere.

With regards to what is legal as values. Either a single token such as

Code: Select all

set $var 1

will work, or something like this

Code: Select all

set $var "affector Test
{
    position $x $y $z
}"

Ok, the above is a little crazy. Here's what happens. The parser accepts and recognizes a quoted string. The parser also flags the variable $var as a variable. During the processing phase, the quotes are removed and the inner string is run through the object inheritance and variable processing phases of compilation AS IF THE STRING WERE IN THE HOST SCOPE. That means those variables inside the variable will work as well. Say that variable definition exists in some abstract base object, and we inherit from it. We can do this:

Code: Select all

Particles/Test : Particles/Base{
    set $x 10
    set $y 10
    set $z 15
    $var
}

And that's right, we automagically picked up the "parameterized" affector from Particles/Base. Ok, back to work!

[Praetor]

Ok, update after my evening's work.

The addition of variable processing uncovered a small algorithm design error. I'd been copying the imported scripts straight into the AST of the main script and then continuing to process that. Not a good idea. I was getting an error about using undefined variables (my errors, and it was detecting it properly). Here's what happens:

Code: Select all

Test/System2{
    emitter Ellipsoid{
        color $var
    }
}

Code: Select all

import Test/System2 from blah
Test/System3 : Test/System2
{
   set $var "1 0 0 1"
}

The Test/System2 existed at the top level next to Test/System3 due to the import, and the contents of Test/System2 were inside Test/System3 due to the inheritance. So, you see, the first time variables are processed $var is assigned to nothing (in the top-level Test/System2) but it is properly assigned by the time we get to the second appearance of $var inside Test/System3. So we were getting an extra error condition when we didn't want one.

To solve the problem I no longer copy the AST of imported script into the main AST. A special import table is kept on the side. Variable processing does not touch the import table, so only those parts which are actually used from the imported script get further processing.

My next hurdle is that I have a sneaking suspicion my handling of "object" is fragile. It expects a possibility of a few different forms 1. abstract type name{ 2. type name{ 3. abstract name{ 4. name{. Well, what about vertex and fragment programs, which have a number of extra pieces built into the object header? Instead it should be more flexible allowing these added pieces. Now it is <abstract> <type> name *whatever {. <> means it's optional. By the way, these rules only apply to top-level objects, which must allow for inheritance to use them. Inner objects have their own, compiler-specific rules that at this point, we really don't care about.

[Kencho]

You've got some pretty nice points and features here, /me likes it
The only real concern I have is about the variables' values... It's really confusing and might hide possible errors in the future (for instance, using a value that is changed 30 lines below, and you aren't noticing it). I've given this a brief thought, and this is my idea:
First, don't consider it. Expand everything, import the subtrees, everything. Then walk the AST and verify no value is being finally used before a value has been given to it. If so, cast an error; otherwise, do the substitutions. I think this is the fusion between Ogre scripting philosophy and #define rules

Besides that, everything looks great, as usual Good job!

[Praetor]

Well, good idea. That is pretty much how it works! The system goes into each scope, and handles variable assignments. Then, within that scope is goes through the variable accesses. If an access occurs and there's no variable to access, an undefined error is logged.

I'm pretty happy with the way the variables are working right now. The confusing part was probably because of the example's complexity. There are "compound" variables there (a variable inside a variable value) which is definitely not the normal case. I was just showing off, really. Don't worry too much, I'm able to keep track of files and line/column numbers so errors thus far have been very informative.

In the shower earlier (where all the good ideas happen) I stumbled upon a better way to identify objects. It remove the current need to store a set identifying object types. ScriptCompilers will just magically "find" these objects without any help. Or at least, that's the idea. I hope to get through these last stages soon and start on the bulk of the ParticleScriptCompiler. I want to get into the thick of actually compiling to uncover any hidden issues. After I get into it a bit I'll probably come back and do some profiling and performance tuning on these earlier stages.

[Praetor]

My idea worked. My method of finding objects not only make it unnecessary to search sets while walking over the tree, but makes it very flexible for finding the objects. The format for object headers now supports extra data inserted after the name before the '{'.

For instance

Code: Select all

abstract emitter Name Ellipsoid{}

Since abstract nodes must be named to be inherited from Ellipsoid is relegated to the "extra data" category, but is obviously important information for the compiler. Previously this long style of header would have been unsupported, now it is.

Here's the technique: when searching for objects, first find the nearest '{'. Walk backward until we either hit the name we are looking for, or hit an invalid token (meaning we aren't inside the object header anymore). If we didn't find the name yet, find the next nearest '{' and repeat.

This is a late night edit, so I'll do more testing tomorrow in between getting Lexi working with max9.

[Kencho]

Okay, will trust you then about this

[Praetor]

I have started the particle script compiler now. I can say, the AST route is proving a good bet. I'm flying through this compilation. The best thing is that plugging in custom compiler nodes (like custom properties and objects) is trival. You just override the processNode function in the listener. You are given an iterator. Returning true means you consumed some input, false means you didn't. You can also override how ParticleSystems are allocated.

All in all, going amazingly well. I'll post back when the compiler is more complete, or if I run into anything unsavory.

[Kencho]

ASTs are a really good invention, indeed. I was amazed of their possibilities when I learned about them. Pure solid gold

And if you find something unsavory (I guess I know what that means ), here I'll be to discuss it, you know

[guilderstein]

Hello folks,

I hope I've finally found the right thread on the scripts

I know the cellar might be already full of them, but I have some ideas,
maybe partly new:


***C++ like /*, */ comment support to .material and .program files?
***Many out there would be really happy having this,
***not having to // line-per-line during testing.
***/I guess all scripts use the same parser though,
*** so particle guys would be hepi too /


***Also, I'm not sure whether this is really just getting crazy, or
***has some actual value: I've spotted HLSL preprocessor directives
***are possible in program definitions, but the required syntax makes
***them rather hard to write/read when you have lots of #defines in shader
***code. (No whitespaces allowed between entries, have to be in 1 line)
***I'm pretty sure hlsl compiler uses the same crowded line for
***the directives, but having whitespaces between them
***while working in the scripts could be a big plus.


***And finally, if I'm at complaining about everything , would it
***be possible to export the preprocessor directives up to material
***definitions?
***I mean, within passes of materials, inside the vertex and fragment
***program_ref entries, right above or below the non-default parameters.
***Maybe that would break some interfaces, but we are going to have lots
***of material versions using the very same shader, but only with different
***prep.directives. As of now, I have to write new program definitions too,
***apart from the materials, in which I write the different flags.
***With this idea, no new program definitions would be needed, just the materials.
***Would be much more readable, easier to debug and maintain.
***Also, artists could tweak shaders much more easily this way, since they would
***see prep.directives right next to textures, etc.
***Even one step still outer, having prep.directions in the pass definition
***might be even more useful, since most vertex and fragment programs
***use the same #defines, and thus both reading and creating the passes
***also maintaining them would be much easier and simpler.

Anyway, these are just what I think people would like to have,
/especially me /, because that would make shaders like MONSTER
lot easier to handle.

Hope its still not too late for them to get in,

Regards,

g

[Praetor]

Do you mean right now you can't pass in preprocessor definitions through the material script? Might be worth looking into yes. And if it's added there's no reason I can't condense the thing down into the "no whitespace" form automatically for you.

As for /* */ style comments, I can put those in. Thanks to spirit comments get automatically skipped as if they were tabs or carriage returns. I can add that rule.

If I didn't understand your meaning with the definitions things, perhaps you could give a short example of what you mean? Also you said you had to write a lot of code over again in materials. Hopefully the new abstract, full inheritance, and variable systems increases code reuse and decreases code copying.

[guilderstein]

Yep, I meant those things. Here goes an example on preprocessor
directives (#define, etc):

Code: Select all

material Offset
{
         pass First
         {
                 depth_bias 10
                 start_light 0

                 //the scope of the line below shall be both vert and frag progs,
                 //so copying the strings to the hlsl compiler is a must for both
                 //behind the hood
                 preprocessor_defines
                              NORMAL_MAPPING = 1,
                              OFFSET_MAPPING  = 1,
                              SPECULAR = 1,
                              AMBIENT_OCC = 1

                 vertex_program_ref Offset_V
                 {
                             //the scope of the line below is limited to this vert prog
                             preprocessor_defines
                                          SOMETHING = 2.456
                 }

                 fragment_program_ref Offset_P
                 {
                             param_named_auto diffcol diffuse_light_color 0
                 }

                 texture_unit diff
                 {
                             texture dummy.jpg
                 }
         }
}

I meant something like this. As of now, you can only have
preprocessor_defines
in the vertex_prog or fragment_prog definition, so if you have a shader which only differs in one flag, like #define NORMAL_MAPPING 0 and #define NORMAL_MAPPING 1, you have to create 4 separate programs, 2 vertex, 2 fragment.

Having this above would be great (vert and frag sharing the prep_directives).

Thanks,

g

[Praetor]

I think they'll need to at least stay encapsulated in the vertex_program_ref and fragment_program_ref objects. I don't want shader properties bleeding out into the pass properties. But how about this?

Code: Select all

material Offset
{
         pass First : OffsetPass
         {
                 depth_bias 10
                 start_light 0

                 set $defines "NORMAL_MAPPING=1, 
                       OFFSET_MAPPING=1,
                       SPECULAR=1,
                       AMBIENT_OCC=1"
                 vertex_program_ref Offset_V
                 {
                             //the scope of the line below is limited to this vert prog
                             set $defines "$defines, SOMETHING=2.456"
                             preprocessor_defines $defines
                 }

                 fragment_program_ref Offset_P
                 {
                             preprocessor_defines $defines
                             param_named_auto diffcol diffuse_light_color 0
                 }

                 texture_unit diff
                 {
                             texture dummy.jpg
                 }
         }
} 

I'm sure it will take some time for people to get used to using the variable system, but the above should achieve what you want with only 1 more line of script. And, if the preprocessor_defines property isn't in the ref objects yet, I'll look into what it'll take to put them there when I get on the material compiler.

Given the above, you won't need multiple versions of a program for varying preprocessor defines. Just create an abstract version of the vertex_program_ref and fragment_program_ref objects, put in a $defines variable, then inherit and override in each specific material.

[Praetor]

I'm still working on the ParticleSystemCompiler. No problems, I just ran into something not SoC related in the middle of the week. I'm planning on changing up the test program a bit. It will still contain unit tests, but it will also contain a visual module. This module will compile scripts then visually show them so they can be verified. So, for now I'm creating a simple application to compile and then show a particle system. There is not much left to do actually. The base compiler only knows a few things about properties.

I need to look into how much of the particle systems are built in to the core to know how many specifics should be known by the basic ParticleSystemCompiler. Either way, it is thus far working out just fine with particle systems. I need to get started on the material compiler, since that will be the monster (in terms of size/time).

[Kencho]

I'm planning on changing up the test program a bit. It will still contain unit tests, but it will also contain a visual module. This module will compile scripts then visually show them so they can be verified. So, for now I'm creating a simple application to compile and then show a particle system.

IMHO, unit tests should be run as standalone test suites. They must be run as an automated batch (for the regression tests) and individually (for the "unit" tests with a class/functionality granularity). You shouldn't add the manual tests there either, as that kills the automated nature of unit/regression/integration tests

[Praetor]

Yeah, I know. Kind of a quick test for me at this point. It would be best if I split them up. When I'm more confident in the pace of my progress I'll do that.

[Kencho]

Hehe, okay. However, I must say test-driven development really makes you more confident of your speed and safety (at least that's been my personal experience so far!).

[Praetor]

Ok kencho, your words ate away at me. I split out the demo from the tests. I now have a ParticleDemo project and the parser unit tests separated. The particle demo is a clone of the regular particlefx demo that comes with Ogre, except I use identical particle scripts compiled with my compiler. This is a nice visual check that they are working, and so far things are going well. I've adjusted to a few particle script quirks and have most of the systems appearing exactly as they do in the old demo. I'll work more this week to finish it up. I want to have a second demo, or a second mode of the same one, which compiles new-style scripts, complete with inheritance, imports, and variables to see how a script taking advantage of these features compares to the originals. I'm hoping it is slimmer, leaner, easier to write and easier to read.

I have not yet explored the customization features, but I'd like to. I want to make sure it is as easy as I hoped to handle custom properties or entire custom script objects just by implementing and registering a listener. More on this later!

[Kencho]

Hehe, good idea about the demo clone! Will be interesting to see the results of your forthcoming tests Can't wait to see the results!

[Praetor]

Ok, I've gotten the demo to look like the old demo. Looks awesome, or maybe it just looks so because of the labor invested. I've been checking out the source making sure I've covered all the parameter possibilities and I think it all works. I've committed the compiler itself and uploaded the tests and demo. The URL to the demo zip file (VC2005 stuff) is http://darkwind.rh.rit.edu/public/OgreS ... rTests.zip

My next step is to recreate the demo script again, but this time taking advantage of all the new features like variables and imports. I want to get this done soon, because I'd like to leave July for the material compiler. I don't expect the material compiler to take a whole month to do, but it'll be nice to have some flex time. I'll still have the compositor and overlay compilers to whip up, but those are babies compared to the material. I'll show everyone the script I make for the particle demo soon. Then I need to start thinking about a good way to compare compiler performance. Post back soon.

[sinbad]

I've only just caught up with this thread and I have to say it's looking great. I'm very excited at the possibilities the new compilers offer - keep it up

[Praetor]

I said I would wrap up playtime with the particle compiler before July and it's getting close! So, I've got some news.

I've created a script that does the same thing as the regular particle demo's scripts did, but taking advantage of some of the new features. I have to admit, there wasn't a lot of opportunity in the demo scripts to really improve using things like variables, but I did my best. I'm sure with more realistic cases the benefits will be much greater!

Here's what some interesting portions of the script looks like:

Code: Select all

import * from base.particle

// A sparkly purple fountain
Examples/PurpleFountain2 : Examples/BaseSystem
{
    material        Examples/Flare2
    particle_width  20
    particle_height 40
    billboard_type  oriented_self
    // Set gravity
    set $vector "0 -100 0"
    
    // Area emitter
    emitter Point
    {
        angle           15
        emission_rate   75
        time_to_live    3
        direction       0 1 0
        velocity_min    250
        velocity_max    300
        colour_range_start  0 0 0
        colour_range_end    1 1 1
    }

    // Fader
    affector ColourFader
    {
        red -0.25
        green -0.25
        blue -0.25
    }
}

We have it all: importing, inheritance, and variables. You'll need see inside Examples/BaseSystem to understand the importance of setting the $vector variable. Here is base.particle:

Code: Select all

// First create an abstract affector used to move particles
abstract affector Examples/LinearForceAddBase
{
	force_vector $vector
	force_application add
}

abstract Examples/BaseSystem
{
    cull_each       false
    quota           10000
    
    affector LinearForce : Examples/LinearForceAddBase
    {
    }	
}

The $vector variable actually controls the parameter inside a LinearForce affector, which is twice removed from the final system (once through inheritance to Examples/BaseSystem, and again through inheritance to Examples/LinearForceAddBase).

I admit this is not a very convincing case, but you should be able to see where this is going. In projects with large amounts of scripts a script author would be able to write or tweak scripts much more efficiently with these new features.

Also in the demo I use the listener to override the mechanism Ogre normally uses to find base.particle. Since I know right where it is I bypass the usual resource system, load the script myself with std::fstream and return a ScriptNodeListPtr by calling the parse function. Implementing a listener will also allow you the ability to override the standard compilation behavior for any of the script nodes, or even add your own custom script nodes. I'll try to whip up an example of that later, perhaps after I've sunk my teeth into the material compiler.

I have just a few small things to finish the particle script compiler before moving on:
1. Make it more strict with regards to particle system naming headers, meaning there cannot be any other tokens between a system's name (like Examples/Fireworks and the '{' that marks the beginning of the system's body).
2. Clear the context after a system errors out or is compiled. Right now the context holds a pointer to the compiling system. That pointer is overridden when a new system starts compiling, but is not explicitely cleared after each compilation. I want it to.
3. Provide access to the context. Listeners when they override the processNode function are given a pointer to the calling ParticleScriptCompiler. They should be able to access parts of the context, like the pointer to the currently compiling particle system, through accessor functions, which need to be added.

[Kencho]

This is very slick The example script might look a bit forced, but it's definitely showcasing the new features very well. Good job!

[Chris Jones]

looks good

about loading it yourself rather than using OGRE, doesnt that defeat the point of OGREs resource system?

what if you have your files inside a zip file? or a custom archive?

[Praetor]

Then you shouldn't override it. The decision to overload the importFile function should be thought out well. Here's a rather over-the-top example, but one that is really quite possible: your file are stored remotely. If you need to download files on the fly you obviously are going to need to override it. Or possibly if you use some generated content.

[spookyboo]

Praetor, instead of

Code: Select all

emission_rate    40

I also want to be able to assign a function instead of a value. This means you will get something like this:

Code: Select all

emission_rate cameraDependency(40)
{
   ...
}

cameraDependency is a function, with the value 40 as argument. Can your system handle this (without adding a ton of code)?