Advanced compilers for Ogre's scripts

[Kencho]

Code: Select all

import BaseMaterial from bases.material
import * from bases.material

Kewl! Though maybe the star looks a bit artificial using that "natural language" form... What about adding a new reserved keyword "everything" or "all"?

Code: Select all

import BaseMaterial from bases.material
import everything from bases.material

[jacmoe]

I think "*" is natural enough - if it isn't, I'd want to write "all" instead.
"Everything" is way too long.. !

Most of the stuff you're doing here, Praetor, is over my head, but it seems like it can provide some very powerful scripts!
I mean, inheritance and polymorphism in materials, and whatever..!
I am looking forward to seeing where this one leads!

[Hudson]

I prefer the C/C++ style

[Game_Ender]

If you are doing an import syntax why not stick to python's:

Code: Select all

from bases.material import BaseMaterial
from bases.material import *

That seems to flow better in my mind.

[jacmoe]

It flows backwards in my mind ...

[Praetor]

@ game_ender yeah the first version was inspired by python. It's been a while since I programmed in it though so I just got the order backwards. I've always thought that python's way is backwards though. Seems most people like the natural language format, which I think is a little less cryptic as well. I almost would prefer to cascade this style downwards into the language (like using "extends" instead of ':' for inheritance) but perhaps that can be done in later versions.

I also like the '*' in the imports. It sees some fairly widespread use with this form of import statement.

So, shall we follow tradition and take the hint from python? from ... import ... or import ... from ...?

@sinbad I suppose keeping up with any property additions or changes in shoggoth will be mostly touch-and-go. I don't see any really good way of keeping up. Especially for the material compiler there should be a good amount of time for review to make sure I caught everything.

[sinbad]

Ok, I'm updating the old serialisers anyway so a diff on those should catch it.

[Lioric]

Make sure you are susbcribed to the CVS mailing list and review all the log messages, you will be up to date about the changes in the material system and where they are being made, you will be aware of any important change that will need to be merged in your branch

[Praetor]

@Lioric Excellent idea. I just subscribed to it. With all these mailing lists I've subscribed to for SoC i suppose it is time I got versed in gmail's filters.

I've put in the import ... from ... syntax as the importing format for now. It can be changed later fairly easily. I've begun stubbing out the ScriptCompiler class. I will probably also be stubbing out a ScriptCompilerListener class soon as well. It will provide the callback mechanism for overriding the importing behavior (if you have some custom way of loading scripts besides Ogre's regular resource loading mechanisms). It will also serve as the base class for the derived compilers' listeners (like MaterialCompilerListener, etc.) which will extend it with their own, specific callbacks and overrides.

[Praetor]

I've moved beyond parsing, and the parsing errors. Parsing errors bail out, so only 1 error is reported per parse, whether or not there is more than 1 syntax error in the text. I've begun on the compiler, which now deals with semantic errors. I want to do these differently.

My goal here is aggregate errors during the compilation phase. A list of errors is maintained. If an error occurs during compilation, the function returns false. The user can then query for the list of errors. This means someone can get a full read-out of all semantic errors, instead of having to find out the errors one at a time. So, I'm going to create a ScriptCompilerError struct which stores error information (codes, most likely, along with file and position info) and store a list as compilation happens in a class variable. At the end of compilation, the list sticks around until the next compilation command. You can query that list at any time.

[Kencho]

Sounds good

[Praetor]

Question: most things in Ogre's script system have a sort of "type" name.

Code: Select all

material Test...
technique Test...
affector Test...

but not the top-level particle system. That just has the name, with no type definition. There is no "particle_system" or "system" or anything before it. Why is that?

[EDIT] It seems this is true for particle and overlay scripts. I suppose I'll have to allow those compilers to override the behavior of finding script objects to allow for this.

[sinbad]

Material scripts used to be the same way, back in 2003. The material/technique/pass keywords were added for clarity in the material overhaul in Ogre 0.13

We didn't update the other scripts just because they didn't need it and the less upgrade hassle the better for users. We made a material upgrade tool for the 0.13 release (which is still hanging around, we should probably get rid of it now!) and would have had to make one for all the other scripts too.

[Praetor]

Well, I've run into problems. The gist of it is, after analyzing particle scripts more closely I've concluded that my parsing rules are too narrow. They were made with assumptions that are based on my knowledge of material scripts. I assumed way too much. I now have to completely redo the parser, to make it more general. The basic concepts remain the same, but my rules are different. I no longer am assuming anything about the structure of the script, other than there are variables, words, and blocks denoted by {}. Newlines are stored to be sorted out by the compilers later. The AST that comes from the parser will be very crude.

[Praetor]

dealing with this now: should I worry about platform issues with regards to newlines. I need to be recognizing specific newlines, and I can use defines with ogre's OGRE_PLATFORM macro. I wanted to make sure this was even necessary though. My gut says not to just rely on \n.

[Kencho]

I've seen a lot of programs ruined because of that. I would pick them all. After all, it's just a couple more rules in the lexer/scanner to recognise the NEWLINE token

[Praetor]

Yeah I've coded it up right now to use \r\n on windows, \n on linux, and \r on apple. Did I get those right?

It just makes writing tests a slight bit more tedious, but it isn't a big deal. Hopefully I'll be back on track with a prototype ParticleScriptCompiler next week.

[Chris Jones]

what happens if the files are made on windows but used on unix? or any of the possible combinations, shouldnt it just not care what type of line endings there are?

[Praetor]

Yes, that would be nice, and right now the parser skips over all line ending characters in any combination. I'd like to keep it that way. At work here are two forces: the flexibility required to support the varied lengths of object "headers", and my desire not to force compilers to pass around specifics about their keywords.

The first problem can be shown with two examples

Code: Select all

Test/ParticleSystem1{}
fragment_program Test/FP cg{}

The length of headers varies greatly. Add in the optional behavior to have either of those inherit from another object and it stars getting very ambiguous. The problem is, once inside an object (like a material) how do you discern between a technique definition and a property definition, without knowing ahead of time keywords like "technique" and "diffuse" and without taking into account newlines? Ambiguity is just bad for a language, and this one has way too much dependence on context for the parser to do this much work.

So, why do I want specific compilers to not have to pass around keywords? These compilers that are specific to one resource type know what keywords they are looking for. Forcing them to store those keywords in a structure, and pass it around for a reusable parser to know will be slow, memory intensive, and inflexible. What happens if a plugin wants to add something? Yes, it can be done. It just isn't the correct way to do it.

I think I can make it work in a newline-independent way. The only landmarks for the parser to organize the AST by during parsing are the '{' '}' characters. Other than that the input will be tagged with a few ID codes and fed as a largely unsorted blob. Since the final compiler knows how many values need to be read for a "diffuse" property, it can easily sort out the input stream of tokens. Same with headers. Based on the context, the material compiler will know that given certain conditions "fragment_program" will validly define a fragment program, and be able to extract the correct information from the header leading up the '{'. Errors can also be more contextual and hopefully, more helpful.

So to summarize. Some back-tracking, yes. But not too far, and I think I can sort it out quickly. It was totally my fault. I made far too many assumptions based on my pretty extensive knowledge of material scripts, and my less-than-adequate knowledge of the other script forms. Won't make that mistake again.

[Kencho]

what happens if the files are made on windows but used on unix? or any of the possible combinations, shouldnt it just not care what type of line endings there are?

Exactly. You can guess the mess when I carried my school work from my home (Linux) yo the university (Windows). "^M" at the end of each line

I think you should treat them all as the same token type, independently on the platform the parser is run on.

[Praetor]

I have just recently finished prototyped code in the ParticleScriptCompiler which implements script inheritance. It need a lot of testing, but there is some indication of good news. The code in that compiler right now is so generic, it can be moved to the ScriptCompiler base class if nothing changes. This means implementors don't need to worry about inheritance concerns, it just works. There are two restrictions: 1) single inheritance only for now (not really a restriction), only top-level objects can be inherited from, so there no way to reference a technique buried inside a material somewhere, you will have to pull that technique's definition out and label it as "abstract"

Code: Select all

abstract technique BaseTechnique
{
    lighting off
}

vs.

Code: Select all

material Test
{
    technique
    {
        lighting off
    }
}

Also, the interfaces for the listeners are beginning to reveal themselves. They already had a mechanism for overriding script importing, which I am already utilizing for testing the import behavior (I'm not even initializing Ogre's resource system in my test cases, so I've hard-coded imported scripts). Two more functions have made their way in for a trial period. They are intended for those wishing to extend existing scripting systems. They return a set of strings which correspond to valid object types (for identifying and delegating object compilation) such as

Code: Select all

<new-object-type>
{
    property1 value
    property2 value
}

and the other to custom property names (like adding a "glow" color property to a material pass).

Next up: more testing of this inheritance implementation and then handling of variables.

[Kencho]

Can't wait to finish my exams and sink my teeth on this!

[sinbad]

Looks like you're making great progress. I have my hands full at the moment btu I intend to take a better look asap. Are you getting enough feedback?

[Praetor]

Pretty good feedback. So far I'm still developing things that have been discussed back and forth for a bit. I just need people like Kencho challenging my ideas. Helped me hammer out variable and inheritance handling, which seems to be working well so far.

I think it'll get more interesting once I start churning out actual, usable compilers for the various scripts.

[Kencho]

Pretty good feedback. So far I'm still developing things that have been discussed back and forth for a bit. I just need people like Kencho challenging my ideas. Helped me hammer out variable and inheritance handling, which seems to be working well so far.

Challenging others' ideas is my expertise Glad that helped you Things are looking pretty good and promising at this moment!