Visual Scripting

[Spanky]

Hey guys,

I'm currently working on an editor (yes.. another one) and I am thinking about things in the future. One of the plugins that I think would benefit my game (an RTS game but will probably reuse this system in the future if it is productive) is a visual scripting system.

I've worked at a game company that used Unreal Engine v3 and they had Kismet which I thought was really useful. I could see a lot of potential with it and in the short time I got to play with it, I did some interesting things with it. I've been meaning to play with it some more lately (it comes with Unreal Tournament 3).

For those that don't know, it's a way to visually lay out blocks which represent 'actions' that you can perform. For instance, you could lay down a block that says "on enter" and then spits out the player that entered. You could then feed that player into a "health drain" block that has a constant health drain of 1.0 units per second or something. It seems a very fast way to prototype game play ideas and to perform scripted events without involving a programmer and possibly screwing up the code base.

The Crytek engine has something similar called FlowGraph.


Click for larger version

I'm just wondering what peoples thoughts about using such a system are. Do people see this as a productive way to get designers more involved in actually creating content or do you see it as a way to allow designers to get by without having to learn to script?

Would people think this could be useful in their projects? I'm interested in hearing about anything related to this as I'm just investigating the viability of a project like this. It's a decent amount of work (for one guy) to do but if it would be profitable in terms of productivity and ease of use, I'd love to look into it.

Shawn

[lithander]

We use a visual scripting system for our game aswell. Basically the scripters build statecharts with a visual tool to define the behavior and interactions of items, characters, dialoges etc.
The statecharts can trigger sequences and the sequences perform a number of actions in the game. There are all kind of predefined actions that will have a character speak a text, walk to a specific position, play a special animation, focus an item etc but there are also script actions that just perform a lua script on execution and thus allow ultimate flexibility.

One of the biggest benefits of this approach is that a statechart allows you to visualize and manipulate the current state of the game while playing in a very intuitive way. While playing you see the states becoming active and inactive. If something happens or doesn't happen like you planned it a detailed event log let's you track the error. Why didn't that event get fired? Why is he still in this state? All these questions can be answered runtime and you can even trigger something manually or revoke it to continue testing without having to restart.

Another plus is that the statecharts are created in the prototype editor which allows us to setup the game logic and play the actual game very early. Some standard sequences are generated that are basically executing some stub actions and can be fleshed out later in the development process when the real assets are available. This doesn't affect the game logic in any way though so the prototype is still playable.

My collegue who came up and implemented the statecharts thing has published an article about it in the latest volume of the AI Wisdom book series in case you want to know more about it.

Visual scripting is a very interesting topic indeed. With the complexity of modern games it's really important to allow for a rather failsave, intuitive way to define behaviour and interactions of your game elements. It's always worth considering how to improve the content creation pipeline. And to polish a games flow it's great if you can play it long before you decided to spend cash on the assets because then you are free to change what isn't fun. Later when a scene is finished such gameplay optimizing changes are too costly in terms of time and money to be an option.

[Spanky]

Awesome. Thanks for the reply. I really appreciate it.

Did you guys experience any sort of huge performance problem with this statechart approach? I think a lot of concern about these visual scripting systems is that they are nowhere near the performance of pure C++ code. I'm just wondering how well it worked out for you guys.

In terms of the cost to develop, do you think the trade-off between fixing incorrect scripts out weighed the cost of developing all of the logic by hand? I could see a few problems with bad scripts or helping a designer implement one correctly but I don't think that they would be as time consuming as trying to write it all by hand considering the amount of 'logic' and 'scripting' that usually is done in a huge game.

In my own thoughts, I can't imagine running more than say 10-15 of these scripts at any given time. There may be a whole bunch of them in a level, say hundreds, but only a few will be actively running at any given time. They are a bit slower but I don't think it would make a huge impact on the game, especially when compared to the amount of programmer time it can free up. Worst case scenario is that some of the bigger, slower scripts will need to be implemented in C++ once they've been fleshed out. But again, this would not take long since the programmer, or a co-op student, could translate the statechart directly into function calls.

From what I've heard, the entire Gears of War was done using their visual node-based scripting system. They obviously had to code up custom nodes to be used but I think those would be far simpler to create and later hook together in a GUI than to program entire systems by hand.

Has anyone got any ideas about how they would implement this type of functionality? I'm thinking that each node would have some data-in, data-out, start events, and then some output events.

For example, a timer would have some data-in as the time to run as well as how many times (repeat or not). The start events could be 'start' and 'reset' while the end events could be 'timer expired'. This would allow the script to be currently 'in' a few different nodes and as each one finishes, it triggers one or more end events which are then fed in as start events to other objects. So you could have 1 event finish (such as a timer) which then starts two other nodes in the system. A node could somehow idle inside of it's own execution state for a bit if it needs to (not actively blocking but delaying the firing of the end events) before it finishes.

[AshSid]

Has anyone got any ideas about how they would implement this type of functionality? I'm thinking that each node would have some data-in, data-out, start events, and then some output events.

Our engine uses "two level scripting". The upper level is done in scripting language (currently Python). The lower level is ControlObject class hierarchy (implemented in C++), its objects are defined by configuration files.
"Simple and frequent" game funcionality is performed by lower level, and higher level does more complicated (but also more rare) tasks. Interface between levels is really simplified, but upper level scripts have access to virtually all atributes of lower level.

[Raketenmann]

I'm working on the same game as lithander and developed that statechart thing he mentioned. At the moment we run hundreds of statecharts at the same time without any problems, but thats not suprising: pure statecharts ( the ones proposed by David Harel back in the 80's) are pure reactive systems. If nothing happens, they don't effect your performance. We are doing an adventure like game and the benefit for us is, that all the hundreds of items only have to be concidered if they are e.g. clicked. So one click fires a single event to a single statechart, that one may trigger some more, but its not much at all.

Performance problems would arise, if you need to send an update-event to every statechart.

The scripting guys in our company have done some python scripting for another adventure before and now had to jump into statechart. Surely it wasn't easy for them at first, because our scripting had some different concepts. But after working with the system for some weeks they are very pleased with it.

I think one major part is to give the scripters all the benefits they are used to from written scripts: Our system is able to copy any selected states into the clipboard and paste them into another statechart. Additionally you are able to paste that selection into a textfile and it then translated into xml. So a simple text file can serve as a multi-slot-clipboard or one could send a selection via an Instant messenger etc....

(For the very interested: http://www.w3.org/TR/scxml is the XML-Standard we rely on.)

[sinbad]

I'll just add my 2c here. I used a heavily visual programming system some years ago called 'VisualAge for Java' from IBM - as a paradigm, it made creating behaviour incredibly fast and mostly intuitive. However it had a major downside that maintaining that behaviour over time was difficult, particularly as debugging was rather esoteric. If you were wondering why your behaviour was off, you had to often debug through generated code to figure out why. It was fine for simple things, but as our needs got ever more complex, we really missed working directly in code rather than trying to follow visual 'wiring' about to figure out what was going on. We used it for over a year so it wasn't a learning curve issue, it was that the system started to really get unwieldy as it grew.

I think for non-programmers it's a great option but as a programmer I found myself feeling quite burned by the whole experience and would think very seriously before using visual programming techniques in another project.

[Dom]

I've been using Virtools for over 7 years now. It uses some kind of hierarchical flow-chart. The elements are quite similar to 'code' functions or components: n inputs and n outputs for triggering the logic flow and separate inputs and outputs for the data flow.

For creating behaviour-logic it's really nice - as long as you deal with High-Level "BuildingBlocks". But you can create your own via Scripting or C++. Some years ago I've read about many people that are used to code behaviour in c++ that they dislike that concept, but actually it's quite close to the components/mvc paradigm. Moreover it allows you to organize your logic in a hierarchical way which has the benefits of that you have real visual level-of-detail - thus you "see" what's going on quickly.

For the debugging, it has a trace mode that highlights active elements in red. You can set breakpoints plus values get updated and displayed in real-time too. This is in my opinion a very important aspect of the concept.

For performance: there are impacts, which is due how they did their implementation (a bit like a parser/interpreter), when you don't consider some rules when organizing complex graphs.

One really need to judge correctly for efficient use what to implement on what level: c++, scripting, visual scripting. Then the visual tool will, in my opinion, be a real productivity boost for creating behaviours.

There are more and more middleware products that add this kind of tools to their pipeline ...
I would be very interested to see the approaches others use for their own implementations.

[Spanky]

From what appears here, it seems that most people view it as a decent idea but current implementations of it end up biting them in the ass later in terms of performance and debugging.

I probably wouldn't use the system as a replacement for C++ code written by a programmer but I think a fast prototyping tool available for a designer would be decent. I know a lot of designers who don't know how to script or even use XML. Giving them an excel spreadsheet sometimes complicates things beyond measure.

@sinbad: If you had a better way of keeping the behaviours up to date and debugging them was a lot easier, would you consider using a system again? And was this system used by programmers or was this used by non-technical people (regardless as to whether they *should* have been technical or not). What were some of the problems you encountered while updating the behaviours? Was it just adding new functionality meant going back over all of the existing scripts that used that node?

@Dom: I had heard of, and seen briefly, Virtools before. That debugging sounds like it could be a huge help in figuring out why your scripts aren't working. Did you have a good experience with this type of thing?

@Raketenmann: Thanks for the info. Having the ability to translate into XML with a copy/paste is a really good idea. Is the system that you guys have a visual one? You mentioned that they had done python scripting so what made you guys decide to move away from text and go to a GUI based approach (if that's what you did)

Has anyone ever heard of a system where you can write these things using a GUI based approach and then easily convert it over to code once it's done? I'm thinking of some way to have an auto-generated file created that will get compiled on the next daily build cycle. Hopefully that could reduce some of the overheard of these types of systems.

I'm thinking about using this more for a simple logic system, not for creating AI inside of this thing. For instance, if you walk into a volume, this will trigger one of these scripts to execute and could dim the lights, spawn some enemies, and maybe play some music or something like that. Simplistic things but that could be chained together in very unique way to create different in-game elements but without having the designer have to compile code each time.

[sinbad]

For the debugging, it has a trace mode that highlights active elements in red.

This was what I was sorely missing in my visual programming experience. Had it been present along with a better way to structure logic, I might have felt completely different about the whole thing. This particular system was intended to be used by junior coders - so they weren't very technical, they just had logical abilities and the drag-drop-wire approach was very easy for them to pick up. The idea was that you put all your more experienced coders to work on the back-end server code, and the junior / designer types could use this tool to do the UI.

The problem was changing the end results later - when the wiring was complex, seeing the underlying sequences and conditional behaviour (wired through visual 'if' and 'switch' beans), it was very hard at-a-glance to see what would happen, and debugging was not possible except through the generated code. Both things meant it did not scale well, giving me the impression of 'quick to create, slow as hell to maintain'.

[Kencho]

However I think this greatly increases productivity in contexts where programming isn't needed; only scripting. I mean, artists love to work with these kind of editors for things like materials or object hierarchies. I clearly see the advantages of using such kind of editors for the Ogre scripts, as well as many other scripts. You can easily create materials, you can easily create compositors...

Using these editors for programming goes much further. It's like the blocks' diagrams (KOP) when programming industrial robots. It's nice to see "okay, property A from here is mapped onto property B there", but having flow control... "property A is mapped onto property B if C is active, but otherwise to D, which is another conditional and..."

Visual scripting: yes; visual programming (as treated here): nope.

[Spanky]

@Kencho: How would you go about setting up all of the various 'triggered events' that an RPG or similar type game might have? Let's assume that the design team can't script at all so they just think of things they want done such as 'when you push this lever, I want this door to open' or 'If you come into this room and your gun is out, the ninja attacks you'.

I see a few options for this type of thing:

1) Get a programmer to code this up in actual C++ code. I could see this taking a lot of time to do this though, especially when a game contains tons of these types of things.

2) Integrate a scripting language such as lua or python to allow someone who knows how to script write these things.

3) Integrate some form of this visual 'programming' (I wouldn't *really* call it programming since it is very basic concepts and would probably only have an if statement support (If health < 50) do A, else do B.

I can see advantages and draw backs to all of them, most of which are fairly obvious. In terms of speed, things get worse from top to bottom. In terms of ease of creation, things get easier from top to bottom.

As far as maintenance and debugging are concerned, there are various trade offs for each. Depending on how valuable your programmer time is, it might be more beneficial to leave a designer to figure out why their thing isn't working.

Shawn

[AshSid]

Spanky,

you can combine all these options, each to do tasks for which it suits the best.

You can also look at Blender, its game engine has both python scripting and "logic bricks" as a kind of simple visual system for game activity definition.

[Kencho]

Put it simple:

That's this:

Code: Select all

if (!((A && B) | C))
  C = C - 2;
else
  A++;

To me, both look as cryptic in the eyes of a non-programmer.

Visual programming/scripting is just a way to abstract the language grammar and data flow to a visual level. You follow the arrows, you follow the data. If you add control flow to that visualization, what's following an arrow? Following data or following control? If following control, when do you have to follow an arrow and when the other? And how many times if you have a loop?

It's all additional complexity to something that a pseudocode language might solve. Write the example above in a BASIC-like language, and it's like speaking in english:

Code: Select all

if not ((A and B) or C) then C = C - 2, else increase A

When you deal with scripts (no control flow), the complexity is much different. You only have to track data, and understand how each block combines both to create new data. Imagine having two blocks representing two texture units, and a third block that reads "Blend: Additive". You easily guess what will come out, without even knowing which syntax Ogre scripts use.

As for the triggers, they're indeed different than if's, switch's and for's. They're more like callbacks or function calls. You have a trigger reading "Player enters zone05" and then you point at some very high level operation. Triggers, functions, events, etc., in a block language, are two or three levels above the language grammar constructions such as assignments or conditionals.

[Spanky]

I think what I was trying to explain didn't come across correctly. When I said visual "programming", I think I made a mistake.

What I was meaning was something like the following (although, I agree that the usage of the 'Player' is a bit unwieldy to say the least):



You can follow the in/out links to see when blocks will get activated. Each block does it's own little action and you can pass in parameters or get the results out (the blocks underneath each of the nodes)

In the picture, it's easy to see that the trigger event that starts the sequence is when 'Trigger_0' is used. Once that happens, you can simply follow the flow across the links. There are actions for cameras and the controller in there and then if the player has less than 50 health (the 50 is typed in so it's not visible) then they are destroyed. It's a rather simple example and I am aware that some of these can get much more complex but I think this is more of what I am talking about, not so much of the 'programming' that you showed Kencho. I would kill myself if I ever had to work on a system like that

There is a bit of logic in this script as you can see that an integer is compared to something else. There are outputs for the various cases but this feels more like a flow chart. I don't think I would have any more logical operations in there other than maybe the ability to increment or decrement a counter. Just very simple things so you can keep track of how many potions the player has or something.

I think that by creating simple events and triggers and very simplistic conditional operations, a lot of the requirements for writing these have gone down. There is no more need to remember textual syntax of how to call a function in lua or how to extract a variable from an array.

I think if these block diagrams are able to be kept clean and organized, as well as adding the ability to create 'templates' so you can create one small sequence to say trip a switch and flick the lights off while vibrating the controller, and then package that up and treat that as one block, the complexity can be controlled to some degree. I've seen some people abuse the system and create very complex diagrams with wires going everywhere but I think that could also happen with text scripts when people don't write nice looking, readable code.

Any thoughts on this type of system Kencho?

[Kencho]

Ah okay. It's mostly the kind of system I was talking about in the last paragraph. This abstraction level looks fine and intuitive to me. The deal is that you highly simplify things when you remove the low level operations and only allow game mechanics' actions, such as triggering an event, firing a trap, and such.

Much better this way, yes

[Raketenmann]

@Raketenmann: Thanks for the info. Having the ability to translate into XML with a copy/paste is a really good idea. Is the system that you guys have a visual one?

Yes it is and it looks like this:


As you may notice, the lately activated states flash up in red and then fade to white. This leaves a redish trail even if several states are activated one after another. If someone complains the poor size of the image: i've done it to emphasize another advantage of visual scripting. Even without reading anything you get a quick overview what is going on. And even in this size or a smaller one, the one that scripted the whole thing can have a look from the corner of his eye to know what is happening if a state flashes up. That said, it's easy to "debug" several statecharts at the sametime while playing the game.

You mentioned that they had done python scripting so what made you guys decide to move away from text and go to a GUI based approach (if that's what you did)

As I mentioned before, we are doing an adventure game. Most of the time you do things like controlling when to display which text and show which cursor etc. It often boils down to "first show this, when clicked once, show this, when clicked twice, remove that, but only if the kitty got fed before".
Statecharts fit just perfectly for that: what they can do, is react on an event ("clicked") depending on what happened before. What they are not good at: evaluate very much after the event and then decide what to do.

Based on that observations, we talked about the drawbacks of the kind of scripting our team has done before and felt that a visual approach could prevent us from many pitfalls.

But I have to admit: this kind of scripting is nothing for me as a programmer. I think, i would feel limited often, because i already think in pattern that fit better into code. But scripters maybe do not. Complex scripts lead to unmaintainable spaghettis. One may say: "Get skilled scripters!". One option. Another one is: ease scripting.

That leads to another point I want to mention: Games tend to user content more and more. At the moment we are at the level of creating assets that than behave automatically correct like in "Little Big Planet" or "Spore". But the only reason users don't edit behaviour imo is because it is to hard to grasp scripting at the moment.

TLDR:
Leave code for coders. Visual scripting is for non-coders, but there it may be the one-eyed king.

Only likes spaghettis on Bolognese or Carbonara: Raketenmann

[Praetor]

Some of the examples seem more complicated to understand...

If something is really a state-machine, and many things can be modeled as such, then a tool for state machine editing is great. I'm all for it! But when it comes to more complicated behavior then I really think the effort of developing the tool might be more than the effort it takes to introduce an artist to a simple scripting language. Often, once a non-technical person gets a slight push in this direction they become a far more valuable asset anyway. I encourage all artists I work with to attempt to understand some aspects of programming.

[Kojack]

Wouter van Oortmerssen (author of Cube and Sauerbraten) has done experimenting with visual languages (and tons of other languages, I learnt OO coding using his E language).
http://strlen.com/proglang/index.html

There was an Amiga game called Gravity, made by Image Works in 1990, which had a visual programming language built in to give orders to your squad of drone ships.
http://www.angusm.demon.co.uk/AGDB/DBA1/Gravity.html

[Spanky]

Praetor: I think having an artist able to script would be an awesome benefit and I do think that they should be able to do a small amount. Designers definitely should be able to use a simple scripting language or something. Programmers have to know how to use all of the tools in the company, even if in a very basic sense, so why doesn't anyone else? I think there are a lot of designers out there that won't/don't want to learn how to script. They will argue that this falls into the job of the programmers (from experience) and won't do it. Upper management can be a pain in this respect too if they aren't willing to listen to why they should be doing it instead of a programmer. I do think that by using scripts, a lot could be reused and changed easily but if it is something that will never be adopted by the company, no matter how hard you push, I think that this would be a good route to take.

I've been looking at Crytek lately and playing with that editor and it has a thing called FlowGraph which is fairly easy to understand. I'm not sure how it's implemented in the back-end but it seems like they could just be C++ modules or script modules that run for the given script.

One way I've been thinking about handling something like this would that each node simply represents a method or module in script/C++. This ends up being nice assuming you keep your parameter list constant because you can change the lua/whatever code that gets run easily without a recompile. If it becomes performance hungry, you can implement the method in C++ and simply have the lua script call that.

My girlfriend would be interested in hooking some stuff up using an editor at some point in a game idea we've been toying around with but she doesn't think like a coder at all so she'd be overwhelmed with even a simple logic system in text. Being able to "see" where the data goes and what happens when I think would help her out a lot. Just being able to trace a line backwards and see how you got somewhere without having to understand syntax would help her a lot.

Having dealt with designers who will never be willing to edit text files, let alone script files, I think that this is the only real alternative that allows for the fast prototyping by *anyone* in the company for different features. It's pretty easy to sit down and figure out how to drag-drop things around. Give them a quick 5 minute intro to show them how to activate one of these things and they'll come up with all sorts of stuff.

I'd agree with Raketenmann though. I'd rather program the logic if I could. I'd rather write in a scripting language so I can avoid the compile cycle if I could but for non-technical people, I think this provides a "safe", easier to use alternative. Combining your code with something like luabind/toLua++/whatever, you can easily the required functionality to write a script. It'd be nice to have some facility for a designer to create their own nodes and write the underlying script for it. Once it passes a programmer code review, it could be an officially supported node. This opens up new doors and abilities for designers who can script since they don't have to wait for a node just to try something. They could easily whip one up with the functionality that is already exposed to the script language and write a new node.

Raketenmann: Thanks for the picture. I could see that being really damn useful when you're trying to figure out what went wrong. I definitely agree on the idea that games are becoming all about content lately. Most big games now a days are requiring tons and tons of content and interaction with that content. Nobody wants to play a game where you can't interact with anything anymore (I'm talking the big title games, not things like xbox live arcade or anything). Easing the creation of these, as well as lower the time to create them is hugely important in getting something done.