Ogitor for OGRE 2.0

stealth977 · Post by **stealth977** » Thu Jan 22, 2015 8:58 pm

Hi all,

Ogitor progress stopped last year after realizing that there is not much chance to improve Ogitor performance due to restrictions of both Ogitor's engine and OGRE version it used (both were mostly single threaded and Ogitor itself had some design issues). Just before the pause, I was experimenting on a new design for Ogitor, that would make it much more efficient and easy to use both as an Editor and also a standalone engine, but again, no matter how efficient the design gets, its mostly limited to what the graphics engine can deliver, since maximum speedup gained is theoretically : speedup = 100% / ( serial_execution% + ( parallel_execution% / num_hardware_threads ) ) which means even when you have a system with infinite CPUs, max speedup gain is : 100% / serial_execution%. In other words, if half of your execution time is serial, even if you have infinite number of CPUS, the maximum speedup would be 2x....

So, I postponed all efforts till a multi-threaded OGRE version arrives and paused Ogitor development (at least my own part of development).

Finally, I see that OGRE 2.0 came a long way and improving rapidly. So, recently I started to work on next generation of Ogitor : Ogitor for OGRE 2.0. There are fundamental changes in new Ogitor:

- First of all, before, Ogitor meant both the editor itself and the engine running behind it, now we have those two separated: now Ogitor means the EDITOR ONLY, and it uses OGF as its engine.

- Second there wont be much change to Ogitor (the editor), it will look and function mostly same, the real change between old and new is the engines: Ogitor's Engine (OLD) vs OGF (NEW)

- OGF (The engine framework behind Ogitor) is now Object/Component based, where old Ogitor was object inheritance based

- OGF is fully multi-threaded, all objects and their components (systemobjects) are running parallel where ever possible and making use of all hardware threads available, using micro-tasks, where old Ogitor's Engine was single-threaded

- OGF does not include any editing specific code or burden (has some helper functionality which is not used unless asked by the program using the framework) where old Ogitor's engine had all kinds of editing burden as extra data structures, safeguard codes executed which have no use out of an editor etc. So, once OGF is not used in an editor environment, it runs full speed without any extra burden both memory and execution wise.

- OGF can run standalone, meaning : By using a loaderstub to ignite the OGF, it will be running the designed scene in your own program, behaving as your game engine. Of course old Ogitor had no such functionality...

- Ogitor using OGF will be saving scenes in its native scene format, but just like Ogitor, it will be possible to export to any format you like by supplying export plugins...

- OGF has CPU profiling information, like how many threads are being used, the workload on each thread, CPU consumption of each System (Graphics, Physics, Input, AI, Scripting etc), and probably will supply other profiling informations such as GPU stats...

- ..............

So, how much of it is already implemented? To be honest only the CORE FUNCTIONALITY is fully implemented at the moment, there are some test Systems implemented as well, like InputSystem, GeometrySystem, GraphicsSystem with few components like Mesh, Camera, Light, Terrain etc. Also the editor of OGF has a lot of work to do as well, but it is improving everyday...

Why did I make this post other than announcing that a new Ogitor is in progress? Well, because I will need help with a specific issue: I do not have a linux machine to test the code...

Anyone who is interested in helping, and has good knowledge of :

- CMake
- Mutexes and Semaphores, on both linux and windows

please send me a PM so we can talk about how you can contribute.

Cheers,
Ismail TARIM

hydexon · Post by **hydexon** » Thu Jan 22, 2015 9:17 pm

It's very interesant your proposal for Ogitor, since Ogitor has a quite long time of inactivity.

I was trying to make my own editor based in Ogitor (because i have very specific needs), but the main problem was really hard to track code like the Move, Scale and Rotate objects, and since the code structure of Ogitor was very different of the mine.

At least you got the Core functionality finished, which is a good thing,and the new architecture of Ogitor and OGF looks promising, the editor always be Qt, right?

al2950 · Post by **al2950** » Thu Jan 22, 2015 10:10 pm

I have to say this all sounds very exciting, especially separating the editor from the engine as I was struggling with this. I am neither a CMAKE expert nor a advent linux user, although I do use both, but would be very interested in helping out in any way if you needed any help.

stealth977 · Post by **stealth977** » Thu Jan 22, 2015 10:36 pm

Well as I said, I do not have a linux machine to test how CORE runs on linux, although the code is crossplatform, following problems may exist:

- Linux compile have minor header or case sensitivity problems or path problems
- The Task Scheduler uses platform specific .h and .cpp files, one written for each platform, although it is possible to write one code for all platforms (for example by using boost) current implementation is much more efficient on Windows platform since it uses some useful API Calls like WaitForMultipleObjects which do not have a linux version. So, the linux implementation uses a little different path and i could not test it on a linux machine

Still I do not think it would take too much time to fix those problems if they do exist. But fixing requires good knowledge of CMake and threads/mutexes/semaphores on both platforms (i say both platforms because windows code is perfect and can be used to understand the system if there is a problem with linux version)

Some other notes:

- Current code has boost, Qt 5.x and Ogre 1.9 dependencies

The reason it still uses Ogre 1.9 is to make sure the framework is stable, once Ogre 2.0 is out with enough stability and features it is very easy to switch to since OGF uses OGRE in only 1 independent module (GraphicsSystem.dll) and thus the code to be changed will be minimal and isolated from the rest of the framework...

frostbyte · Post by **frostbyte** » Thu Jan 22, 2015 11:21 pm

Well as I said, I do not have a linux machine to test how CORE runs on linux

well, you can have one in 5 minutes( or less ), just take the live version and install it on usb key/drive
just make sure you enable persistent-storage on the installer, and then you can treat it almost like regular linux( eg-install drivers and stuff... )
works like a charm

for windows...( this are the popular ones... )
http://www.ubuntu.com/download/desktop/ ... on-windows
http://www.linuxliveusb.com/en/home
for mac...https://sevenbits.github.io/Mac-Linux-USB-Loader/

this is good news, funny enough, we just had a discussion( today ) about how ogitors code is problematic to act as base for a game engine...
code is already on the loose, i like that...

Post by **jacmoe** » Thu Jan 22, 2015 11:42 pm

The problem with Ogitor has always been that a new version is always in progress.
Making it impossible to get anything stable up and running.

I've been meaning to get back into Ogitor programming for quite some time now, but the main problem is that it is in dire need of refactoring.
The editor and the core is just too intertwined, so if this new new Ogitor - which (unfortunately) hasn't been developed in the public - doesn't improve that, I am not sure if I'm going to be excited.
I needs to be separated and the editor needs to be using QT: QOBJECT, signals/slots, etc. And not suffering from NIH syndrome.

I've actually been working on a new site for Ogitor, which is leaner, more modern and easier to maintain, so it's weird that Stealth977 decided to pop out of the woodwork now.

I can't help with Linux because I don't know about mutexes and semaphores - and I don't want to know about them.
I don't run it these days, except in VMs.
May I suggest looking into Vagrant?
I am using that to quickly fire up dev machines for web work, and it could be used for Ogre3d work as well.

frostbyte · Post by **frostbyte** » Thu Jan 22, 2015 11:59 pm

May I suggest looking into Vagrant?

for now, mostly all vm gpu drivers are horibble( except with passthrough enabled gpu/vm )
live usb install gives you full native performance...very easy to set up...

funny enough, we just had a discussion( today ) about how ogitors code is problematic to act as base for a game engine...

I've actually been working on a new site for Ogitor, which is leaner, more modern and easier to maintain, so it's weird that Stealth977 decided to pop out of the woodwork now.

some magic is happning...mighty OGRE is starting to wake up...

Thyrion · Post by **Thyrion** » Fri Jan 23, 2015 9:34 pm

wow. out from nowhere.

instantly inspected the source.

i'am currently trying to move ogre into a thread, too. I choosed a lockless queue: http://www.ogre3d.org/forums/viewtopic.php?f=1&t=82349.
My first question as i saw your code, was... damn, did i do it wrong

I push complete std::function into a queue:
pros:
- no event id lists needed
- no switch case needed
- no locks needed
- it's easy to write thread code everywhere

cons:
- problems with parameter object lifetimes (need to make a copy of each parameter, with the exception of the referenced instance)
- i think higher performance costs vs event ids

Another question arises, why you choosed a thread pool?
Just because of scalability, or performance?
Now i'am wondering what price do i have to pay performance based with std::function.

I am curious, if this makes the use of the ogitor source easier or harder (threading can be the source of long debug sessions (realy needed for an editor?)) ...
First hurdle for me, will be to get the source compiled (i tried it hard and long with the current ogitor version, but gave up annoyed)

.

Independent from that, i'm glad to see that ogitor is still alive. *cheer on*

btw: i liked your work with ofs.

stealth977 · Post by **stealth977** » Fri Jan 23, 2015 10:23 pm

Thyrion I hope you cloned OGF code and not ogitor code, because new code is in OGF repository. Soon we will make a new repository on bitbucket for it once we cleanup some old remains.

And about your questions:

There is no one single way of multi-threading. You choose whatever suits you well. I personally do not like using function pointers in multi-thread environments, message queues seem more predictable to me. As for the performance, the messages are queued in thread local lists and then combined and resolved (btw i do store message params as well, it is not just "someone moved" message but it is more like "someone moved" and "he/she moved 2 inches" type messages) Since the messages are handled in parallel (the message list is divided in small chunks and queued as jobs in the thread pool) I currently do not have much concern about their speed.

OGF's way of threading is like:
Every Frame:
1 - Messages from previous frame are handled in parallel on as many threads as there are or needed
2 - Each System OGF uses have a SystemTask that is queued like GeometrySystemTask, GraphicsSystemTask, InputSystemTask, PhysicsSystemTask etc....
3 - Each Task can also spawn as many children as they like
4 - We wait for all the tasks to complete and switch to next scene

as little as possible is handled in serial to improve concurrency

N0vember · Post by **N0vember** » Fri Jan 23, 2015 10:48 pm

Thyrion wrote:i'am currently trying to move ogre into a thread, too. I choosed a lockless queue: http://www.ogre3d.org/forums/viewtopic.php?f=1&t=82349.
My first question as i saw your code, was... damn, did i do it wrong
I push complete std::function into a queue:
pros:
- no event id lists needed
- no switch case needed
- no locks needed
- it's easy to write thread code everywhere
cons:
- problems with parameter object lifetimes (need to make a copy of each parameter, with the exception of the referenced instance)
- i think higher performance costs vs event ids

Threading has always been the last item on my roadmap, so I never seriously implemented it, but I always thought I'd go the lockless queue of tasks, too. With function pointers also.
I believe it's a very common approach and I don't believe the cons are as bad as you think. I don't really see how a function pointer can be so bad performance-wise. The fact that it is lockless should outweigh any other overhead. It's just more complicated to find the right architecture I believe, but the result will be worth it.

stealth977 · Post by **stealth977** » Fri Jan 23, 2015 10:53 pm

I do hope you know what you mean by LOCKLESS

The name is kinda misleading since LOCKLESS does not mean it doesnt stall the CPUs, actually it does stall all threads at the same time though not so much as a lock would stall. Thats why I use thread local queues, they dont need locks and they do not stall the threads in any way, only downside is that you need to combine them afterwards or can iterate them 1 by one as well.

Post by **jacmoe** » Fri Jan 23, 2015 11:02 pm

Just wanted to chip in and say that I am splendidly excited about how and where Ogitor is heading.

I have felt that the code was in great need of refactoring for a long time, but I really couldn't decide on how to go about it all.
I wanted it to be more like the code from the RealXtend project, but then Stealth977 pops out of the woodwork with something much better.

Post by **jacmoe** » Fri Jan 23, 2015 11:27 pm

I want Ogitor to have full and official support for OpenGEX.

Dotscene is dead. It never was a standard, nor widespread in use. For the same reason.
Now, it seems like all the kool kids are raving about OpenGEX. And it actually seems like other engines (game or 3d) are at least seriously contemplating supporting it. AssImp is working on it as well, which is handy.

If anyone has any code for OpenGEX and Ogre: give it.

Post by **Kojack** » Sat Jan 24, 2015 6:19 am

OpenGEX is quite interesting.
Most of it seems to match up with things Ogre can do.

The main problem I see is that OpenGEX is not just a scene format. It contains everything: scene graph, animations, meshes, skeletons, etc. It takes the place of .mesh, .skeleton and .material too, but in text format instead of Ogre's binary formats (except for .material of course, that's text). That's going to be less efficient. Probably better than collada though.

I can't see any support in the spec for multiple vertex buffers in a mesh. Ogre requires that for morph animation, having position data in a different buffer to all the other vertex attributes. That could be done at loading time of course, but other combinations can't be predicted.

Transforms are made from concatenations of 4x4 matrices. Multiple transforms of any type (translation, scale, rotation) can appear in a single node. This means skew can be introduced (if you did a rotation then non uniform scale, instead of non uniform scale then rotation), which Ogre can't handle.

Materials have no shader support and only a small subset of Ogre's material ability. Basically they list colours and textures (things like diffuse colour, diffuse texture, specular colour, specular texture, etc).

So some translation will need to be done when loading, which is moving some processing from the exporter to Ogre itself. Unless we used OpenGEX as an intermediate format.

Still looks like a cool thing to have, but probably shouldn't entirely replace what we have now.

Post by **jacmoe** » Sat Jan 24, 2015 6:29 am

Normally you are as sharp as a knife Kojack

However, OpenGEX is just like the xml files Ogre uses for import/export - ie interchange.
I would stop using Ogre if OpenGEX was going to be the main format.

I would really love to see Ogre get rid of xml altogether in favor of json.

Rapidjson or libjson are two very good c++ libs.

That doesn't mean, of course, that the binary format is not what's used.

If the format is lacking, then I guess it's a good time to contact the C4 guy so that it can be added.

Post by **Kojack** » Sat Jan 24, 2015 7:12 am

If Ogitor used OpenGEX, Ogre doesn't, and dotscene is dead, then how do we get scenes from ogitor to ogre?

Post by **jacmoe** » Sat Jan 24, 2015 7:22 am

Because Wolfmanfx said that he would make sure that Ogre supported it.

Ogre did not ever officially support .scene - not even Pjcast managed to create a ddl for .scene, so what we ended up with was (and still is) hundreds of slightly different dotscene loaders/exporters.
OpenGEX is different because it follows an official spec, and - more importantly: used outside of the Ogre pen.
AssImp is getting it, MODO has written an exporter/importer for it. Blender, Maya and MAX (or what the name is nowadays) has it ..

Ogitor has tried to maintain a .scene exporter/importer, but it's hopeless because it has no format.

Of course, Ogitor also has it's own format, filesystem based and binary, so it's just for generic import/export. Like grabbing a scene from Blender/Maya or exporting one for a game engine/project.

<edit>
It probably need to have application specific extensions to it. Like: paging and terrain, materials, etc. in relation to Ogre. Or scripts in relation to a custom 'engine'.
Since it's just json, that shouldn't be a problem, as long as other import scripts just ignores the custom bits.

I definitely think it's worth pursuing.

Post by **jacmoe** » Sat Jan 24, 2015 7:41 am

It has been painful to witness at least one failed GSoC project trying to create an Ogre Collada importer/exporter, and Xavier almost burned out on a similar (yet failed) endeavor, and then there's at least one more OgreCollada project in the ogreaddons graveyard. We ended up with 4 (?) non-functional Collada plugins.

So, a lot of talented people have obviously believed in the idea of a capable interchange/exchange format.

Post by **jacmoe** » Sat Jan 24, 2015 7:49 am

Kojack wrote:If Ogitor used OpenGEX, Ogre doesn't, and dotscene is dead, then how do we get scenes from ogitor to ogre?

Oh, I forgot two of the most obvious answers:
1) The dotscene exporter is just an example of how one would go about making one.
2) There is an OFS plugin for Ogre so that people can load it and use it in their own projects.
However, the architecture currently sucks.

And that's what this topic is about, basically.

Post by **jacmoe** » Sat Jan 24, 2015 10:21 am

Kojack wrote:Probably better than collada though.

Yes, indeed:

OpenGEX page wrote:The OpenGEX format was created because Collada, the open standard that we all hoped would provide a well-supported asset exchange format, has proven to be an unreliable mess. The most common source of problems has been the poor quality of the Collada export plugins available for software such as 3D Studio Max and Maya, and we attribute this to Collada’s over-engineered design and its mutating under-specified format.

Thyrion · Post by **Thyrion** » Sat Jan 24, 2015 1:32 pm

jacmoe wrote:I want Ogitor to have full and official support for OpenGEX.

Dotscene is dead. It never was a standard, nor widespread in use. For the same reason.
Now, it seems like all the kool kids are raving about OpenGEX. And it actually seems like other engines (game or 3d) are at least seriously contemplating supporting it. AssImp is working on it as well, which is handy.

If anyone has any code for OpenGEX and Ogre: give it.

just to throw in my thoughts, too

i think OpenGEX would be nice as exchange format between 3d programms, but should be converted into the ogre formats like now the xml files.
dark_sylinc already said he probably will write the code and use it?(somewhere in the "what happend with ogre" thread)

Currently i'm rewriting my resources(from xml) to flatbuffers (https://github.com/google/flatbuffers). What about that?
If i look into the OgreScriptTranslator.cpp beast, with 8k loc it looks like a good candidate,too...

It's super fast and easy to read the flatbuffers from game side.
What i don't like about it, is the creation of the flatbuffer data (currently sucks from C# side, but from c++ side you could use json data i suppose

).

@jacmoe btw welcome back mister cook!

Post by **dark_sylinc** » Sat Jan 24, 2015 3:48 pm

Aw yeaaaahhhhhhh!!!!!! Jacmoe is back!!! We missed you!!

Thyrion wrote:i think OpenGEX would be nice as exchange format between 3d programms, but should be converted into the ogre formats like now the xml files.
dark_sylinc already said he probably will write the code and use it?(somewhere in the "what happend with ogre" thread)

My idea is to support ogex natively, just like the XML format. Both formats are almost identical, except one is xml and the other json-like. In other words, one is human-readable, the other is not.
OpenGEX looks like a very well designed format, and has everything our XML format has and more (except for animations referencing other animations for binding pose conversion, but ogex is flexible for us to include that data anyways).
Many guys tried to do the same with Collada, but Collada sucks.

Klaim · Post by **Klaim** » Sat Jan 24, 2015 4:08 pm

Also, it gets a lot of traction currently so if something is missing it looks totally possible for the Ogre team to suggest an addition to the format if it helps Ogre implementation and is general enough to be useful for others.

Post by **jacmoe** » Sat Jan 24, 2015 8:55 pm

It would be totally cool if Ogre used json instead of xml for intermediaries.
XML as a format is really overkill imo.
And there is actually binary json if someone wanted it. And it ties well with the web.

I know that the Ogre format in itself has become a standard of it's own, but I think that if Ogre officially decided to go with OpenGEX, that would increase the traction even further.

Btw: thanks for the welcome, guys. Have been missing you too. <3

Post by **dark_sylinc** » Sat Jan 24, 2015 10:11 pm

jacmoe wrote:It would be totally cool if Ogre used json instead of xml for intermediaries.
XML as a format is really overkill imo.
And there is actually binary json if someone wanted it. And it ties well with the web.

Since you weren't in the loop I won't smack you for not knowing

OpenGEX is in OpenDDL; which is like JSON, but not exactly JSON; both developed by Eric Lengyel (the C4 engine main dev) with the help of crowdfunding.
ODDL is very much like JSON, but main differences are: no ":", vector sizes can be declared (i.e. float[3] for a Vector3), and floating point can be in hex (yes!!!).
Here's an example.

Ogre Forums

Ogitor for OGRE 2.0

Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0

Re: Ogitor for OGRE 2.0