[SoC 2009 - Accepted] Improve Ogre's Compositor Framework

Threads related to Google Summer of Code
Locked
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

[SoC 2009 - Accepted] Improve Ogre's Compositor Framework

Post by Noman »

Hello all

Project accepted! Wiki page

Abstract:

Ogre's Post Processing (Compositors) framework, while being quite powerful, is still lacking some features. Numerous users have said that they found themselves implementing their own custom post processing solution because they couldn't do what they wanted with the compositor framework.
This project's aim is to address this problem, adding options for more advanced chain setups and more flexible composition options, enabling advanced rendering techniques like deferred rendering to be easier to set up, more flexible and more reusable.

Project Proposal

Ogre's Post Processing (Compositors) framework, while being quite powerful, is still lacking some features. Numerous users have said that they found themselves implementing their own custom post processing solution because they couldn't do what they wanted with the compositor framework.

Before I start suggesting the changes themselves, I will present two "wishful use cases" that could, theoretically, be solved by a post processing framework:

* Deffered shading post processing support. Consider this case : someone set up a deferred shading system using the compositor framework. Afterwards, they would like to add SSAO to the scene, like nullsquared's demo : viewtopic.php?f=11&t=47927. SSAO requires the depth and normal passes, so the compositor has scene passes that calculate these themselves. But the deferred shading system already created those textures! We would like to be able to use them from the next compositors in the chain.
* Depth based effects. Even when not under deferred shading, many rendering techniques require simliar full-scene renders (like depth). We would like to define a "utility compositor" that does the depth pass of the scene, and then the other compositors would be able to use its output.

Gameplan

The features that will be added to the compositor framework, desgined to make it more modular are :

* Inter-compositor communication : A compositor should have access to the other compositors' local textures. (It has been suggested to be able to use named texture units, but that is not chain-specific).
* Inter-compositor dependencies : A compositor should be able to trigger the enabling of another compositor, if it needs its result as input.
* Auto ordering of compositors : Slightly related to the previous addition, if a compositor depends on another one, they should be ordered accordingly. The chain can be auto-ordered to make sure that each compositor runs after its dependencies (can also find cyclic dependencies).
* Scriptable chains : A compositor chain should be configurable by script, rather than by hard coding it. Chains will be a bit longer (since utility compositors will be introduced), and I think its healthy to separate the chain definition from the code. (Applying the same chain to two viewports etc).
* Allow auto-coupling of compositors and CompositorListener code - If someone created a compositor that needs a listener to set up matrices (example use case : nullsquared's SSAO), we would like to be able to automatically bind the listener whenever the compositor is added to the viewport, without code interference. I have already done something simliar with the CompositorLogic framework ( http://www.ogre3d.org/wiki/index.php/Co ... _Framework ), but if we are changing the way compositors work, it shouldn't be an external package. Maybe part of the compositor script will be defining a binded plugin.
* Allow none-competing material scheme handlers - Some compositors need to render the scene differently. (This is already supported using schemes). Not all materials have that scheme defined, so the compositor can use code to prepare the correct material. Currently, the MaterialManager defines a MaterialListener class that can do just that (handleSchemeNotFound callback and work from there). However, there can currently only be one listener. So, if two different compositors render the scene with different schemes, they both need to be the single MaterialManager listener. In order to solve this, MaterialManager will be changed to allow single scheme binding, so that each compositor listener will be able to receive its relevant handleSchemeNotFound calls without competing with the others.

Schedule

The project will consist of many stages. Each part is designed not only to help the final goal of the project, but to be a self contained building block that Ogre users will be able to use. The final product will use them all, but like any infrastructure, they need to be modular.

1) (May 23 - June 5)Inter-compositor communication. Compositors will be able to define textures as publicly available to other compositors to use, and they will also able to reference textures that other compositors shared. This will include either sanity checking or auto fixing of chains (or both), so that only legal combinations are created.

Testcase 1) (June 5-10) Separation of the Deferred Shading compositors to two stages - one that generates the GBuffer, and one that uses it to create the final picture. Add another optional compositor that uses the GBuffer to create an SSAO buffer and apply that to the scene.

2) (June 11-18) Add auto-coupling of compositors and their related code in scripts - some compositors require code to setup their rendering properly. We want the compositors to be able to define a plugin name (or similar) that needs to be loaded for them to operate.

3) (June 18- July 25) Modify the MaterialManager::Listener interface to allow none-competing scheme handlers, so that different handlers for different schemes will be able to co-exist and receive their respective callbacks without using a 3rd manager.

Testcase 2+3) (July 25 - July 1) Create a depth-based composition technique that automatically generates the depth rendering setup through the "Depth" material scheme. This handler will be loaded automatically when the compositor's binded plugin is loaded.

4) Scriptable chains (July 2 - July 8): It should be possible to define compositor chains in scripts. This will make it easier to apply predefined sets of compositors to many viewports.

5) Add a render_lights composition pass (July 8 - July 29) (parallel to render_scene, render_quad etc). This will allow the compositor to render its own sequence per-light. Design is not complete yet, so extra time is added to decide on the details of this one.

Testcase 5) (July 29 - August 10) Add texture shadow support to the Deferred Shading demo. The shadow maps will probably (design is not yet final) be generated during the render_lights sequence (or they will be able to be pulled in somehow), and then casted on the scene.

Uber-testcase) (As much time as I have left) Complete the refactoring of the Deferred Shading demo by making it completely script based. The demo will just load the scene and apply a compositor chain script to it, and the rest of the job will be done by the compositor & associated plugin.

Future Extensions

Hopefully the final testcase will be done by the summer, but if not, it is an excellent point to continue from. Adding more flexibility to the deferred shading compositor chain (alpha blended objects for example) will make it even better.

In addition to that, more composition technique options have already been suggested, but are not part of this project because of time constraints. We can always do more from that list :)

Why You're The Person For This Project

I have been in the Ogre community for a few years now, but I have been most active in the last 2 years. Besides participating in last year's summer of code and adding geometry shader support to ogre, I have also submitted patches and made a few other minor contributions as well ( http://www.ogre3d.org/wiki/index.php/Co ... _Framework , viewtopic.php?f=5&t=47172 ). I want to make this year's SoC project more ambitious than last years - I have more time this year and more knowledge about Ogre and 3D rendering in general.

(Application ends here)

As always, comments, suggestions and whatever in this topic.
Last edited by Noman on Thu Apr 23, 2009 8:20 pm, edited 5 times in total.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: Improve Ogre's Compositor Framework

Post by tuan kuranes »

Nice !
I clearly see the "compositor local texture sharing/reuse" need (even if ogl 3.0 and DX10 finally give access the read depth buffer finally, still might be useful for "special" depth buffer). and definitly like the "compositorchain" scriptable idea (might leads to "editors/chain visualization" tool ?)

some remarks:

- What about some more constraint/error checking so that compositor building would be less complex (bad input, output, circular dependencies, etc..) ?

- Deffered shading might be very hard to make "pluggable" imho

- Not sure it it fits in the "utility compositor", but anyway that would be a very nice : a pluggable "compositor cache" based on "shader reprojection cache" papers. ( papers - cached ssao demo with source). That "utility compositor" would be of use in many other compositor and therefore added in a simple way to many effects like blur, depth field, shadows, ssao, ... in fact any complex reusable shader results from previous frame might gain large FPS from that.

- other often used compositor could use a simplified syntax, becoming predefined or "utility compositor" ? (Depth, Blur, Bilateral filter (blur with limits, useful for nice upscaling), hdr/tonemapping)
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by sinbad »

Bear in mind that there's already an ability to share texture resources to a degree in the compositor, using the 'shared' keyword on texture defs (see the Cthugha notes). Currently this is about saving memory and avoiding recreating textures when switching techniques dynamically (think temporarily enabling fullscreen effects). What it doesn't explicitly let you do is directly reference the results of previous compositors that are not the final output though.

Making deferred shading easier - I have had this one on my list for a while actually, but haven't found the time for it, so I'd like to see it get done, but it requires more than compositor support. It needs some support for separating forward and deferred rendering passes configurably - for example it makes sense to perform global lighting like ambient, directional lights and static lighting in the forward pass, saving the 'light independent' results in a separate gbuffer channel, as well as light-dependent diffuse/normal/depth/specular gbuffer areas). I requires integrating light iteration into post-passes (rendering light geometry, optionally using stencilling), and allowing the option to render shadow textures for a given light on demand, re-using the same textures, effectively making an infinite number of shadow casting lights possible (depending on GPU speed of course). Transparencies need considering - usually you want to forward render those with regular lighting afterwards, although sometimes they can be rendered into the 'light independent' forward render with the static lighting, if lighting on the transparencies is not complex. So, there's a lot of things to do here! It's of course possible to build a system that does this already, but I'd love to see it made it easier to access. But, it's definitely quite a big project - it could use the features you've described in the compositor system too, although it would probably need plenty of its own too and I'm not sure both are doable in the time.
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Noman »

Thanks for the comments!
What about some more constraint/error checking so that compositor building would be less complex (bad input, output, circular dependencies, etc..) ?
I do have some of that currently in mind. Currently I only had sanity checks that were related to my additions in mind, but general improvements always have a place as well. Circular dependencies will definitely be checked (that was already in plan), don't know about the others yet. Will probably decide when I build the timeline for the project.
Not sure it it fits in the "utility compositor", but anyway that would be a very nice : a pluggable "compositor cache" based on "shader reprojection cache" papers.
Hm. Interesting. The primary focus of the project is to improve the framework to support such ideas, not necessarily introduce a utility compositor library. Of course, some samples will be needed. I prefer providing simple techniques as samples, I believe they make the connection between the framework and the user code easier to understand. I didn't read the papers yet, I'll definitely read them before I choose target samples.
other often used compositor could use a simplified syntax, becoming predefined or "utility compositor" ? (Depth, Blur, Bilateral filter (blur with limits, useful for nice upscaling), hdr/tonemapping)
I was thinking something along the lines of this (lets say, SSAO compositor):

Code: Select all

target rt
{
    // Start with clear texture
    input none
    // SSAO pass
    pass render_quad
    {
        // Renders a fullscreen quad with a material
        material Ogre/SSAOQuad
        // The MRT target containing the normals
        input 0 DeferredPass/output_tex 0
        // The MRT target containing the depth
        input 1 DeferredPass/output_tex 1
    }
}
If the DeferredPass compositor would declare a texture like

Code: Select all

texture output_tex target_width target_height PF_A8R8G8B8 PF_A8R8G8B8 PF_A8R8G8B8 PF_A8R8G8B8 public
Meaning a 4 MRT texture that is 'public' (available to the other compositors).

Ideally, once that compositor is enabled, if the DeferredPass is not connected to the chain, it will be. It makes referencing utility compositors very straightforward and easy (just need to use the texture in a target pass). Perhaps the syntax should be more direct and not a string convention, but those are the minor details.
Deffered shading might be very hard to make "pluggable" imho
True, and this is pretty much what sinbad said as well. I refrained from saying that this project will make deferred shading a drag-and-drop solution in Ogre. That is indeed, somewhere between very hard and impossible. I do think that there are numerous improvements that can make the task easier, and make some of the components required for deferred shading more reusable.
I did think about how I would do the actual lighting sequence of deferred rendering in compositors, and maybe a 'render_lights' target pass (parallel to render_scene / render_quad). That could also be the place where per-light shadow maps could be calculated.
But then, would Ogre have a standard way of creating the geometry for those lights? Would the user have to write code to create the geometry? Theoretically a geometry shader could do this, but that would push the hardware requirements too high. Perhaps the compositor-binded plugin could do some of the work, but again that hurts the ease of use and reusability, unless 'standard' options are introduced as well.

This might change the compositor framework from being a 'post-processing' framework to a 'custom render sequence' framework, as it does more than process the scene. Its more of a conceptual change than one that requires code changes, but I think it might be a step in the right direction for Ogre. The compositor framework is already more than a post processing framework, but It does need some more flexibility options in the general part.

I agree that this turns the SoC project into a more ambitious one, but I am prepared. I also think that with proper planning, the project can be a set of relatively disconnected parts, that when all of them are finished, deferred shading (and similar techniques) will be a much easier task, but each one of them is also useful without the rest.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by sinbad »

Circular dependencies will definitely be checked (that was already in plan), don't know about the others yet. Will probably decide when I build the timeline for the project.
texture output_tex target_width target_height PF_A8R8G8B8 PF_A8R8G8B8 PF_A8R8G8B8 PF_A8R8G8B8 public
Again, be sure to check my trunk enhancements because there's already some functionality like this. For example you can use the keyword 'shared' - this makes the surface available for re-use by any other compositor (even in different chains / targets), although currently it doesn't make any assertions about the content of it. The routine that looks for shared surfaces to re-use already checks for dependencies within the same chain on this surface to avoid overwriting of content too early.

The scope of 'public' also needs to be defined - is this public within the same compositor chain, or across all chains for example? Both options are probably useful in different contexts - sometimes you might want to use a surface as input for many different targets (of the same size, or of different sizes; think downsampling the depth buffer to improve texture cache performance for use with soft particles and other effects), sometimes you might want a surface to be available to all stages of processing on this one target, but that if you had more than one target with that compositor, that the surface content is unique to each one. You might then combine this with my 'shared' functionality - ie that each compositor chain has it's own unique version of the surface locally, but that the same physical surface is shared between them all and the contents regenerated for each one - saving memory but keeping content local to the chain.

I think you're right - the compositor framework has already started to sneak in 'custom render sequence' functionality, and it makes sense to extend this further. I also agree that even if you don't manage to complete everything required to implement an advanced deferred renderer entirely within script, the components that get built will probably end up underpinning that final solution anyway. Of course, it may well be that the deferred renderer is the best test for this functionality anyway so maybe it makes sense to make that the target, but I think it's fair to say that not all components of it would necessarily need to be completed to be useful.
cloud
Gremlin
Posts: 196
Joined: Tue Aug 08, 2006 6:45 pm
x 14

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by cloud »

Something I've wanted several times is support for 3D render targets. I couldn't see this being possible in the current framework, although I may be way wrong. I know there are several ways to do it e.g. instancing a quad and using INSTANCE_ID in shader or for ogre to sort of do rendering to each slice, but to do it in code, combine and make changes becomes somewhat messy.. at least in my code it is.
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Noman »

@sinbad
I know about the 'shared' functionality, and it may make the 'public' (or whatever it will be called eventually) implementation easier. But in terms of the interface between ogre and the user, they are two different options - one is solely about performance, and the other is about modularity.

When I was talking about sharing between compositors, I was talking about sharing between the chain, not globally. I'll have to think about the possible scopes of sharing. Per-chain is the most obvious one, but there are other use cases as well. If there are two viewports rendering the same camera, they should be able to share the G-Buffer (or similar) between them. About completely global sharing, the only use case i see is for utility textures used with the 'only_once' option. Perhaps a 'static' option should be introduced for that. Per chain/camera/global options?

@cloud
I don't know much about 3D render targets (and the status of Ogre's support for them currently). I didn't plan to extend Ogre's driver interface (and driver implementations) in this project, and if its not currently supported in code, I don't know if I'll be able to get to add support for it in the compositor scripts / API. Could you elaborate a bit more on that?
cloud
Gremlin
Posts: 196
Joined: Tue Aug 08, 2006 6:45 pm
x 14

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by cloud »

Its supported in code e.g.

Code: Select all

		
GpuProgramParametersSharedPtr vparams = material->getTechnique(0)->getPass(0)->getVertexProgramParameters();

		Ogre::uint depth = mVolume->getDepth();
		for(int i=0; i<depth; i++)
		{
			vparams->setNamedConstant("slice", (float) i);

			// get the render target for the slice
			Ogre::RenderTarget* rt = mVolume->getBuffer()->getRenderTarget(i);

			Ogre::Viewport* vp = rt->addViewport(mCamera);
			vp->setClearEveryFrame(false);
			vp->setOverlaysEnabled(false);

			// write the slice to a buffer texture (using a compositor)
			Ogre::CompositorManager::getSingleton().addCompositor(vp, compositorName);
			Ogre::CompositorManager::getSingleton().setCompositorEnabled(vp, compositorName, true);
			rt->update();
			Ogre::CompositorManager::getSingleton().removeCompositor(vp, compositorName);
			rt->removeAllViewports();
		}
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Noman »

I see.
OK, I pretty much understand what you want and what needs to be done. It sounds like a possible job. Could you describe a potential composition technique that could use this option? (I don't want to add features without showcasing them because it just bloats the code base, so I want good showcase sample of this)

Thanks!
cloud
Gremlin
Posts: 196
Joined: Tue Aug 08, 2006 6:45 pm
x 14

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by cloud »

My use for it can be seen here http://www.ogre3d.org/phpBB2/viewtopic. ... f909a62890 see the ogrehead coming out of the terrain, the ogrehead is in one render volume the denisty function is created using that as an input.

For showcasing maybe.. volumetric fog, I'm sure there are lots of good papers that use render volumes for this (and I've not read any of them) .. But perhaps something like a render volume where you use camera position add a time based displacement, and use uvw to get a world space position in the pixel shader then output a spectral perlin noise (0-1 fog density) then a depthmap of scene, then a quad pass to cast rays into the scene and somehow figure out the total amout of fog and combine that with the scene.

I guess it would be better to have something that combined 2 or more render volumes (and obviously some actual code), I'll certainly keep a lookout for this kinda thing and if it happens and I find something good I'll let you know.
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Praetor »

Yeah, volume clouds and fog could definitely use rendering to volume textures. But honestly, I'd say this falls outside the scope of what you should do for the summer. I'd be excited about you working on the compositor framework stuff, and leave adding new texture target types like volume textures for later.
Game Development, Engine Development, Porting
http://www.darkwindmedia.com
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Noman »

I'll probably write a full version of my application and submit it this weekend..
I'll have a look at some samples to see the complexity.

In terms of adding support to the framework, it doesn't look like a very hard task, but the sample that comes along with it might be a big bottleneck in the process.
I'll decide when I plan the schedule in the application, but I have a feeling that unless I find a sample that is simple enough for me to feel good with the timeline estimation for it, I might pass on this option.

Stay tuned...

[Edit]
Submitted first version of application. 3D Targets currently out, sorry :(
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by tuan kuranes »

Does that conform to scope :

- Referencing "utility" compositor RTT result (Depth, Final, Blur) would be possible by script (obviously using some "viewport naming/number" linking) in material ? meaning, not having to add "programmaticaly/manually" Utility compositor RTT result name as each material pass texture unit name perhaps using some texture unit state new field :

Code: Select all

texture_composititor compositor_name viewport_name
- On the same idea, adding "in material pass" compositor referencing/linking : being able to use result of last pass render as an RTT input and label compositor resulting RTT so that it can be used in following pass ? "instanced" if many material using this exists ? All by script ?
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Noman »

I don't think I fully understood your last post...

First idea : You are talking about referencing 'public' compositor textures from .material scripts? I'm not sure about this... Compositor and Material setups have been pretty much separate from one another until now, and I don't see a reason to change that. The compositor can change the texture units of the materials that it uses in the render_quad pass, and thats all it takes, IMO. My earlier wishful code example of what a render_quad pass looks like (SSAO) shows how I plan for this to be possible. Is there any reason for a different interface?

Second idea : You mean having RTT passes in standard materials and not in compositors? I'm really not sure I understand what you suggested... Can you try to write a 'wishful script' of a technique that you could achieve using this?
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by tuan kuranes »

Compositor and Material setups have been pretty much separate from one another until now, and I don't see a reason to change that
On the contrary, it's the current major flaw of the system, sort of limiting compositor to "Fullscreen Effects" only.
For instance, my ideas are about being able to do gpu gems 3 human skin material pipeline, and finding a way to achieve that using materials and compositor only (no C++ code).
Complex Materials (hair, skin, raytraced, etc..) now have many passes that needs "post-processing" as given by compositor.
Is there any reason for a different interface?
It's the other side of the idea, when compositor results are used in material.
Think of current "shadow_texture" keyword in materials. (shadow is sort of a "compositor result")

Say you render a Depth Buffer once using a compositor, and want to use that in all materials in the scene.
You attach your compositor to a viewport, naming both using a special name, then in all material, you can reference the Depth Buffer Texture result using the syntax posted above ?

Code: Select all

material 
{
  technique 
  {
      pass 
     {
           // here you add some shader using depth buffer (say Early-Z)
         
           texture_unit
           {
                texture_compositor Depth DephtViewportMain
           }  
           texture_unit
           {
                texture_name diffuse.dds
           }
           ....
     }
}
Second idea
it's about using compositor in material without any C++ code.

Code: Select all

material human_skin
{
  technique 
  {
      pass diffuse
     {
      // some standard diffuse pass
     }
    pass diffuse_blur
     {
       // we get last pass result as a RTT input for the blur compositor
        compositor blur input_pass=diffuse
     }
     pass specular+diffuse_blur
     {
       texture_unit
       {
          // using last pass diffuse_blur result
          //texture_compositor compositor_name current_material pass_name
          texture_compositor blur human_skin diffuse_blur
       }
       // some standard specular pass 
       // along with a shader that will use diffuse_blur for final lighting.
   }
}
Currently to achieve this, I have a "diffuse" technique that will be used in the blur compositor, which I must instance and get resulting RTT texture name so that I can add it manually to the "specular+diffuse_blur" pass.
Each time I have to apply that material, I have to code all that, instancing compositor, getting names, renaming... when that could be done automatically, with compositor/RTT reuse and other advantages ?
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009] Improve Ogre's Compositor Framework

Post by Noman »

Ah, now I see what you suggested. You want to be able to use the compositor framework for single-object effects and not for full-scene effects. Interesting and useful!

So the 'input previous' of the compositor (or chain) would be rendering a certain pass onto a texture, and passing that to the compositor script. From my experience with the compositor framework, this would be quite a change. The compositor framework is currently pretty tightly coupled with the concepts of viewports and render queues. My first guess is that it will be quite a big change for Ogre. Not just code wise, but structurally wise. Besides being a harder task, it would change ogre quite a bit, meaning that I would have to work very closely not just with my mentor, but with the entire Ogre team (mostly sinbad). That might be possible if sinbad was my mentor, but I don't make those kind of decisions, and I don't know if its healthy for a SoC project to require that kind of synchronization between the mentor and student.

To become an option for this SoC project, I think this idea would have to be THE project, and not just a part of it. Its a legitimate idea, but I don't think it will fit as 'another part' in this one...
I'd like to hear other people's opinions about this, but that's my estimate.

I wanted to concentrate on full-scene / custom rendering sequence techniques, rather than single object techniques...
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by Assaf Raman »

I tend to agree with Noman, I do see the benefits of Tuan's idea but I think the goal for this proposal is full-scene / custom rendering sequence techniques and I recommend not getting too off track.
I do agree that single object techniques are interesting. We should add the support Tuan's suggested - I guess on our own (or as a patch from someone nice that knows what he it doing). I am working and going to work in then next few weeks on the mobile port, but when it is done - I may write the code needed for what Tuan wants. This can be done also after the summer project is over.
what do you think Tuan?
Watch out for my OGRE related tweets here.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by tuan kuranes »

I wanted to concentrate on full-scene / custom rendering sequence techniques, rather than single object techniques...
No problem, suggestions were about finding if some part of design if current application could help/fit in the "single-material effects".
My first guess is that it will be quite a big change for Ogre.
At least having that in mind while doing the design could give interesting foresights.

Better chaining/ordering/dependency would surely also ease the "no C++ code single-material effects" goal.
It seems to me that some code could be shared ? (unique referencing of RTT input or output, simplified "utility compositor" syntax, dependency checking, etc. )

@Assaf Raman: I would say that we should wait at least the end of Noman complete design, at least to be sure not to duplicate effort/design.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by Assaf Raman »

@tuan: agreed.
Watch out for my OGRE related tweets here.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by sinbad »

Also agreed - even if for the moment the subjects tuan raised are out of scope (and I think they are), it's always worth bearing in mind potential extensions and other applications that might come later, because that will inform your thinking for the design.

The sort of thing tuan proposes isn't actually such a big conceptual leap; having selective renders of certain objects, with post-processing, isn't actually that much different to the way compositors work now, it's just the dependencies that are different. Right now, it's assumed that the only dependent of a compositor pass will be a viewport quad pass, when in this case it's actually the combination of the viewport and a material; the offscreen renders are never actually used directly in a quad render necessarily, they go via a material. This is something that can be created manually in code now, but it would be nice to express it declaratively of course. The 'trigger' for the offscreen rendering is the same (the viewport - since even when the material is the dependent, it's the viewport that forms the basis of the pre-passes), but you would need the ability to wire the actual textures used to the material inputs automatically, and also to form visibility masks based on the objects using that material etc. I don't think anything in this summer of code project will preclude these extensions later, but it's worth raising it as a future extension, for purely contextual purposes.
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by Noman »

The reason I do think it is rather a big change, is because when I ask myself "When will it happen" I don't see any way to make just minor changes.
Since the off-screen object pass would only be defined in the material script, the render sequence wouldn't know that it needs to do these passes until its about to render the object.
We will need to add a 'collectOffScreenPasses' call to the scene graph or render queue, that would check all the visible objects for their active technique and see if it has an off screen pass, or that when a technique is about to be rendered, if an off-screen pass is encountered, the viewport will temporarily switch to the RTT, the off-screen pass will be rendered and then the regular rendering will continue. But won't that have a performance impact?

I get the drift, and I will make sure that the work that I do (if I get selected) in this SoC will benefit users of this future system. Named compositor chains and cross-references would definitely still fit in, the rest will probably work out nicely too... (Don't see any interferences yet)
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by Noman »

Thinking out loud here, this might belong in a different thread, and is unlikely to be part of this SoC.
Assuming that both my improvements and tuan's get implemented somewhere along the line, will Ogre's shadowing system still need to exist separately?
We could attach a compositor (called shadow) that renders the scene to a texture using a 'shadow' material scheme, and the materials in the regular passes (or the deferred passes) would be able to address these textures. There is some duplication in Ogre with techniques having 'shadow materials' and having access to 'shadow textures', rather than having techniques that are for the scheme 'shadow' and being able to address textures from the compositor 'shadow.

That way, normal rendering would cause the shadow maps to be pre-rendered (and thus being somewhat limited to the number of shadow casting lights), while deferred rendering would be able to render a shadow map & apply it to the scene in a sort of loop.

Again, I will NOT be adding a 'remove ogre shadow framework' stage to this year's SoC project, but I do think that this is a possible step in the future, and perhaps a smart one. Dunno how stencil shadows will fit in this architecture though. Just thinking out loud :)
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by sinbad »

Noman wrote:We will need to add a 'collectOffScreenPasses' call to the scene graph or render queue, that would check all the visible objects for their active technique and see if it has an off screen pass, or that when a technique is about to be rendered, if an off-screen pass is encountered, the viewport will temporarily switch to the RTT, the off-screen pass will be rendered and then the regular rendering will continue. But won't that have a performance impact?
You wouldn't actually do it during the rendering phase, but during the queueing phase. At the point Renderables are added to the queue, you know what material they're using, so at this point you can identify what prerequisite pre-render tasks are required, before you even start rendering the primary viewport. So no need to switch out to a RTT at the point you render the material to the main viewport - yes that would be very inefficient! You would queue up these requirements so that they could be amalgamated (e.g. multiple objects using the same material in the view), and trigger the union of the pre-renders needed before starting the main viewport update. In this way it's actually very like the way compositors work, it's just that the way you identify the need to perform the extra RTTs, and how you wire up the results is different.

Regarding the (texture) shadows, you raise a very interesting point. You could indeed represent the shadow texture renders differently to the way they are now, as a special case of the more general system you're proposing. Of course, they use different view settings which makes it slightly more complicated, and there are many different techniques that would need to be represented (automatic multipass, manual multipass, integrated shadows, additive and modulative variants), but in theory, you could make texture shadows a configuration of the kind of system you're proposing. We (I) would like to extend the flexibility of the shadow system some more anyway in the future, particularly allowing progressive rendering of the shadow textures for deferred shading (ie, don't always render all the shadow textures up front, instead allow them to be rendered 'just in time', re-using the same textures). This is a multi-pass approach of course which is less efficient in forward rendering than the integrated shadows approach (where you try to do 1 combined pass with all shadow textures already available, so up-front rendering makes sense), but for deferred rendering it potentially makes more sense to just render the shadow texture for the light you're about to render in a deferred manner (although you would have to switch render targets more often). The fact that in deferred shading it pays to pull out 'global' lights into early MRT render passes rather than doing them as a post-pass, such as directional lights and global illumination, also supports making this process as configurable as possible. So, given these potential extensions, it might well make more sense to think about refactoring the texture shadow sequence so that it can be expressed this way instead - possibly just as an option though, since the benefit of the integrated pipeline right now is that it's fairly easy.
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by Noman »

Hm. Yes that would work. The only problem with that approach is that each material that has an RTT pass will need to have its own instance of the compositor - meaning that local compositor textures will have to exist once per object. Not a huge drawback, its probably still the right thing to do.

Its true about the advantage of the simplicity of ogre's current shadow framework rather than containing it in the compositor framework. Perhaps a combination could be used (Ogre's internal implementation of texture shadows would be compositor based?) though its far too early to judge if that will be possible. It's probably a better idea to have this conversation at the end of the SoC if the project gets selected.

Small update - updated the first post with the application that I submitted a few days ago, for everyone to see...
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: [SoC 2009 - Submitted] Improve Ogre's Compositor Framework

Post by tuan kuranes »

compositor textures will have to exist once per object
That has to be specified somehow by the user, as some pass resulting may be just intermediate, and can be reused in other "single effect material". In the skin example, the first diffuse pass RTT is to be reused, where the second pass RTT (blurred diffuse) has to be kept until material applied.
Isn't there something like that in current Compositor ? Intermediate RTT reuse ?
It's probably a better idea to have this conversation at the end of the SoC if the project gets selected.
I still think we have to make sure that gsoc design will be at least compliant, if not designed to ease that future work.
Thinking all along the wrok how/where shadow framework could be replaced/shared seems a good thing to me.

Specifically, the "shadow material" current "linking" inside material could be shared/replace/enhanced by some way to fit both compositors and shadow framework ?
Locked