[submitted] Scene manager

[Praetor]

@ Lee "hmmmm?" Posting your curious mumblings and grumblings out loud now? (cue snarky laugh to take the sting out of the words)

@ Falagard I can see a possible recursion problem, but even if there currently is one, I don't think it would be too hard to guard against would it? Each cell can have a boolean telling whether it has been visited or not. Each time it is visited it is flipped. As you are leaving one cell to visit another, first check that the booleans don't match.

[Lee04]

Posting your curious mumblings and grumblings out loud now? (cue snarky laugh to take the sting out of the words)

Yeh I put in a message I wanted too delete later, but I couldn't find the delete button. So "hmmmm" had to do.

[farakon]

@Wolfmanfx: yes, I m preparing one as part of documentation to be released when the scene manager is fully implemented.

@Falagard: no recursion problem, neither in visibility determination, nor in building/decomposing. As for CSG/Bsp/level compile time, it is dependant of the size and complexity of the scene, the test scene I used took between 5s (in debug mode) on my AMD 1800, downto 1.5s I think on pentium 2.6 Gigs.

[Lee04]

Hi

Question 1

How about when you put different materials on all different wall sides all over. Does that work fine when you move around, sense Ogre sorts it's rendering on materials....?

Question 2

And does this case work?

---------------------------------------
l.................. l ............... l.................l....view position 1
l.................. l ............... l.................l....View position 2

.....room1..........room2.......rooom3.........View position 3
l.................. l ............... l.................l
l.................. l ............... l.................l
----------------------------------------

Question 3

Another question can a static mesh or several , entity or movable block the view to a portal?



Question 4

How does it know which room it is in when it starts? Do you have to tell it?
Or can it figure that out? Strange question perhaps.

Lee04

[farakon]

answers
1. If I understood the question right: the scene manager first gets occlusion queries result, then issues them, later it will put everything normally to the render queue or ask respective scene managers to do so as they see fit.

2. didn't understand the image, can u please explain it more?

3. I m thinking of making that addition soon, but for the moment no, i m thinking of adding both occlusion queries and occludee ability to movers and to whole scene manager (the later is a bit tricky, because the scene manager might be issuing its own queries)

4. The BSP is there for that, u get in which room the camera is, and from there u go.

[Lee04]

Hi

1. I don't understand your implementation but does it work well with different materials on different wall parts of the levels and entites etc Without flickering then you are in the clear.

Because Ogre sorts it's rendering on materials.

I used the Ogre shadow demo when I developed my plug-in to make sure all effects materials decals, shadows types etc works.
It's totoal generic for portal,s meshes, entities, clustering queries into one etc. As I said before SoC project of it's own merrits.

You have no materials applied yes or shadows or so...
Will it work, will there not be flickering when you move about?

2. Can you peek through many rooms portals in a room and the tests the different rooms? Can you set a threshhold for each portal of many pixel it should start to be visiable on or is it set to the same for all portals can it be changed?

3. OK.

4. OK I see.

Good job, I very impressed so far.

cheers

Lee04

[Ajare]

I think it would be a very good idea if you could store meta data with the polygons to be processed. This way you can flag whether polygons are, for instance, transparent or reflective, because both will have an impact on visibility determination.

[farakon]

Well Lee04, for the 1st question, all occluders are rendered are flat color, and for the portals, the material doesn't really matter as I disable writing to color and depth buffer before issuing and queries.

I didn't test it with shadows yet, I think I should correct something there before handling it.

For the portals and pixel count I didn't add it yet, as I was really into finishing the concept before adding more quirks, u r thinking ahead for now the portals are considered visible if one pixel is visible. I m currently doing a todo list, and this one is going to be on it.

Another feature I want to add is siscoring and viewport adjustement, something I envisioned but didn't do due to the lack of time.

For the visibility determination I m using the portals to determine which cells are visible. So as long as portals are visible the respective cells are visible. there are problems in using cells only with occlusion queries when portals are bigger on the other side, so the cell won't be considered visible. I thought it's better to use the portals for visibiliy query.

Again I m not sure with what u want to know, it seems u r refering to some sort of problems u had when developping ur plugin, maybe u can explain me more, so I can avoid some problems.

@Ajare: for the moment, the portals/occluders are flat materials, theoreticlaly it's possible to use transparent materials, but the result isn't assured.

[Lee04]

Well Lee04, for the 1st question, all occluders are rendered are flat color, and for the portals, the material doesn't really matter as I disable writing to color and depth buffer before issuing and queries.

Try different materials on all the walls and objects, move about,you see what I mean....Ogre sorts rendering on materials result is bad flickering.

Demos are nice, but Ogre is an engine of some complexity.

You can cheat, but then you don't get a good ground for "early z out" which is as important as hardare occlusion query, because that can cull pixel shading for several passes off almost occluded objects and save a lot of GPU pixel shadiing instructions. And for per pixel litt scenes thats important. Every light on a object adds a pass of shading.

Some screens has 1980x1024 pixels or even more, so it's important.

Thats why testing for one pixel is not so optimal a hole sector can be deamed visible and rendered with all it content. If it's the out door scetor then the out door scene that starts to be visible on one pixels....

My screen has 2.027.520 pixels....
So I test on say 10.000 pixels.

I just whish people would cooperate more than trying to reinvent the wheels over and over again....

[Lee04]

edit

[SuprChikn]

<offtopic>
@Lee04: the delete post button is the 'x' button right next to the 'edit' button. I'm pretty sure it's not there after someone has replied to your post though, so that wont help you now, but rather in the future.
[edit] hmm, it seems you're not allowed to delete your posts made in this forum (the SoC forum), but that button is there in other forums. Damn. [/edit]

Also, if you want to do some ascii diagram, put it in 'code' tags, not 'quote' tags, as 'code' tags will preserve spacing and ensure all characters are fixed width (so it will look the way you type it in notepad, etc).
</offtopic>

[farakon]

You can cheat, but then you don't get a good ground for "early z out" which is as important as hardare occlusion query, because that can cull pixel shading for several passes off almost occluded objects and save a lot of GPU pixel shadiing instructions. And for per pixel litt scenes thats important. Every light on a object adds a pass of shading.

I don't really need early z-out, I m using zbuffer to write occluders and then disable writing to both color and depth buffer, appart that if u use 1/4 resolution (viewport and scissoring) it'll get even better.

Occluders uses simple materials, they are not used for final rendering, the render queue is cleared after occlusion query and fresh. Of course my experience in Ogre isn't that greate, but later things are gone more or less normally!

I just whish people would cooperate more than trying to reinvent the wheels over and over again....

Cooperation? r u talking about ur plugin? sadly it'snt open source for the moment, and no much information available about it, if it was, it would have meant surely less work for me, I hope u have re-talked with sinbad about the relicense.

[Lee04]

I don't really need early z-out,

Yes, I know you don't need it. That's why it's missed in the final solution...
Thats why I give the heads up know.

I just say that most projects using expensive shaders and a couple of passes per object should. But if you don't add it then they can't without redoing your project from scratch.

It's funny with the real time 3D graphics communites, they are so happy to get something on the screen so they care less of what the engine can dismiss from rendering all together...Something real engines and applications really must concider....

Usually the boss dosn't have the insite of early z out or hardware occlusion query or the actuall costs of rendering of these things so they are so pleased just seeing that something nice is rendered....

Untiil the end of the project when they get into problems....

Just because plug-ins are in one licence today dosn't mean we can't find a way to use it with Ogre somehow.

I have propsed two different ways to the Ogre team and we have offered you to use it while doing your SoC.

@SuprChikn: Yes it's enoying I know. This time I had to weight my words carefully. So I don't have to delete again...

cheers

Lee04

[sinbad]

I'm sure I've pointed this out before, but early-Z out is incredibly easy to use in Ogre, and has been since about v0.13 - it doesn't need any fancy plugins to use. Just include a pass at the top of every (non-transparent) material that looks like this:

Code: Select all

material Whatever/You/Want
{
    technique
    {
        pass
        {
            colour_write off
        }
        
        // Your regular passes follow
    }
}

Ogre renders pass 0 of every material in the render queue first (then all pass 1's, etc), so this gives you your early z-out behaviour. I wish you wouldn't keep implying that it's not doable without some custom trickery. Perhaps you need to use such trickery for your particular purposes, but that doesn't mean the technique itself can't be used in many other conditions (I use it myself).

And for the record, the fact that your plugin is not open source means it can never be part of Ogre or the SoC. That's the same for everyone, Lee.

[Lee04]

but early-Z out is incredibly easy to use in Ogre,

So how does it sort the renderbles in this pass?

the fact that your plugin is not open source means it can never be part of Ogre or the SoC

I had two suggestions the first very doable. We just need to go out with that offer when unlimmited editon ships.

The second one is up to you.

None of them requires it to change license model.

You can mix license models in applications, people use Novodex PhysX with their Ogre applications for instance, this is not different and no it won't ship with Ogre it will ship seperate just like Novodex stuff.


Lee04

[Praetor]

Yeah, but you can't mix licenses in SoC. It is program to extend and work on open-source only. They aren;t very flexible rules. Now, you wanted to provide a non-open-source plugin through your own channels, I'm sure there would be interest.

Why is there need to sort in this pass? Wouldn't it just store the smallest depth value written?

[Lee04]

I would assume that Ogre sorts from back-to-front on that pass.

It can use early z out allready in this pass.

See what I mean?

cheers

Lee04

[sinbad]

So how does it sort the renderbles in this pass?

See RenderQueueInvocation, you can change the usual behaviour to distance based if you like. However, regardless of whether you do this, early z out is useful. You could render every object in the scene to the depth buffer (only) in a completely arbitrary order and it could still benefit from early z out, because the point is that you pre-seed the entire depth buffer; subsequent (colour, pixel shaded) passes all benefit from the very fast depth rejection. You only need to sort by ascending distance if you're planning on getting early z-out onthe very first pass, e.g. you're using pixel shaders in the very first pass.

the fact that your plugin is not open source means it can never be part of Ogre or the SoC

I had two suggestions the first very doable. We just need to go out with that offer when unlimmited editon ships.

None of them requires it to change license model.]

We had a discussion at the time and I explained my position. The rest of the discussion is probably OT for this thread so I don't want to derail it any further by explaining again.

[Lioric]

When Tuan and I, were talking about our Hardware Occlusion Queries implementations, you kept saying the word "cheat" of any implementation besides yours, and here too

There is no "cheat" and "correct" implementations, this is game development, not scientific research and visualization, we favor look and feel over physically correct behavior or result

There are only the "usable" and "non-usable" implementations, if it feels good and performs good, then its usable

The issue seems to be the difference in your implementation, it seems you use the full object with all of its final materials and shaders in the Occlusion Query stage, when we or other implementations uses the BB or a low detail convex hull or proxy with a very simple material and the most simple buffer settings needed for the query to perfom and get the results async

In our projects, we use the "color_write off first pass" trick to do the early-Z out, and it works good, the full depth buffer its updated and non visible pixels are not processed into the shader units

[Praetor]

I'd have to even question a claim of "physical accuracy" here for occlusion queries. This is still graphics, not something like physics simulation. So... making something that looks right is the key. Since occlusion queries shouldn't even affect the look of something, just the performance I don't see any sort of argument here. If one implementation cuts down the pixels processed by 50% and the other by 55% it hardly seems like a sticking point.

[Lee04]

the plug-in can use the high res. LOD of a model or the low res. it can use a bb.

There is no "cheat" and "correct" implementations, this is game development, not scientific research and visualization, we favor look and feel over physically correct behavior or result

It's time you don't have the time so you cutt corners.
But please tell people what it dosn't do. Or they belive otherwise.

Feeling and look has with performance to do in the end if you saturate the system.

Well if you just have one pass one material and two objects on the screen you won't have any problem. Look and feel is OK.

But if you want to use DX9 Sm3.0 features and do some heavy scenes you need to save your computers energy and put it where it counts.

What I mean with cheap trick is rendering too an off screen texture usually smaller then screen size. Which results in a depth buffer in the RTT that can't be used for early z out because it's not the same size.

This means you need two extra passes of your scene one for hw occlusion query one for depth whoich is even more expensive...

And if you use the same size on the RTT and the backbuffer you use up too much memory.

I also mean that any solutions that only use bb has tio high fill rate cost.

And finally that the implementations are not general. Everything can't be occluders occludes automatically. It's just for the scene manager nodes or it's just for the portals. And if you want to use hardware occlusion query on something else in the scene as well you are done for.

The plug-in does the depth pass and hardware occlusion test in one pass on the backbuffer using the hole Ogres rendering system.

You don't need to set a special pass for every technique for every material in your application.

It's more generic and it's exactness is only depending on how you use it.

So you see, it's very different from those one of solutions for particular demos or applications that visualize an empty city, structure thing.

It's more geared for giving feed back to a game engine via an API on what’s really visible on a really detail level using callback interface.

Using boxes... well that’s fine to start learning hardware occlusion or doing a little specialist thing. It's not what the manual suggests.

There might be applications out they’re stopping and waiting for a query to be ready. Next you will claim that’s OK too? Having the CPU sitting idle doing nothing is one of the seven sins.

Bad accuracy is when a bb is visible but the object in it isn't.
And the BB takes up more fill rate than the object.

When I did the plug-in Ogres rendering options was not as advanced as they are today. Still you can get a way with alot.

I am so sorry if I am unjust in my feers that fast implementations may lead people to belive it's general solutions.,

Another "cheat" is thoose apps that use hw occlusion query but they render the frames with Ogres low level API only. They don't support Ogres renderques or rendering at all which this plug-in does. It supports basic Ogre rendering pipeline.

So hopefully you now understand what I mean by "cheats", they are plenty the ones you read about and think thats sound general and fine but they are not.

One cheat for early z out is to sort on materials but make an exeception for the a couple of big objects in the front (good occluders) and render them first and the rest of the scene sorted on materials.

There are just so many cheats...

Being a good game programmer means knowing them and using them
where they fits.

Middleware developers however needs the general solutions that works with the hole system, because different customers want's to use it for different applications with different needs. Thats why our plug-in is general
and as complete as possible.





'

[Falagard]

use hardware occlusion query on something else in the scene as well you are done for.

Not exactly, if I understand correctly what he's done with the portal and cell occlusion is to determine if a cell should be rendered if the portal is visible using his occlusion query technique, then if it is visible, he passes it down to the scene manager that is handling cell.

If that scene manager has an occlusion query implemention, such as the PLM2's occlusion system, then it would take over at that point and perform occlusion culling at the per object level (or however the particular scene manager implements it).

This seems like the most flexible solution for a portal/cell system, allowing the scene manager to handle the scene while the portal/cell system decides what is visible for gross culling.

I haven't tested out PLM2's occlusion stuff yet, but I intend to. If I understand correctly it does full object rendering for occluders, and then performs bounding box hardware occlusion queries for occludees.

Anyhow, I don't want to get into specifics about PLM2's solution, because even if it isn't the best solution for scene manager, farakon's portal/cell solution still seems the best for a high level portal/cell system that passes the actually rendering down to the cell's scene manager.

[Lee04]

If that scene manager has an occlusion query implementation, such as the PLM2's occlusion system, then it would take over at that point and perform occlusion culling at the per object level (or however the particular scene manager implements it).

The scene managers don't do the rendering and hardware occlusion query stuff per say that I leave to Ogre and the plug-in.

having scene managers worry about hardware occlusion rendering stuff seams hard way to work if you want people to build and create new scene managers like that.

With our plug-in scene managers or hybrid BSP systems alike just create queries attach them to objects they like to have them on portals and objects etc. and va la it works.

I think everything is good project wise I just whish Ogre had a solid take on hardware occlusion with early z out. That’s why we did the plug-in.
We want Ogre to be a state of the art engine.

But mostly there are so many other things that are so important for Ogre team to do so things like this is put to the side.

Personally I think this is very basic thing in an 3D engine of today.
If Ogre had a general robust hw occlusion query support we wouldn't see this half solutions.

We added the low level API to help the process. Nothing happens and then this.... I a bit disappointed in open source It's like this genetic algorithm you make a creature and set a goal that it should go somewhere and you give it leggs, but when the genetic algorithm is finished it drags it over the floor rather then walks....

I have created a monster out of Ogre I am so sorry.

Lee04 Frankenstein

[farakon]

If we want to be objective, let's talk about the implementation time. I m new to Ogre, in a matter of 3 monthes I had to relearn OGRE, implement a CSG, and a scene manager using hardware occlusion query. Neither one of them is a simple thing to do. I have never claimed it's stable, nor claimed it to be perfect, and never claimed to be very complete, it still need some work before it finalised, a month or two maybe and then you have what you want. The question how much time (one programmer time) did it took you to finish your plugin?

You keep talking about cheap tricks, I want to ask you, do you know about optimization? what is optimization anyway? isn't it finding little things you can use for ur interrest and use them?

The concept behind the plugin was simple, the implementation was not, the portal system using occlusion query seems a good solution since it reduces occlusion query overhead. And as Falagard noted, queries are done for portals, from there u get the rest. The scene manager is left to do whatever it wants later, seems a viable solution for me, if u don't have a scene manager attached it'll call the default generic functionalities.

Lee04: I v already mentioned that I appreciate ur offer for ur plugin, but SoC license is incompatible and would blow the SoC spirit. I do agree with sinbad, discussing ur plugin and license is not the subject of this thread.

Cheers

[Lee04]

I have never claimed it's stable, nor claimed it to be perfect, and never claimed to be very complete, it still need some work before it finalized, a month or two maybe and then you have what you want.

@Farakon: No you haven’t, you just said it's working.

I know from experience that what seams to be working can be just a lucky coincident. Like you not using different materials on the wall parts yet could possibly lead to that the sorting is OK until you add materials on the walls. That’s why I mentioned it and that is not off topic. Nor is hardware occlusion queries sense you are adding it to your project and talking about it. Yeh you can get pissed of at my that’s another issue

I am sorry if I try to guide you like a beginner you probably have good help from the other guys here, I am just worried you don't catch these things. I don't know your level of expertise. I am just trying to help because I want to see a good system in the end.

Lee04: I v already mentioned that I appreciate ur offer for ur plug-in, but SoC license is incompatible and would blow the SoC spirit.

@Farakon: No one told me this incompatibly with the SoC license I didn't even know there where a separate license for that. I just didn't get a replay explaining this issue.

So you can't use DirectX either then?

In our projects, we use the "color_write off first pass" trick to do the early-Z out, and it works good, the full depth buffer its updated and non visible pixels are not processed into the shader units

@Loric: But do you do the hardware occlusion query test in this pass as well? Can you use whatever object entity as well at different LODs?