New InstanceManager: Instancing done the right way

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
User avatar
Pyritie
Gnome
Posts: 363
Joined: Wed Feb 25, 2009 6:15 pm
Location: UK
x 8

Re: New InstanceManager: Instancing done the right way

Post by Pyritie »

Does this work with mogre, and if it doesn't, would it be difficult to make it work with it?

My scene has about 1200 trees which make a huge difference to speed (visible gives me 25-50FPS, invisible gives me a near-constant 60), using about 12 different models. At the moment I'm using static geometry, which is the best solution so far but also makes the game use up about twice as much RAM.

Though I tried using ogre 1.7's instance manager and it wasn't much faster than regular entities. I'm assuming this instancemanager is a lot better than that, right?
I use Mogre, BulletSharp, LuaNetInterface, irrKlang, and Miyagi! | Ponykart (Showcase)
Image
jonim8or
Goblin
Posts: 287
Joined: Mon Dec 08, 2008 4:49 pm
x 10

Re: New InstanceManager: Instancing done the right way

Post by jonim8or »

The speed improvement depends on a lot of factors. What I think are the two most important factors:
1) does the target hardware support some kind of hardware instancing?
2) do you use scenenodes to position the trees, or do you set the positions of the instancedEntities directly?
I find the extensive test that was done earlier in the thread by Mattan very helpful (although the exact figures really depend on your hardware and your scene)
Image
User avatar
Pyritie
Gnome
Posts: 363
Joined: Wed Feb 25, 2009 6:15 pm
Location: UK
x 8

Re: New InstanceManager: Instancing done the right way

Post by Pyritie »

jonim8or wrote:The speed improvement depends on a lot of factors. What I think are the two most important factors:
1) does the target hardware support some kind of hardware instancing?
uh, I think so? The normal ogre instancing thing requires a shader
2) do you use scenenodes to position the trees, or do you set the positions of the instancedEntities directly?
I set the positions and stuff of "InstancedGeometry.InstancedObject"

I find the extensive test that was done earlier in the thread by Mattan very helpful (although the exact figures really depend on your hardware and your scene)
Image
Hrm, my instanced things are all not moving and have no animation with shadows optional... the speed increase doesn't look that significant for something like that
I use Mogre, BulletSharp, LuaNetInterface, irrKlang, and Miyagi! | Ponykart (Showcase)
Image
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

@dark_sylinc
1) Can you remove warning C4512: 'Ogre::InstanceManager' : assignment operator could not be generated (I use Visual Studio 2005 sp1)
For example you can declare assignment operator and copy constructor as private functions and don`t implement it.

2) Add #include "OgreInstanceManager.h" #include "OgreInstancedEntity.h" #include "OgreInstanceBatch.h" to common Ogre header "Ogre.h"
https://sourceforge.net/tracker/?func=d ... tid=302997

3) See at constructor InstanceBatch::InstanceBatch

Code: Select all

{
	assert( mInstancesPerBatch );
	//Force batch visibility to be always visible. The instanced entities
	//have individual visibility flags. If none matches the scene's current,
	//then this batch won't rendered.
	mVisibilityFlags = std::numeric_limits<Ogre::uint32>::max();

	if( indexToBoneMap )
		assert( !(meshReference->hasSkeleton() && indexToBoneMap->empty()) );

	mFullBoundingBox.setExtents( -Vector3::ZERO, Vector3::ZERO );
	mName = batchName;
}
How do you think, what will be in release build?

Code: Select all


	if( indexToBoneMap ) /* m2codeGEN fix */ {
		assert( !(meshReference->hasSkeleton() && indexToBoneMap->empty()) ); /* m2codeGEN fix */ }

	mFullBoundingBox.setExtents( -Vector3::ZERO, Vector3::ZERO );

Regards,
Vadim
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

m2codeGEN wrote: 1) Can you remove warning C4512: 'Ogre::InstanceManager' : assignment operator could not be generated (I use Visual Studio 2005 sp1)
For example you can declare assignment operator and copy constructor as private functions and don`t implement it.
A level 4 warning eh? I guess that's why I missed it (VS 2008 Express here).
I'm not sure how to proceed here. I definitely do NOT want accidental copies, which could happen even if the function is private when done from inside InstanceManager. The reason behind this is to raise an alert on future developers that InstanceManager is maintained by SceneManager, and the name is important, so you can't go carelessly making copies. Besides there shouldn't be a need for that.
Another way to workaround this is to include a pragma, an which may I'm not too keen with either.
I'm hearing for options or arguments.
2) Add #include "OgreInstanceManager.h" #include "OgreInstancedEntity.h" #include "OgreInstanceBatch.h" to common Ogre header "Ogre.h"
https://sourceforge.net/tracker/?func=d ... tid=302997
Done. For some reason Mercurial refused the patch. I hate that in Hg. Don't know why it happens so often.
So I did it manually.
3) See at constructor InstanceBatch::InstanceBatch

Code: Select all

{
	assert( mInstancesPerBatch );
	//Force batch visibility to be always visible. The instanced entities
	//have individual visibility flags. If none matches the scene's current,
	//then this batch won't rendered.
	mVisibilityFlags = std::numeric_limits<Ogre::uint32>::max();

	if( indexToBoneMap )
		assert( !(meshReference->hasSkeleton() && indexToBoneMap->empty()) );

	mFullBoundingBox.setExtents( -Vector3::ZERO, Vector3::ZERO );
	mName = batchName;
}
How do you think, what will be in release build?

Code: Select all


	if( indexToBoneMap ) /* m2codeGEN fix */ {
		assert( !(meshReference->hasSkeleton() && indexToBoneMap->empty()) ); /* m2codeGEN fix */ }

	mFullBoundingBox.setExtents( -Vector3::ZERO, Vector3::ZERO );

Nice catch! Although if a properly implemented assert is used this shouldn't happen, it's better safe than sorry.

Thanks
Dark Sylinc
Gdlk
Halfling
Posts: 53
Joined: Mon Dec 05, 2011 9:43 pm
x 1

Re: New InstanceManager: Instancing done the right way

Post by Gdlk »

Hi!

I dont know if this is the right place for the question, if it isn't, sry =(

The question is How can I generate a mesh supported by the instanceManager (i.e. without shares vertices) ?, I had all the system ready to work, and with the robot mesh work great!, but with my mesh (exported from blender) the system dont work. With the debug I found that the shares vertices line is where crash it =( (the robot mesh has set in false, my mesh in true)

Thanks in advance!
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

I found another issue in InstanceManager. It`s about double precision problem and big scenes. It`s vertex shaking or Jitting
After some time I will give a code that can fix this.

But first, let's take a look at the problem. Let us consider only the technique HardwareInstaceBasic.
1) HardwareInstaceBatch is the successor of the classes Renderandeble and MovebleObject. Thus he should be attached to Scene Nodes to be rendered. Accordingly, when it can be drawn, the calculation of MODELVIEW matrix is done on the central processor (in doubles). But

Code: Select all

void InstanceBatchHW::getWorldTransforms( Matrix4* xform ) const { *xform = Matrix4::IDENTITY;}
So gl_ModelViewProjection is a combination of the view and projection matrix.
2) In a Vertex Shader position of vertex calculates in world coords. But in glsl 4.0-- mat4 and vec4 is a float type.

Code: Select all

uniform mat4 viewProjMatrix;
void main(void)
{
	mat4 worldMatrix;
	worldMatrix[0] = uv1;
	worldMatrix[1] = uv2;
	worldMatrix[2] = uv3;
	worldMatrix[3] = vec4( 0, 0, 0, 1 );
	vec4 worldPos		= vertex * worldMatrix;
	gl_Position			= viewProjMatrix * worldPos;

Prospective ways of the decision of problems.

1) As attributes of the vertices transfer calculated on the CPU modelview matrix. Need modify InstanceBatchHW::updateVertexBuffer

Code: Select all

attribute vec4 vertex;
attribute vec3 normal;
attribute mat4 modelView

void main()
{
   vec4 worldPos = vertex * modelView;
But using a static buffer immediately becomes not possible.

2) Use relative coordinates entities when recording in the vertex buffer. And in method InstanceBatchHW::getWorldTransforms returns transform of parent SceneNode (let`s set him position in center of all entities). Of course there's a problem, if the fact will be too far removed from the SceneNode position. But it is better than the jitter in world coords.
need modify Vertex Shader like this

Code: Select all

attribute vec4 vertex;
attribute vec3 normal;
attribute mat4 relativeTransform;  // for small writing

uniform mat4 modelViewProjection;

void main()
{
   vec4 vertexPos = relativeTransform * vertex;
   gl_Position = modelViewProjection * vertexPos;
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

Hi, thanks again for the feedback. I know english isn't your mother language but you should try to exercise it more often, your grammar makes it hard to understand where you're getting at.
m2codeGEN wrote:I found another issue in InstanceManager. It`s about double precision problem and big scenes. It`s vertex shaking or Jitting
It's a common problem in all rendering engines. Normal Entity rendering has the same problem.
I'm very familiar with extremely large scenes.
m2codeGEN wrote:Accordingly, when it can be drawn, the calculation of MODELVIEW matrix is done on the central processor (in doubles).
I'm afraid that's quite a big oversimplification.
  • In D3D, calculation will be done single or double precision depending on the floating point mode.
  • In GL, you'll have to set the fp control word manually and check it periodically to be sure that you're running at double precision mode. Chances are this is true, but not guaranteed.
  • In both cases, the calculation may be performed at double precision internally, but it is eventually truncated/rounded to single precision when stored in memory, although alleviated when compiled with OGRE_DOUBLE_PRECISION.
  • In both cases, the problem is NOT because it's done in doubles (although it helps); but rather because the world matrix multiplication is done in the shader. Because we do (vertex * world) * viewProj; the first parenthesis becomes too large and loses a lot of precision; then multiplying by viewProj will resize those numbers down, but the precision has already been lost. The solution is to first perform (world * viewProj) and then multiply it by the vertex. Watch out for the order of operations. Needless to say, when your technique (i.e. HW VTF) uses more than one bone per vertex, the number of operations grows very quick because you're performing lots of (world*viewProj). Performing "relative" world transforms for bone poses (as you propose) can help alleviating the problem too.
  • You'll note the same jittering & shaking when rendering regular Entities when skeletally animated in the vertex shader.
  • Calling SceneManager::setCameraRelativeRendering( true ) will greatly alleviate these problems; but it's possible you will still see some jittering and shaking in the small details (i.e. the eyes of a 1.75m character).
  • Be sure all your libraries (Ogre included) is compiled with Floating Point Model 'precise' or 'strict'. Ogre default parameter is 'fast' which is very innacurate.
So, to sumarize:
  • In D3D9, set the floating point mode to 'Consistent'
  • In all rendersystems, use _controlfp and periodically check it's still in double processing mode.
  • Compile with floating point model 'precise'
  • Call SceneManager::setCameraRelativeRendering( true )
Edit: Forgot to mention; the best, ultimate solution is to translate everything to origin. Many games do this; and all professional physics engines support this feature. With large scenes (+1000 units), floating point accuracy & stability becomes a problem in all aspects, not just graphics: AI, pathfinding, physics simulation, etc.
For example Havok has it's own method called "shiftAllObjects" to solve exactly this problem

Cheers
Dark Sylinc
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

dark_sylinc wrote:I know english isn't your mother language but you should try to exercise it more often, your grammar makes it hard to understand where you're getting at.
Well, I will try to give to it more attention. Often I can't be understood and in a native language :)

I have already build OGRE with the OGRE_DOUBLE_PRECISION and use GL render system. Ogre::Entity attached to SceneNode visualized without graphic artifacts. The problem is very similar to this http://www.fxpression.com/phpBB3/viewto ... 72b2015b42

Call SceneManager::setCameraRelativeRendering( true ) reduces fps in 2-3 times (we can`t use static vertex buffer).
But the universal decision doesn't exist. I suggest to make a small patch which will carry out calculations in relative coordinates, and you then estimate it and if necessary will make changes to a repository.
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

I can`t submit path (SourceForge.net executed with error on page). So diffs will be posted here.
NewInstacing. Small performance improvement

Code: Select all

# HG changeset patch
# User Vadim Sergeev <m2codeGEN@rambler.ru>
# Date 1328265980 -14400
# Branch v1-8
# Node ID 2127c412f549c090a1d5254edac8ebbf199017c4
# Parent  4ac925a83e6a315eedb672124ab08d81b02653bd
NewInstacing. Small performance improvement

diff -r 4ac925a83e6a -r 2127c412f549 OgreMain/src/OgreInstanceBatch.cpp
--- a/OgreMain/src/OgreInstanceBatch.cpp	Tue Jan 31 02:02:13 2012 -0600
+++ b/OgreMain/src/OgreInstanceBatch.cpp	Fri Feb 03 14:46:20 2012 +0400
@@ -166,6 +166,8 @@
 			//InstancedEntity is not part of the scene graph (i.e. ultimate parent is root node)
 			//to avoid unnecessary wasteful calculations
 			mVisible |= (*itor)->findVisible( mCurrentCamera );
+         if (mVisible)
+            break;   // performance trick
 			++itor;
 		}
 	}
@@ -219,13 +221,12 @@
 
 		for( size_t i=0; i<numFloats >> 2; i += 3 )
 		{
-			const Vector3 worldTrans( mat3x4[(i+0) * 4 + 3], mat3x4[(i+1) * 4 + 3],
-										mat3x4[(i+2) * 4 + 3] );
-			const Vector3 newPos( worldTrans - cameraRelativePosition );
+			const Vector3 worldTrans(mat3x4[(i+0) * 4 + 3], mat3x4[(i+1) * 4 + 3], mat3x4[(i+2) * 4 + 3]);
+			const Vector3 newPos = worldTrans - cameraRelativePosition;
 
-			mat3x4[(i+0) * 4 + 3] = newPos.x;
-			mat3x4[(i+1) * 4 + 3] = newPos.y;
-			mat3x4[(i+2) * 4 + 3] = newPos.z;
+			mat3x4[(i+0) * 4 + 3] = (float)newPos.x;
+			mat3x4[(i+1) * 4 + 3] = (float)newPos.y;
+			mat3x4[(i+2) * 4 + 3] = (float)newPos.z;
 		}
 	}
 	//-----------------------------------------------------------------------
Updated
patches in SourceForge.net
https://sourceforge.net/tracker/?func=d ... tid=302997
https://sourceforge.net/tracker/?func=d ... tid=302997
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

Hi!

Have you tested there is actual performance improvement?
Why did you change the assignment on constructor by assignment by '=' operator? Is there a special reason?
The C cast will be changed to static_cast; which is the C++ way of handling conversions.

The if(mVisible) break; can be replaced by "while( itor != end && !mVisible )" which tends to compile into branch-prediction-friendlier code.

I can't download the local translation patch :( I get a php file of SF.Net website instead :\
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

dark_sylinc wrote: Have you tested there is actual performance improvement?
Hi! This is small improving for ShaderBased and VTF
dark_sylinc wrote: Why did you change the assignment on constructor by assignment by '=' operator? Is there a special reason?
As a whole it is the equivalent operations. Assignment ' = ' operator returns the link to Vector3, but to read the code begins hardly easier.
dark_sylinc wrote: The C cast will be changed to static_cast; which is the C++ way of handling conversions.
You are absolutely right. It is necessary to add that if static_cast it is not has gone right, it will be applied reinterpret_cast.
For the built in data types (such as float, int, etc.) C application cast is quite admissible, since the code is easily read. For the user data types it is forbidden (our corporate standards)
dark_sylinc wrote: The if(mVisible) break; can be replaced by "while( itor != end && !mVisible )" which tends to compile into branch-prediction-friendlier code.
:\
I have simply shown a way for improving. Such variant more can will approach for the working version.

Code: Select all

void InstanceBatch::updateVisibility(void)
{
   bool bVisible =  mVisible = false;   // <-- may be register variable
   if (!mInstancedEntities.empty())
   {
      InstancedEntityVec::const_iterator itor = mInstancedEntities.begin();
      InstancedEntityVec::const_iterator end  = mInstancedEntities.end();

      do {
         bVisible = (*itor)->findVisible( mCurrentCamera );
      } while(!bVisible && ++itor != end );
   }
   mVisible = bVisible;
}
Or use the pointer on the pointer instead of iterators. But then the container should to be serial is mandatory.
dark_sylinc wrote: I can't download the local translation patch :( I get a php file of SF.Net website instead :\
Yesterday I too had difficulties with this site. Today all works.

But all above described are trifles :D It is much more important, that you have paid the attention to this patch since I could forget casually something
http://sourceforge.net/tracker/?func=de ... tid=302997
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

m2codeGEN wrote:Hi! This is small improving for ShaderBased and VTF
The reason I ask is that it may not be a performance improvement, but rather worse. Do you have a comparison in milliseconds?
m2codeGEN wrote: I have simply shown a way for improving. Such variant more can will approach for the working version.

Code: Select all

void InstanceBatch::updateVisibility(void)
{
   bool bVisible =  mVisible = false;   // <-- may be register variable
   if (!mInstancedEntities.empty())
   {
      InstancedEntityVec::const_iterator itor = mInstancedEntities.begin();
      InstancedEntityVec::const_iterator end  = mInstancedEntities.end();

      do {
         bVisible = (*itor)->findVisible( mCurrentCamera );
      } while(!bVisible && ++itor != end );
   }
   mVisible = bVisible;
}
Or use the pointer on the pointer instead of iterators. But then the container should to be serial is mandatory.
Oh my God. NO.
We're not in 2000 anymore. An IF statement (or conditional evaluation to be exact) is not a performance improvement. It has to be actually tested to see that.
http://assemblyrequired.crashworks.org/tag/intrinsics/
http://research.scee.net/files/presenta ... CAP_09.pdf (see slide 27)

This applies for PCs, handhelds and consoles; but specially for the two latter ones.
Since processor speed doubles every 1,5 years, but memory bandwidth & latency doesn't; the conditional evaluation as a performance improvement is less & less an "improvement". Furthermore Ogre is expanding to those areas other than PC, so it's a no go.

"if" statements can increase performance in those areas where massive ammount of code will be skipped.
m2codeGEN wrote: But all above described are trifles :D It is much more important, that you have paid the attention to this patch since I could forget casually something
http://sourceforge.net/tracker/?func=de ... tid=302997
I kept getting a corrupted file until I signed in; then I was able to download your patch. I'll be evaluating it this weekend.

Cheers
Dark Sylinc
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

Thanks, I read this document.
In our case mInstancedEntities.size() == 2500 for example. I don't think that is necessary to execute all 2500 iterations.
Moreover 95 % of the Ogre code are written in the bad style if to follow article instructions. I am poor assembly programmer and I trust the compiler :)
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

I've reviewed your patch (haven't got the time to apply it yet). My conclusions:

I like your changes, I really do. I like that:
  • You fixed the bool to int warnings. They were annoying
  • You fixed buffer overflows when compiled with double precision.
These two above will go to Ogre 1.8; they're kinda important non-breaking changes.

As for the local transformations, which is the main point of your patch:
I like it too; BUT what I don't like is where the code is used. I'm not comfortable with the idea that a perfectly working scene breaks when recompiling with OGRE_DOUBLE_PRECISION. It's not what a common user would expect; it's just not intuitive.
Furthermore this means that you can no longer use the same shaders (with little modifications through macros) for Ogre Entity & InstancedEntity

Luckily my brain "clicked" when seeing your code and saw a solution. Like I said in a previous post; this problem is also common to regular Entity(ies) using HW skeletal animation. My proposal is:
  • For InstancedEntity, add a new option to tell whether the user wants local or full transforms
  • For regular Entities, add a new material variable called world_matrix_local_array_3x4 (and add a new function MovableObject::getWorldLocalTransforms() ) that returns the local bone transformations; and modify world_matrix, worldview_matrix & worldviewproj_matrix so that they always return the world matrix of the scene node (IIRC, the problem currently is that they'll return the first matrix of the array which is one of the bones'). Note that Ogre::Entity already contains the local transforms in their memory so it's not necessary to "undo" the world transformation like you do with Instancing
This way, handling of Entities & InstancedEntities can be done consistently; and users may want to use light & camera bindings in object space for their lighting calculations. This change will go to the default branch, not 1.8
I'm not sure I have the time to implement this by myself; would you like to do it instead? Eventually the precision in Entity problem is going to kick you just like InstancedEntity did unless you always stick to SW animation.
You have to understand I need to ensure the system is usable for everyone and covers everyone's needs, not just a particular situation.

I'll be uploading the overflow and bool to int fixed around these days.

Thanks
Dark Sylinc
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

It is good that we have found things in common :)
dark_sylinc wrote: As for the local transformations ...
I'm not comfortable with the idea that a perfectly working scene breaks when recompiling with OGRE_DOUBLE_PRECISION. It's not what a common user would expect; it's just not intuitive.
Furthermore this means that you can no longer use the same shaders (with little modifications through macros) for Ogre Entity & InstancedEntity
In it you are certainly right, but this first decision, which has come to to me mind not breaking Ogre API

Code: Select all

void InstanceBatch::getWorldTransforms( Matrix4* xform ) const
{
#ifdef OGRE_DOUBLE_PRECISION
   *xform = _getParentNodeFullTransform();
#else
   *xform = Matrix4::IDENTITY;
#endif
}
Changeover of variable "viewProjMatrix" by a variable "worldViewProjMatrix" would not lead to performance penalty (We continue to multiply only one matrix). In a case if Real is float type the world matrix will be identity.
dark_sylinc wrote: My proposal is:
  • For InstancedEntity, add a new option to tell whether the user wants local or full transforms
  • For regular Entities, add a new material variable called world_matrix_local_array_3x4 (and add a new function MovableObject::getWorldLocalTransforms() ) that returns the local bone transformations; and modify world_matrix, worldview_matrix & worldviewproj_matrix so that they always return the world matrix of the scene node (IIRC, the problem currently is that they'll return the first matrix of the array which is one of the bones'). Note that Ogre::Entity already contains the local transforms in their memory so it's not necessary to "undo" the world transformation like you do with Instancing
This way, handling of Entities & InstancedEntities can be done consistently; and users may want to use light & camera bindings in object space for their lighting calculations. This change will go to the default branch, not 1.8
I have understood idea, but these are already essential changes in API.
By the way, I not so like the decision to transfer transformation of instancies in attributes of a vertex shader as a matrix 4х3. Yes it saves uniform variables for ShaderBased, but on the other hand we reserve 3 textural units at usage HardwareBasic technics. Can be costs then to the successor of class InstanceBatch to define in what format to write down transformations in a vertex shader?
dark_sylinc wrote: I'm not sure I have the time to implement this by myself; would you like to do it instead? Eventually the precision in Entity problem is going to kick you just like InstancedEntity did unless you always stick to SW animation.
In our company I am the team leader of programmers engaged in three-dimensional visualisation, therefore time for the free creativity at me also end-to-end.
I could deal with this problem, but most likely not earlier than May. Till May at all of us forces are thrown on the internal project.

Regards,
Vadim
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

m2codeGEN wrote: Changeover of variable "viewProjMatrix" by a variable "worldViewProjMatrix" would not lead to performance penalty (We continue to multiply only one matrix). In a case if Real is float type the world matrix will be identity.
Don't know why you bring this up, but I'm not concerned about performance in this case.
If it's because I said "Note that Ogre::Entity already contains the local transforms in their memory so it's not necessary to "undo" the world transformation like you do with Instancing" I was talking about Ogre::Entity.
In instancing you're doing:

Code: Select all

+#ifdef OGRE_DOUBLE_PRECISION
+
+      Vector3 posSceneNode = (mFullBoundingBox.getMaximum() + mFullBoundingBox.getMinimum()) * 0.5;
+      mFullBoundingBox.setMaximum(mFullBoundingBox.getMaximum() - posSceneNode);
+      mFullBoundingBox.setMinimum(mFullBoundingBox.getMinimum() - posSceneNode);
+
+      getParentSceneNode()->setPosition(posSceneNode);
+
+#endif
Which is correct, but in "Entity" (no Instancing) that's not even necessary, as the variables are already cached. That's what I meant.
m2codeGEN wrote: By the way, I not so like the decision to transfer transformation of instancies in attributes of a vertex shader as a matrix 4х3. Yes it saves uniform variables for ShaderBased, but on the other hand we reserve 3 textural units at usage HardwareBasic technics. Can be costs then to the successor of class InstanceBatch to define in what format to write down transformations in a vertex shader?
May be I didn't explain myself correctly. For instancing; the local transform would be used exactly as in your patched code (except the developer could opt to use your behaviour or the old one; rather than hardcoding through macros)
As for normal Entities (no instancing); the worldMatrix array would be binded with world_matrix_local_array_3x4 instead of the usual world_matrix_array_3x4; and then they should multiply against worldViewProj instead of the usual viewProj matrix.

"world_matrix_local_array_3x4" should not be used for instancing (i.e. raise an exception).
m2codeGEN wrote: In our company I am the team leader of programmers engaged in three-dimensional visualisation, therefore time for the free creativity at me also end-to-end.
I could deal with this problem, but most likely not earlier than May. Till May at all of us forces are thrown on the internal project.
Similar schedule here. I'll check in the patch for the buffer overrun & the compiler warnings; but the local transform code will have to wait until one of us is ready to polish it. I hope you can still run your own modified code meanwhile.

Cheers
Dark Sylinc
User avatar
m2codeGEN
Halfling
Posts: 52
Joined: Tue Apr 26, 2011 9:13 am
Location: Russia, Tver
x 2

Re: New InstanceManager: Instancing done the right way

Post by m2codeGEN »

I think it it will be interesting. Early results experiments of large forests rendering using New InstanceManager
[youtube]UiWeXwSAdWU[/youtube]
Nodrev
Gremlin
Posts: 193
Joined: Fri Jan 25, 2008 6:55 pm
Location: Nantes / France
x 17

Re: New InstanceManager: Instancing done the right way

Post by Nodrev »

Hi everyone,

I'm currently writing a paged entity system (pretty like PagedGeometry plugin, but using Ogre's internal paging system), and was planning to build static geometry for near pages, but I take a look to this Instance manager, and it seems it feet well to my needs, but I have a little problem.
In my paged entity system, I made a class "EntityPagedWorldSection" which inherit from "PagedWorldSection", and takes a "Entity" as parameter (pretty much like pagedTerrain, but with an entity instead of TerrainGroup). I next have 2 PageContent inherited classes, EntityPageContentImpostor for far tiles, and EntityPageContentBatched for near tiles. Obviously, i want to use instanced entities for the near tiles.

But as it's written in the documentation:
It's wasteless to create two InstanceManagers with the same mesh reference, instancing technique and instances per batch count.
So, my plan was to set up the InstanceManager for a mesh when creating the "EntityPagedWorldSection" object, and then call "createInstancedEntity" function in the prepare process of each "EntityPageContentBatched" instance.
My problem is ideally, I should set up one batch per "EntityPageContentBatched" instance, but batches creation process seems to be quite "rigid" in InstanceManager": a newly batch is created when the first one is "full of entities". I can't create an instanced entity in a specified batch neither.

The other solution is to create an instanceManager per "EntityPageContentBatched" instance, but as far as i understand how it works, it shall not be very efficient.
Does anybody saw another method?

<edit>My real question is, does modifying the createInstancedGeometry function this way

Code: Select all

InstancedEntity* InstanceManager::createInstancedEntity( const String &materialName, InstanceBatch* targetInstanceBatch=0)
means anything? If no targetInstanceBatch is provided, then the actual behavior stay the same, but It's possible to force the batch on which the instanced entity will be added. What do you think of that?</edit>
Nodrev
Gremlin
Posts: 193
Joined: Fri Jan 25, 2008 6:55 pm
Location: Nantes / France
x 17

Re: New InstanceManager: Instancing done the right way

Post by Nodrev »

For information, finally, I just went the easy way, ie one InstanceManager per subentity on every page.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

I just saw your post. Sorry for the late answer.

Using multiple InstanceManagers will waste video RAM, to be exact. If memory or bandwidth isn't your bottleneck you'll probably won't see much difference.

Anyway, alternatives I can think of:
  • Inherit InstanceManager and manage it yourself by allocating one yourself. This way you can have full control on how batches are created/destroyed. The only "gotcha" is that InstanceManager::_updateDirtyBatches needs to be called in each _renderScene() call. To overcome that, we could add something like SceneManager::registerCustomInstanceManager if you're interested.
  • At loading time, call getMaxOrBestNumInstancesPerBatch and then InstanceManager::createInstancedEntity as many times as necessary (i.e. you need 5 batches? Call it numInstancePerBatch * 5 times) and then remove all of them. Later use getInstanceBatchIterator()->createInstancedEntity() to control on which batch the entity gets created.
  • Same as above, but instead of calling createInstancedEntity so many times, we add a function "preallocateBatches( numBatches ). This is possibly the most flexible, efficient and cleanest solution.
Cheers
Dark Sylinc
Nodrev
Gremlin
Posts: 193
Joined: Fri Jan 25, 2008 6:55 pm
Location: Nantes / France
x 17

Re: New InstanceManager: Instancing done the right way

Post by Nodrev »

Ok, thanks, I will keep that idea in mind, but for the moment i'm wasting time to make paged component working, it's a lot of retro-engineering, as nothing is documented, and it seems that it contains some bugs :/
I will stay "the easy way" for the moment, while i'm debugging paged compo, but thanks for the hint, it'll help me when I'll just have to optimize it.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

@m2codeGEN:
I added your changes involving compiler warnings and double to float conversions. Thanks for the patches.
However, as I said I would do, the "local" transformation enhancement were left out.

I manually typed the code changes. Hg seems to hate me. It refuses to apply patches, don't know why.
User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
x 126

Re: New InstanceManager: Instancing done the right way

Post by masterfalcon »

@dark_sylinc Would you be able to close the patches on the tracker too?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5433
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1341

Re: New InstanceManager: Instancing done the right way

Post by dark_sylinc »

Last time I checked, I'm not added in SF.Net so I don't have those access privileges.
Edit: My sf.net account name is 'darksylinc' (no spaces)