Page 1 of 1
[2.1] Multithreading tutorial questions
Posted: Mon Oct 08, 2018 11:11 am
by Slicky
(1) For physics related code that goes in the logic thread in for example logicgamestate. How do I get access to a mesh there? The reason is I wan't to parse the mesh to create a physics object. Meshes are loaded and created in the graphics thread. Would I have to recreate similar in the logic thread?
(2) How do I best parse a v2 mesh to get the geometry? I've been looking at OgreAssimp and the mesh classes in Ogre. Should I try and use code from there? I don't see any helper functions.
Thanks
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 4:10 pm
by Hrenli
I don't have direct answers. But indirect answer to 1st question might be: in most cases people want different representations of visible and collision meshes to make physics engine work easier. Which means you try to describe your objects in simpler primitives (using spheres/cubes/bounding objects or reading more complex predefined meshes from a file). In other words - don't try to read the visible mesh in the physics thread.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 4:57 pm
by Slicky
Yes I do some physics with primitives but also use some mesh based items that can't be duplicated easily with primitives.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 5:59 pm
by al2950
Slicky wrote: ↑Mon Oct 08, 2018 11:11 am
(1) For physics related code that goes in the logic thread in for example logicgamestate. How do I get access to a mesh there? The reason is I wan't to parse the mesh to create a physics object. Meshes are loaded and created in the graphics thread. Would I have to recreate similar in the logic thread?
Yes, this is irritating! Part of me thinks that a render engine should deal with the graphics thread behind the scenes and provide an API that can be directly used from the logic thread. But some believe this is more game engine territory. Passing data via a lock-free queues is not too hard, but it means you will need to request data, and either wait for it, or process asynchronously.
Slicky wrote: ↑Mon Oct 08, 2018 11:11 am
(2) How do I best parse a v2 mesh to get the geometry? I've been looking at OgreAssimp and the mesh classes in Ogre. Should I try and use code from there? I don't see any helper functions.
This is what I use
Code: Select all
void MeshTools::getMeshInfo(Ogre::MeshPtr mesh,
int &outVertexCount,
Ogre::Vector3* &outVertices,
int &outIndexCount,
Ogre::uint32* &outIndices,
Ogre::Matrix4 transform)
{
//First, we compute the total number of vertices and indices and init the buffers.
unsigned int numVertices = 0;
unsigned int numIndices = 0;
Ogre::Mesh::SubMeshVec::const_iterator subMeshIterator = mesh->getSubMeshes().begin();
while (subMeshIterator != mesh->getSubMeshes().end())
{
Ogre::SubMesh *subMesh = *subMeshIterator;
numVertices += subMesh->mVao[0][0]->getVertexBuffers()[0]->getNumElements();
numIndices += subMesh->mVao[0][0]->getIndexBuffer()->getNumElements();
++subMeshIterator;
}
//allocate memory to the input array reference, handleRequest calls delete on this memory
outVertices = new Ogre::Vector3[numVertices];
outIndices = new Ogre::uint32[numIndices];
outVertexCount = numVertices;
outIndexCount = numIndices;
unsigned int addedIndices = 0;
unsigned int index_offset = 0;
unsigned int subMeshOffset = 0;
//Read submeshes
subMeshIterator = mesh->getSubMeshes().begin();
while (subMeshIterator != mesh->getSubMeshes().end())
{
Ogre::SubMesh *subMesh = *subMeshIterator;
Ogre::VertexArrayObjectArray vaos = subMesh->mVao[0];
if (!vaos.empty())
{
//Get the first LOD level
Ogre::VertexArrayObject *vao = vaos[0];
bool indices32 = (vao->getIndexBuffer()->getIndexType() == Ogre::IndexBufferPacked::IT_32BIT);
const Ogre::VertexBufferPackedVec &vertexBuffers = vao->getVertexBuffers();
Ogre::IndexBufferPacked *indexBuffer = vao->getIndexBuffer();
//request async read from buffer
Ogre::VertexArrayObject::ReadRequestsArray requests;
requests.push_back(Ogre::VertexArrayObject::ReadRequests(Ogre::VES_POSITION));
vao->readRequests(requests);
vao->mapAsyncTickets(requests);
unsigned int subMeshVerticiesNum = requests[0].vertexBuffer->getNumElements();
if (requests[0].type == Ogre::VET_HALF4)
{
for (size_t i = 0; i < subMeshVerticiesNum; ++i)
{
const Ogre::uint16* position = reinterpret_cast<const Ogre::uint16*>(requests[0].data);
Ogre::Vector3 vec;
vec.x = Ogre::Bitwise::halfToFloat(position[0]);
vec.y = Ogre::Bitwise::halfToFloat(position[1]);
vec.z = Ogre::Bitwise::halfToFloat(position[2]);
requests[0].data += requests[0].vertexBuffer->getBytesPerElement();
outVertices[i + subMeshOffset] = transform*vec;
}
}
else if (requests[0].type == Ogre::VET_FLOAT3)
{
for (size_t i = 0; i < subMeshVerticiesNum; ++i)
{
const float* position = reinterpret_cast<const float*>(requests[0].data);
Ogre::Vector3 vec;
vec.x = *position++;
vec.y = *position++;
vec.z = *position++;
requests[0].data += requests[0].vertexBuffer->getBytesPerElement();
outVertices[i + subMeshOffset] = transform*vec;
}
}
else
{
throw Exception("Vertex Buffer type not recognised", "MeshTools::getMeshInfo");
}
subMeshOffset += subMeshVerticiesNum;
vao->unmapAsyncTickets(requests);
////Read index data
if (indexBuffer)
{
Ogre::AsyncTicketPtr asyncTicket = indexBuffer->readRequest(0, indexBuffer->getNumElements());
unsigned int *pIndices = 0;
if (indices32)
{
pIndices = (unsigned*)(asyncTicket->map());
}
else
{
unsigned short *pShortIndices = (unsigned short*)(asyncTicket->map());
pIndices = new unsigned int[indexBuffer->getNumElements()];
for (size_t k = 0; k < indexBuffer->getNumElements(); ++k)
{
pIndices[k] = static_cast<unsigned int>(pShortIndices[k]);
}
}
unsigned int bufferIndex = 0;
for (size_t i = addedIndices; i < addedIndices + indexBuffer->getNumElements(); ++i)
{
outIndices[i] = pIndices[bufferIndex] + index_offset;
++bufferIndex;
}
addedIndices += indexBuffer->getNumElements();
if (!indices32) delete[] pIndices;
asyncTicket->unmap();
}
index_offset += vertexBuffers[0]->getNumElements();
}
subMeshIterator++;
}
}
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 7:45 pm
by dark_sylinc
Hi!
You need to do a ping pong (i.e. with via queueSendMessage): Logic requests to Graphics, Graphics downloads mesh data from GPU, Graphics sends a pointer back to Logic thread.
I never got the time to show how to do this because while it's not super complex, it's not trivial either.
The code al2950 shows how to download the mesh data from GPU.
Alternatively IIRC user 0xc0deface has had some moderate success hacking the Mesh loading system to use the NULLRenderSystem, and thus be able to load the mesh from the Logic thread. But I don't know how well that works.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 8:50 pm
by Slicky
I was thinking along the same lines today of adding another message back to the logic thread. It seems kind of funny since the logic thread requested it but since I realised it isn't an item until the graphics thread creates it needs a message back.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 8:57 pm
by Slicky
al2950 wrote: ↑Tue Oct 09, 2018 5:59 pm
This is what I use
Code: Select all
void MeshTools::getMeshInfo(Ogre::MeshPtr mesh,
int &outVertexCount,
Ogre::Vector3* &outVertices,
int &outIndexCount,
Ogre::uint32* &outIndices,
Ogre::Matrix4 transform)
{
//First, we compute the total number of vertices and indices and init the buffers.
unsigned int numVertices = 0;
unsigned int numIndices = 0;
Ogre::Mesh::SubMeshVec::const_iterator subMeshIterator = mesh->getSubMeshes().begin();
while (subMeshIterator != mesh->getSubMeshes().end())
{
Ogre::SubMesh *subMesh = *subMeshIterator;
numVertices += subMesh->mVao[0][0]->getVertexBuffers()[0]->getNumElements();
numIndices += subMesh->mVao[0][0]->getIndexBuffer()->getNumElements();
++subMeshIterator;
}
//allocate memory to the input array reference, handleRequest calls delete on this memory
outVertices = new Ogre::Vector3[numVertices];
outIndices = new Ogre::uint32[numIndices];
outVertexCount = numVertices;
outIndexCount = numIndices;
unsigned int addedIndices = 0;
unsigned int index_offset = 0;
unsigned int subMeshOffset = 0;
//Read submeshes
subMeshIterator = mesh->getSubMeshes().begin();
while (subMeshIterator != mesh->getSubMeshes().end())
{
Ogre::SubMesh *subMesh = *subMeshIterator;
Ogre::VertexArrayObjectArray vaos = subMesh->mVao[0];
if (!vaos.empty())
{
//Get the first LOD level
Ogre::VertexArrayObject *vao = vaos[0];
bool indices32 = (vao->getIndexBuffer()->getIndexType() == Ogre::IndexBufferPacked::IT_32BIT);
const Ogre::VertexBufferPackedVec &vertexBuffers = vao->getVertexBuffers();
Ogre::IndexBufferPacked *indexBuffer = vao->getIndexBuffer();
//request async read from buffer
Ogre::VertexArrayObject::ReadRequestsArray requests;
requests.push_back(Ogre::VertexArrayObject::ReadRequests(Ogre::VES_POSITION));
vao->readRequests(requests);
vao->mapAsyncTickets(requests);
unsigned int subMeshVerticiesNum = requests[0].vertexBuffer->getNumElements();
if (requests[0].type == Ogre::VET_HALF4)
{
for (size_t i = 0; i < subMeshVerticiesNum; ++i)
{
const Ogre::uint16* position = reinterpret_cast<const Ogre::uint16*>(requests[0].data);
Ogre::Vector3 vec;
vec.x = Ogre::Bitwise::halfToFloat(position[0]);
vec.y = Ogre::Bitwise::halfToFloat(position[1]);
vec.z = Ogre::Bitwise::halfToFloat(position[2]);
requests[0].data += requests[0].vertexBuffer->getBytesPerElement();
outVertices[i + subMeshOffset] = transform*vec;
}
}
else if (requests[0].type == Ogre::VET_FLOAT3)
{
for (size_t i = 0; i < subMeshVerticiesNum; ++i)
{
const float* position = reinterpret_cast<const float*>(requests[0].data);
Ogre::Vector3 vec;
vec.x = *position++;
vec.y = *position++;
vec.z = *position++;
requests[0].data += requests[0].vertexBuffer->getBytesPerElement();
outVertices[i + subMeshOffset] = transform*vec;
}
}
else
{
throw Exception("Vertex Buffer type not recognised", "MeshTools::getMeshInfo");
}
subMeshOffset += subMeshVerticiesNum;
vao->unmapAsyncTickets(requests);
////Read index data
if (indexBuffer)
{
Ogre::AsyncTicketPtr asyncTicket = indexBuffer->readRequest(0, indexBuffer->getNumElements());
unsigned int *pIndices = 0;
if (indices32)
{
pIndices = (unsigned*)(asyncTicket->map());
}
else
{
unsigned short *pShortIndices = (unsigned short*)(asyncTicket->map());
pIndices = new unsigned int[indexBuffer->getNumElements()];
for (size_t k = 0; k < indexBuffer->getNumElements(); ++k)
{
pIndices[k] = static_cast<unsigned int>(pShortIndices[k]);
}
}
unsigned int bufferIndex = 0;
for (size_t i = addedIndices; i < addedIndices + indexBuffer->getNumElements(); ++i)
{
outIndices[i] = pIndices[bufferIndex] + index_offset;
++bufferIndex;
}
addedIndices += indexBuffer->getNumElements();
if (!indices32) delete[] pIndices;
asyncTicket->unmap();
}
index_offset += vertexBuffers[0]->getNumElements();
}
subMeshIterator++;
}
}
Thanks for this. I will take a look tomorrow. I assume this is v2 since there are two Ogres.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 09, 2018 10:50 pm
by al2950
Slicky wrote: ↑Tue Oct 09, 2018 8:57 pm
al2950 wrote: ↑Tue Oct 09, 2018 5:59 pm
This is what I use
Code: Select all
void MeshTools::getMeshInfo(Ogre::MeshPtr mesh,
int &outVertexCount,
Ogre::Vector3* &outVertices,
int &outIndexCount,
Ogre::uint32* &outIndices,
Ogre::Matrix4 transform)
{
//First, we compute the total number of vertices and indices and init the buffers.
unsigned int numVertices = 0;
unsigned int numIndices = 0;
Ogre::Mesh::SubMeshVec::const_iterator subMeshIterator = mesh->getSubMeshes().begin();
while (subMeshIterator != mesh->getSubMeshes().end())
{
Ogre::SubMesh *subMesh = *subMeshIterator;
numVertices += subMesh->mVao[0][0]->getVertexBuffers()[0]->getNumElements();
numIndices += subMesh->mVao[0][0]->getIndexBuffer()->getNumElements();
++subMeshIterator;
}
//allocate memory to the input array reference, handleRequest calls delete on this memory
outVertices = new Ogre::Vector3[numVertices];
outIndices = new Ogre::uint32[numIndices];
outVertexCount = numVertices;
outIndexCount = numIndices;
unsigned int addedIndices = 0;
unsigned int index_offset = 0;
unsigned int subMeshOffset = 0;
//Read submeshes
subMeshIterator = mesh->getSubMeshes().begin();
while (subMeshIterator != mesh->getSubMeshes().end())
{
Ogre::SubMesh *subMesh = *subMeshIterator;
Ogre::VertexArrayObjectArray vaos = subMesh->mVao[0];
if (!vaos.empty())
{
//Get the first LOD level
Ogre::VertexArrayObject *vao = vaos[0];
bool indices32 = (vao->getIndexBuffer()->getIndexType() == Ogre::IndexBufferPacked::IT_32BIT);
const Ogre::VertexBufferPackedVec &vertexBuffers = vao->getVertexBuffers();
Ogre::IndexBufferPacked *indexBuffer = vao->getIndexBuffer();
//request async read from buffer
Ogre::VertexArrayObject::ReadRequestsArray requests;
requests.push_back(Ogre::VertexArrayObject::ReadRequests(Ogre::VES_POSITION));
vao->readRequests(requests);
vao->mapAsyncTickets(requests);
unsigned int subMeshVerticiesNum = requests[0].vertexBuffer->getNumElements();
if (requests[0].type == Ogre::VET_HALF4)
{
for (size_t i = 0; i < subMeshVerticiesNum; ++i)
{
const Ogre::uint16* position = reinterpret_cast<const Ogre::uint16*>(requests[0].data);
Ogre::Vector3 vec;
vec.x = Ogre::Bitwise::halfToFloat(position[0]);
vec.y = Ogre::Bitwise::halfToFloat(position[1]);
vec.z = Ogre::Bitwise::halfToFloat(position[2]);
requests[0].data += requests[0].vertexBuffer->getBytesPerElement();
outVertices[i + subMeshOffset] = transform*vec;
}
}
else if (requests[0].type == Ogre::VET_FLOAT3)
{
for (size_t i = 0; i < subMeshVerticiesNum; ++i)
{
const float* position = reinterpret_cast<const float*>(requests[0].data);
Ogre::Vector3 vec;
vec.x = *position++;
vec.y = *position++;
vec.z = *position++;
requests[0].data += requests[0].vertexBuffer->getBytesPerElement();
outVertices[i + subMeshOffset] = transform*vec;
}
}
else
{
throw Exception("Vertex Buffer type not recognised", "MeshTools::getMeshInfo");
}
subMeshOffset += subMeshVerticiesNum;
vao->unmapAsyncTickets(requests);
////Read index data
if (indexBuffer)
{
Ogre::AsyncTicketPtr asyncTicket = indexBuffer->readRequest(0, indexBuffer->getNumElements());
unsigned int *pIndices = 0;
if (indices32)
{
pIndices = (unsigned*)(asyncTicket->map());
}
else
{
unsigned short *pShortIndices = (unsigned short*)(asyncTicket->map());
pIndices = new unsigned int[indexBuffer->getNumElements()];
for (size_t k = 0; k < indexBuffer->getNumElements(); ++k)
{
pIndices[k] = static_cast<unsigned int>(pShortIndices[k]);
}
}
unsigned int bufferIndex = 0;
for (size_t i = addedIndices; i < addedIndices + indexBuffer->getNumElements(); ++i)
{
outIndices[i] = pIndices[bufferIndex] + index_offset;
++bufferIndex;
}
addedIndices += indexBuffer->getNumElements();
if (!indices32) delete[] pIndices;
asyncTicket->unmap();
}
index_offset += vertexBuffers[0]->getNumElements();
}
subMeshIterator++;
}
}
Thanks for this. I will take a look tomorrow. I assume this is v2 since there are two Ogres.
Yes, I have a v1 as well, but that is taken pretty much directly from the wiki
Re: [2.1] Multithreading tutorial questions
Posted: Wed Oct 10, 2018 11:24 am
by Slicky
I am working away on the ping pong messaging. I thought it wasn't going to be that hard. I am getting my custom message but the data is corrupted. I thought maybe it was a casting error but after fiddling around I can't resolve it yet. The cge shouldn't be out of scope since it a pointer to an object that has just been created.
In:
Code: Select all
void GraphicsSystem::gameEntityAdded(const GameEntityManager::CreatedGameEntity *cge)
{
I added:
Code: Select all
const void* ptr = reinterpret_cast<const void*>(cge);
this->queueSendMessage(mLogicSystem, Mq::GAME_MESH_CREATED, ptr);
In:
Code: Select all
void LogicSystem::processIncomingMessage( Mq::MessageId messageId, const void *data )
{
The message is received but when I cast it back it is garbled data when I debug on the reinterpret_cast line.
Code: Select all
case Mq::GAME_MESH_CREATED:
{
const GameEntityManager::CreatedGameEntity* cge = reinterpret_cast<const GameEntityManager::CreatedGameEntity*>(data);
mCurrentGameState->MeshCreated(cge );
}
Any ideas?
Re: [2.1] Multithreading tutorial questions
Posted: Wed Oct 10, 2018 11:55 am
by al2950
gameEntityAdded is called from GraphicsSystem::processIncomingMessage which recieves a pointer to the data in the 'messageQueue'. However you pass that pointer back into the 'messageQueue' to go back the other way. However the data that that data points will be deleted (Actually it does not get deleted, but it will most likely be overwritten by another message.
Its a little difficult to explain as you are try to pass by reference an object that is held by the message queue through the message queue, but the message queue overwrites the reference before the message queue processes the next message...
But to fix your problem , change this
Code: Select all
const void* ptr = reinterpret_cast<const void*>(cge);
this->queueSendMessage(mLogicSystem, Mq::GAME_MESH_CREATED, ptr);
to this (basically dont pass a pointer, pass in the data by value, and the contents of the GameEntity will copied to the message queue)
Code: Select all
this->queueSendMessage(mLogicSystem, Mq::GAME_MESH_CREATED, *cge);
Re: [2.1] Multithreading tutorial questions
Posted: Wed Oct 10, 2018 12:06 pm
by Slicky
Excellent - that worked. Thanks.
Onward... to the mesh parsing.
Re: [2.1] Multithreading tutorial questions
Posted: Wed Oct 10, 2018 2:11 pm
by Slicky
Further updates:
(1) I have realised that passing the cge to the logics thread isn't going to work because when I try and get the meshptr that kind of operation can only be done in the graphics thread.
(2) Given the above I'm planning on just passing the relevant data to the logic thread. The mesh parsing code you gave me is great. I have hit one snag. I think I need normal data. Do I just need work out how to expand the utility code to get normals?
Re: [2.1] Multithreading tutorial questions
Posted: Wed Oct 10, 2018 3:01 pm
by al2950
Slicky wrote: ↑Wed Oct 10, 2018 2:11 pm
(2) Given the above I'm planning on just passing the relevant data to the logic thread. The mesh parsing code you gave me is great. I have hit one snag. I think I need normal data. Do I just need work out how to expand the utility code to get normals?
Yes see this line
Code: Select all
requests.push_back(Ogre::VertexArrayObject::ReadRequests(Ogre::VES_POSITION));
I only request VES_POSITION, you just need to add VES_NORMAL. However they may be in QTangent format which may confuse you. I dont know too much about this so you may need some extra help!
Re: [2.1] Multithreading tutorial questions
Posted: Wed Oct 10, 2018 8:33 pm
by dark_sylinc
You can see in OgreSubMesh2.cpp SubMesh::_dearrangeEfficient, how to convert from QTangent to a normal:
Code: Select all
if( itElements->mSemantic == VES_NORMAL && itElements->mType == VET_SHORT4_SNORM )
{
//Dealing with QTangents.
Quaternion qTangent;
const int16 *srcData16 = reinterpret_cast<const int16*>( srcData );
qTangent.x = Bitwise::snorm16ToFloat( srcData16[0] );
qTangent.y = Bitwise::snorm16ToFloat( srcData16[1] );
qTangent.z = Bitwise::snorm16ToFloat( srcData16[2] );
qTangent.w = Bitwise::snorm16ToFloat( srcData16[3] );
float reflection = 1.0f;
if( qTangent.w < 0 )
reflection = -1.0f;
Vector3 vNormal = qTangent.xAxis();
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 30, 2018 4:23 pm
by Slicky
I know this might be more a general C++ question but for example I have passed a pointer to the camera (via a message) that exists in the graphics thread to the logics thread to say throw a ball based upon camera position.
When I try and do that I get a crash in updateview;
Code: Select all
const Vector3& Camera::getDerivedPosition(void) const
{
updateView();
return mDerivedPosition;
}
Is this because it is effectively calling a function in the graphics thread?
I like the concept of threading but this is adding complexity. It is equally possible I am missing something. I hope I don't have to keep adding ping pong messages.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 30, 2018 5:02 pm
by al2950
Slicky wrote: ↑Tue Oct 30, 2018 4:23 pm
Is this because it is effectively calling a function in the graphics thread?
Quite possibly, although could not say for certain.
You sadly need to have a stricter boundary between Ogre and your logic code. This does mean duplicating a lot of code. For your example above you should have your own representation of a Camera which is used in your logic thread, and the only thing that should interact with Ogre::Camera is the Graphics thread and the Graphics thread side of your message queue. You should also have your own representation of SceneNodes and other stuff! This does begin to fit nicely into an ECS (Entity Component System) if you have not come across it before
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 30, 2018 5:11 pm
by Slicky
I've enjoyed exploring the multi threaded version but am beginning to wonder if it is going to be too tricky to pass all these items backward and forward. My best guess is that it is a thread issue. I'm "inadvertently" calling updateview by requesting the position from the logic thread.
I might have to switch back to single threaded Ogre. I'm not 100% sure yet. When I created a server setup for another app I replicated a lot of client code but it was more cut and paste. This is within the same app.
Thanks for the response. I haven't heard of ECS I will go and do some reading.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 30, 2018 7:17 pm
by dark_sylinc
The crash sounds weird because I don't think it should crash (unless it's an assert).
However it is a race condition so that's definitely not a good practice.
Like al2950 said, the problem is that you need a stricter boundary between Ogre and your application. You should aim at sending the position & orientation to the logic system for whatever calculation you need (such as aiming in an FPS, selecting an object in 3rd person game, selecting units in an RTS) rather than sending the Camera pointer directly.
If you're struggling with the threading then it is wise to make it single threaded. But regardless, you should aim at having a stricter boundary between Ogre and your application, which is independent from threading.
Re: [2.1] Multithreading tutorial questions
Posted: Tue Oct 30, 2018 7:37 pm
by Slicky
Thanks for the reply. I should probably carry on for a bit longer and try. Maybe there won't be much more need for messaging but I can't be sure yet. Like you said I should probably focus more on data than pointers except that means you could be getting data when you don't really need it.
I will try and think through the separation. I have been trying to separate logic from rendering but often they depend on similar items.