[SOLVED]Dynamic texture updated by thread

[thejesusbr]

Hi, everyone!

I am coding a app using Ogre where I need to update a dynamic texture. The update speed will be ~30fps, but I need that the main loop runs at ~120fps. I have a manager class that take the images from a camera, do some processing with OpenCV and so, update the texture. The texture is created by the core application and ist poiter is passed to the manager class.

If I call for the updateTexture method from de manager class inside the main loop, my fps is bound to the update speed of the manager class. I not want this. I need to run the main loop as fast as I can, but the texture will be updated at ~30fps.

I'm thinking on using a trhead to do de updating of the texture, but I am not so sure about what I have to do, so I'm seeking sugestions. Right now, I'm trying to implement threads using ptypes. I'm working on Linux now, but the code will be migrated to Windows, but keeping the portability.

The code of the updateTexture method:

Code: Select all

void VideoManager::updateTexture()
{
	if (videoCapture->isOpened())
	{
		videoCapture->grab();
		videoCapture->retrieve(frame, CV_CAP_OPENNI_BGR_IMAGE);
		videoCapture->retrieve(depthMap, CV_CAP_OPENNI_DEPTH_MAP);
		videoCapture->retrieve(disparityMap, CV_CAP_OPENNI_DISPARITY_MAP);

	}
	else return;

	// Get the pixel buffer
	mPixelBuffer = texturePtr->getBuffer();
	// Lock the pixel buffer and get a pixel box
	mPixelBuffer->lock(Ogre::HardwareBuffer::HBL_DISCARD); // for best performance use HBL_DISCARD!
	const Ogre::PixelBox& mPixelBox = mPixelBuffer->getCurrentLock();
	pDest = static_cast<Ogre::uchar*> (mPixelBox.data);
	if (displayMode == RGB)
	{
		pSour = static_cast<Ogre::uchar*> (frame.data);
		if (isMirrored)
		{
			for (size_t j = 0; j < mCapHeight; j++)
			{
				pSour += (mCapWidth - 1) * 3;
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour++; // B
					*pDest++ = *pSour++; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
					pSour -= 6; //2 * canais
				}
				pSour += (mCapWidth + 1) * 3;
			}
		}
		else
		{
			for (size_t j = 0; j < mCapHeight; j++)
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour++; // B
					*pDest++ = *pSour++; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
				}
		}
	}
	else if (displayMode == DEPTH)
	{
		const float scaleFactor = 0.05f;
		cv::Mat show;
		depthMap.convertTo(show, CV_8UC1, scaleFactor);
		pSour = static_cast<Ogre::uchar*> (show.data);
		if (isMirrored)
		{
			for (size_t j = 0; j < mCapHeight; j++)
			{
				pSour += (mCapWidth - 1);
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
					pSour -= 2;
				}
				pSour += (mCapWidth + 1);
			}
		}
		else
		{
			for (size_t j = 0; j < mCapHeight; j++)
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
				}
		}
	}
	else if (displayMode == DISPARITY)
	{
		pSour = static_cast<Ogre::uchar*> (disparityMap.data);

		if (isMirrored)
		{
			for (size_t j = 0; j < mCapHeight; j++)
			{
				pSour += (mCapWidth - 1);
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
					pSour -= 2;
				}
				pSour += (mCapWidth + 1);
			}
		}
		else
		{
			for (size_t j = 0; j < mCapHeight; j++)
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
				}
		}
	}


	// Unlock the pixel buffer
	mPixelBuffer->unlock();
}

And the code of mainLoop method:

Code: Select all

void OgreManager::mainLoop()
{
	Ogre::LogManager::getSingletonPtr()->logMessage("*** Starting Main Loop ***");
	frameTicker->reset();
	while (stateManager->getCurrentState() != SHUTDOWN)
	{
		if (mainWindow->isClosed())
		{
			stateManager->requestStateChange(SHUTDOWN);
		}
		//Input Manager;
		inputManager->capture();

		Ogre::WindowEventUtilities::messagePump();

		if (stateManager->getCurrentState() == GUI)
		{
			//Mostre a GUI
			pGUIRenderer->setRenderingEnabled(true);
		}
		else if (stateManager->getCurrentState() == SIMULATION)
		{
			//Roda a simulação
			pGUIRenderer->setRenderingEnabled(false);
                        if (videoOn) videoManager->updateTexture();
		}
		physicsManager->addStepSimulation(33);
		ogreRoot->renderOneFrame();
		frameTicker->reset();
	}

I will made an update to this post as long as I modify the code to use threading. Someone have sugestions?

QUICK UPDATE:
I encapsulated the updateTexture() method around the execute() method inherited by de pt::thread class from ptypes/pasync lib and it give me a segfault on the mPixelBuffer->unlock() call. Why it happened? I need to create the texture in the same trhead, maybe?

[syedhs]

I have mentioned this in other thread but here is my pseudocode:-
setup

Code: Select all

create two identical textures.
set active_texture_index to 0 (first one).
lock the texture buffer
create thread with pointer bit as the parameter.

FrameListener::frameStart (loop)
[code]
has active texture been filled up?
if yes then
     unlock the texture buffer
     use this texture buffer
     active_texture_index = 1 - active_texture_index
     lock the texture buffer, and pass it to the thread the pointer bit.
end

p/s: You have to be quite comfortable with thread programming, ie you should know about how thread communicate with the parent process (in Windows, specifically you have to know Event, WaitForMultipleObject). And never allow the thread to quit unless the parent process quit too. If the job is finished, then the thread should be idle awaiting for next command.

[thejesusbr]

Thank you for replying, syedhs. I'll do it. Can you post the link for the thread that you mentioned?

[syedhs]

Actually the post in here is much more detail

[thejesusbr]

Let's see if I get it... I will create two identical textures, one in the main thread and another in the video updating thread. So, I'll render the texture that is in the main thread and when the other texture updates, I lock the main thread texture and copy the data from the other thread. That's it?

[thejesusbr]

I managed to solve this by a different aproach. I've made a simple event mechanism based on trhead triggers and a state variable. When a frame arrives, the newFrame flag is setted to true and the thread goes sleep. In the main loop in, another thread, I check the flag. If it's true, I update the texture. In the update method, I set the flag to false and signal the frame grabber thread to wake up. So, the main thread keeps jumping the texture update until the frame arrives again, which causes the flag to be true. It works as I expect. Below, the final code:

The VideoManager code, running the capture process on a background thread

Code: Select all

//Here, goes the code that will be running on a background thread
void VideoManager::execute()
{
	while (!get_signaled())
	{
		printf("Thread Launched\n");
		printf("Capturing...\n");
		if (videoCapture->isOpened())
		{
			videoCapture->grab();
			videoCapture->retrieve(frame, CV_CAP_OPENNI_BGR_IMAGE);
			videoCapture->retrieve(depthMap, CV_CAP_OPENNI_DEPTH_MAP);
			videoCapture->retrieve(disparityMap, CV_CAP_OPENNI_DISPARITY_MAP);
                       //Notify the arrival of a new frame
			newFrame = true;
                        //Set a semaphore to put the thread to sleep
			copyTrigger.wait();
		}
		else return;
	}
}
//Here is the code that updates the texture.
void VideoManager::updateTexture()
{
	// Get the pixel buffer
	Ogre::HardwarePixelBufferSharedPtr mPixelBuffer;
	mPixelBuffer = texturePtr->getBuffer();
	printf("Taking buffer\n");
	// Lock the pixel buffer and get a pixel box
	mPixelBuffer->lock(Ogre::HardwareBuffer::HBL_DISCARD); // for best performance use HBL_DISCARD!
	printf("Buffer locked\n");
	const Ogre::PixelBox& mPixelBox = mPixelBuffer->getCurrentLock();
	printf("Box taken\n");
	pDest = static_cast<Ogre::uchar*> (mPixelBox.data);
	printf("pDest set\n");
	if (displayMode == RGB)
	{
		pSour = static_cast<Ogre::uchar*> (frame.data);
		if (isMirrored)
		{
			for (size_t j = 0; j < mCapHeight; j++)
			{
				pSour += (mCapWidth - 1) * 3;
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour++; // B
					*pDest++ = *pSour++; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
					pSour -= 6; //2 * canais
				}
				pSour += (mCapWidth + 1) * 3;
			}
		}
		else
		{
			for (size_t j = 0; j < mCapHeight; j++)
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour++; // B
					*pDest++ = *pSour++; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
				}
		}
	}
	else if (displayMode == DEPTH)
	{
		const float scaleFactor = 0.05f;
		cv::Mat show;
		depthMap.convertTo(show, CV_8UC1, scaleFactor);
		pSour = static_cast<Ogre::uchar*> (show.data);
		if (isMirrored)
		{
			for (size_t j = 0; j < mCapHeight; j++)
			{
				pSour += (mCapWidth - 1);
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
					pSour -= 2;
				}
				pSour += (mCapWidth + 1);
			}
		}
		else
		{
			for (size_t j = 0; j < mCapHeight; j++)
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
				}
		}
	}
	else if (displayMode == DISPARITY)
	{
		pSour = static_cast<Ogre::uchar*> (disparityMap.data);

		if (isMirrored)
		{
			for (size_t j = 0; j < mCapHeight; j++)
			{
				pSour += (mCapWidth - 1);
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
					pSour -= 2;
				}
				pSour += (mCapWidth + 1);
			}
		}
		else
		{
			for (size_t j = 0; j < mCapHeight; j++)
				for (size_t i = 0; i < mCapWidth; i++)
				{
					*pDest++ = *pSour; // B
					*pDest++ = *pSour; // G
					*pDest++ = *pSour++; // R
					*pDest++ = 0; // A
				}
		}
	}
	printf("Buffer copied\n");
	// Unlock the pixel buffer
	mPixelBuffer->unlock();
	printf("Buffer unlocked\n");
       //In the end of the updating process, notify the main thread that the data is up-to-date
	newFrame = false;
       //Signals the semaphore to wake up the frame grabber. So, the threads waits until a
       //new frame arrives from the cam to notify the main thread again
	copyTrigger.post();
}

The main loop running on the main thread:

Code: Select all

void OgreManager::mainLoop()
{
	Ogre::LogManager::getSingletonPtr()->logMessage("*** Starting Main Loop ***");
	frameTicker->reset();
	while (stateManager->getCurrentState() != SHUTDOWN)
	{
		if (mainWindow->isClosed())
		{
			stateManager->requestStateChange(SHUTDOWN);
		}
		//Input Manager;
		inputManager->capture();

		Ogre::WindowEventUtilities::messagePump();

		if (stateManager->getCurrentState() == GUI)
		{
			//Mostre a GUI
			pGUIRenderer->setRenderingEnabled(true);
		}
		else if (stateManager->getCurrentState() == SIMULATION)
		{
			//Roda a simulação
			pGUIRenderer->setRenderingEnabled(false);
			//Each loop, check if a new frame arrives. If true, update the texture
			if (videoOn && videoManager->getNewFrame()) videoManager->updateTexture();
		}
		physicsManager->addStepSimulation(33);
		ogreRoot->renderOneFrame();
		frameTicker->reset();
	}
	Ogre::LogManager::getSingletonPtr()->logMessage("*** Application shuting down... ***");
	ogreRoot->shutdown();
}

The frame rate drops too low, leaving it pratically imperceptible, since the copy is made on a separate thread from the updating. Very good. It not the final solution that I want (updating in background), but have solve the problem, anyway.
Syedhs, thanks for the reply, it helped a lot! Next step, set a independent timer to update the physics... Quite easy!