Scene switching and terrain...

[slapin]

Hi, all!
I need to temporarily switch scenes with Ogre so to implement "magic" interiors which do not completely match exteriors,
and some other needs. But I don't want to touch heavy machinery which is behind terrain to remove it and re-create again (as it is fragile and source for a lot of trouble already). Can I create alternate scene and run them both at the same time and just switch camera? Or is there some easier solution?
The problem is generally the terrain task system which either ends endlessly or crashes because data it wants no longer exists and there is no way to tear down terrain safely...

[sercero]

I think people use portals to solve that problem.

[slapin]

please elaborate?

[slapin]

In Godot I'm used to the following scenario - I have simulated exterior world which is open world and one can walk around anywhere,
however once player enters a door the exterior world is unloaded and completely new scene is loaded (while retaining some global data). However exactly this scenario is not appropriate to my current architecture as Terrain module is not too happy about being suddenly unloaded so it would be much better to
do some different approach but I can' find one yet. Basically I need interiors to vaguely resemble exteriors to be believable but not exactly match exteriors as they need to be slightly larger and have some game design shenanigans implemented. So I wonder how it is generally done and what are my options?
Any help is appreciated.

[slapin]

Found that I can switch main window viewport's camera to different cameras in another scenes.

[slapin]

Also found about PCZ SceneManager, but the setup is rather complex...

[slapin]

I think I will stick to camera switching as PCZ SceneManager is too complex to set up for me at the moment...

[rpgplayerrobin]

However exactly this scenario is not appropriate to my current architecture as Terrain module is not too happy about being suddenly unloaded

So the entire issue is actually that you have not yet found a way to correctly unload the terrain you have created?
I am not using the Ogre terrain so I could not help you there, but if I would be you I would dig deep into how to actually unload it correctly.
One thing I can think of is that some background tasks might still be running on it, making it harder to unload instantly?

[slapin]

However exactly this scenario is not appropriate to my current architecture as Terrain module is not too happy about being suddenly unloaded

So the entire issue is actually that you have not yet found a way to correctly unload the terrain you have created?
I am not using the Ogre terrain so I could not help you there, but if I would be you I would dig deep into how to actually unload it correctly.
One thing I can think of is that some background tasks might still be running on it, making it harder to unload instantly?

"my" terrain is based on Endless World example with custom height map code, so not much there...
yes, pending tasks tend to read just deleted data so it is kind of impossible to unload it at current stage.
it is kind of impossible by design and changing the implementation will require a lot of head scratching...
Digging deep would require some way to flush current task queue and prevent creating new tasks and there are no handles for that,
it is async out of control... According to valgrind it is crashy enough to just suddenly move camera too far from previous point as it
produce too many tasks and if you do it fast enough some tasks will access some wrong memory...
Need sync points but that sounds like huge job to implement...

[slapin]

Finally got major showstopper for camera switching - everything crashes with assert from ImGUI

Code: Select all

src/ogre-build/imgui-1.91.9b/imgui.cpp:10227: void ImGui::ErrorCheckNewFrameSanityChecks(): Assertion `(g.FrameCount == 0 || g.FrameCountEnded == g.FrameCount) && "Forgot to call Render() or EndFrame() at the end of the previous frame?"' failed.

which basically means that what I want to do is simply impossible now with my state of affairs

[slapin]

Looks like switching cameras in my case is not an option, so it looks like I will have to keep open world scene and replace building's exterior with loaded interior in place, but I will have to check and free space for that... PCZ SceneManager looks attractive but I'm afraid it will be dead end as everything else, so I'd stick to simple things...

[paroj]

However exactly this scenario is not appropriate to my current architecture as Terrain module is not too happy about being suddenly unloaded

So the entire issue is actually that you have not yet found a way to correctly unload the terrain you have created?
I am not using the Ogre terrain so I could not help you there, but if I would be you I would dig deep into how to actually unload it correctly.
One thing I can think of is that some background tasks might still be running on it, making it harder to unload instantly?

"my" terrain is based on Endless World example with custom height map code, so not much there...
yes, pending tasks tend to read just deleted data so it is kind of impossible to unload it at current stage.
it is kind of impossible by design and changing the implementation will require a lot of head scratching...
Digging deep would require some way to flush current task queue and prevent creating new tasks and there are no handles for that,
it is async out of control... According to valgrind it is crashy enough to just suddenly move camera too far from previous point as it
produce too many tasks and if you do it fast enough some tasks will access some wrong memory...
Need sync points but that sounds like huge job to implement...

you can disable Threading in the Terrain component or globally via the Ogre compile flag. Also you could just hide the terrain via visibility masks. Of course this will not help you with memory. It should be safe to unload the terrain textures though, which should consume the bulk of the memory.

[slapin]

Well, I would not sacrifice performance so much as it more or less works in normal mode with slow enough moving speed, the concern is editor with teleporting and chunk invalidation, so I would prefer less radical solution.
I'm just overwhelmed with things like this now and need some steady path one bite at time...

[rpgplayerrobin]

So the entire issue is actually that you have not yet found a way to correctly unload the terrain you have created?
I am not using the Ogre terrain so I could not help you there, but if I would be you I would dig deep into how to actually unload it correctly.
One thing I can think of is that some background tasks might still be running on it, making it harder to unload instantly?

"my" terrain is based on Endless World example with custom height map code, so not much there...
yes, pending tasks tend to read just deleted data so it is kind of impossible to unload it at current stage.
it is kind of impossible by design and changing the implementation will require a lot of head scratching...
Digging deep would require some way to flush current task queue and prevent creating new tasks and there are no handles for that,
it is async out of control... According to valgrind it is crashy enough to just suddenly move camera too far from previous point as it
produce too many tasks and if you do it fast enough some tasks will access some wrong memory...
Need sync points but that sounds like huge job to implement...

you can disable Threading in the Terrain component or globally via the Ogre compile flag. Also you could just hide the terrain via visibility masks. Of course this will not help you with memory. It should be safe to unload the terrain textures though, which should consume the bulk of the memory.

Isn't there some way to just destroy the terrain in a safe way?
With std::future you can basically force it to end and then loop and wait until it is done, is there no similar thing for the terrain that he can use? Because it seems like a very simple thing to just unload/destroy the terrain.

Otherwise, if I were you slapin, I would completely remake the terrain code from scratch, because it sounds to be very flawed if it simply cannot do that or handle teleportation of the camera across far distances.

[paroj]

Otherwise, if I were you slapin, I would completely remake the terrain code from scratch, because it sounds to be very flawed if it simply cannot do that or handle teleportation of the camera across far distances.

remaking from scratch is a bit radical, if it does everything but that, no? "It cannot do that" mostly means it is not implemented or the implementation has bugs..

[slapin]

Pasically I need 3 features

1. - force unload chunk/force load chunk
2. teleport
3. teardown.
   Which basically looks like 2 functions waitWhileAllTerrainTasksCompleteAndDisableLocked()(Aka freeze()) which would create completion variable with background thread which should complete all terrain tasks and freeze terrain operations (blocking main thread) and second enableAndWaitUntilFullySettled()(AKA unfreeze()) which will enable operation, force all tasks creation and make sure all tasks finish (blocking main (or the one which runs it) thread). Everything else will work as intended as with frozen terrain the unload operation will be safe. Or chunk reloading could be implemented as background task,and this task should be run after all other tasks complete or a crash will occur... however for teleport to work 2 these operations are a must. Freeze() before, then set camera position, then unfreeze(). For teardown it is enough to freeze() then deconstruct the usual way...

@paroj what do you think?

[slapin]

I think std::condition_variable should be good for the task...

[paroj]

there already is
https://ogrecave.github.io/ogre/api/lat ... 7fc061558b

however, the implementation might be incomplete.

Ideally you would modify these tasks like:
https://github.com/OGRECave/ogre/blob/4 ... .cpp#L1906

to create futures like:
https://github.com/OGRECave/ogre/blob/4 ... ue.cpp#L54

and then wait for their completion. internally futures use condition variables, so they are just a nicer/ extended interface for those.

[slapin]

waitForDerivedProcesses() affects only tasks in OgreTerrain.cpp and waits for 3 variables to be cleared:

Code: Select all

    void Terrain::waitForDerivedProcesses()
    {
        while (mDerivedDataUpdateInProgress || mGenerateMaterialInProgress || mPrepareInProgress)
        {
            // we need to wait for this to finish
            OGRE_THREAD_SLEEP(50);
            Root::getSingleton().getWorkQueue()->processMainThreadTasks();
        }   
            

}

which is quite naive as it does not prevent new tasks from being created and

Code: Select all

void Terrain::handleResponse(const WorkQueue::Response* res, const WorkQueue* srcQ)
{
...
}

is recursive function which can call itself through other functions and create more tasks which can lead to uncontrollable numbers of tasks in queue,
some of which could be no longer needed... so waitForDerivedProcesses() does not guarantee that the task queue would be empty on exit
and have no any safety guarantees... What is needed is kind of barrier function which would say "now the terrain is frozen and can be safely manipulated" and "unfreeze" which will continue operations and refresh the terrain.

[slapin]

Well, I experimented a bit and made the following PR:

https://github.com/OGRECave/ogre/pull/3527

The overall stability under valgrind drastically increased, teleports work perfectly
under valgrind. Of course that is not complete fix but it gets much better.
However I encountered very rare busyloop-like hang which might mean rarely
tasks do not complete and hang forever (never run so futures never complete).
Please look and any advice is appreciated.

Also I need advice on how to limit formatting in QtCreator to actually modified code
or it is very hard to make PRs otherwise.

[slapin]

Also it seems visual studio do not support std::future properly, so currently the PR won't build on Windows.

[rpgplayerrobin]

I use std::future on Visual Studio 2017 in Windows and it works. What version did you use? Maybe std::future was not supported on your C++ version?

[slapin]

The one which is used on github CI, C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.29.30133/bin/HostX64/x64/cl.exe
I worked it around using std::shared_ptr, see PR, https://github.com/OGRECave/ogre/pull/3527

[slapin]

It is quite common for MSVC to not support any C++ standard completely so I'm not surprised. The containers in MSVC require copy-assignment operator support for emplace, which (the emplace) is specifically required to support move semantics (i.e. no copy-assignment nor copy constructor should be required) by standard. And std::future is not copy-able but moveable by standard.
And all compilers except MSVC support that. The funny part is android's clang specifically considers using std::move with emplace as error...

[slapin]

The code example:

Code: Select all

std::list<std::future<void> > futures;
...
std::future<void> foo()
{
auto task = std::make_shared<std::packaged_task<void()>>([]{return;});
(*task)(); // just a dummy, too lazy to start a thread
return task->get_future();
}
...
void bar()
{
	futures.push_back(std::move(foo()));
	// futures.emplace_back(foo()); /* other option */
}

and check on different compilers if that can interest you