Local Client/Server game : Multi-Thread or Multi-Process?

[Klaim]

So I'm making a game on spare time and it's designed in a client(with --context-dependent-- "view" of the game) / server (with the "true" state of the game) architecture to concentrate on multiplayer and totally separate the state of the game from the player's interpreted view of it.
Event in local (solo) game, I have the server side executed in a separate thread. The client and server only communicate by the network.

Some games did this before, it's not new. But for example, NeverWinter Nights was a game where the server side was in a different process.

I'm writting the server part in a way that is totally separate from the rest of the game, making it easy to move in a dedicated server application/process instead of using threading.
I started by using thread because AFAIK it's simply easier to put in place, so I just thought I'll move the server side in a process later if I find that it's more interesting to have it that way.

For instance I only think that setting the server as a separate process will only make the communication between the main application and the server app "harder" than just manipulating a thread (i'm thinking about a "master" crash making loose control of the server "slave" process, for example), but I'm not sure about that, never manipulated processes that are dependant on other processes before...

So, maybe some people here did have some experience to share trying to figure out the pros and cons of threading vs processing in this kind of case?
I'm not planning to take a decision yet (lot of work to do) but any input about the differences are welcome, just to share knowledge and maybe learn some things about processes and threads that can be a future problem.

[xavier]

I see no reason to have the client and server in the same process. It's far simpler to manage, IMO (and IME) as different processes.

[Klaim]

When you're talking about "simpler", can you be more explicit please? Maybe the memory context separation between processes is of great help?

I'm really noob when it comes to multi-process applications v__v;

[madmarx]

Multi process gives you easier support for exception.
Multi process gives you 'separate' singleton.
Multi process won't give you same type_id.
Apart from that, I find it easier in multi thread to manage thread prorities and CPU dispatching.

Debuging can be easier in multi thread, or not, it depends on how you use it, and what your ide allows. I suppose this could be #1 priority.

Anyway, I have always used boost::interprocess when I needed to share objects between processes, and it works great.

[Klaim]

Hmm I see...

I'll take a look at boost::interprocess, thanks for the precisions about the processes! Good to know.

[xavier]

Multi process gives you easier support for exception.
Multi process gives you 'separate' singleton.
Multi process won't give you same type_id.
Apart from that, I find it easier in multi thread to manage thread prorities and CPU dispatching.

Debuging can be easier in multi thread, or not, it depends on how you use it, and what your ide allows. I suppose this could be #1 priority.

Anyway, I have always used boost::interprocess when I needed to share objects between processes, and it works great.

I find debugging separate processes much simpler than multiple threads when the processes are supposed to be running independently -- such as in a client/server application. Two instances of VS running works just fine for me.

[Klaim]

Today I wanted to see what's the behavior of Visual Studio (2008) in case you're having an application launching a process and when this application end or crashes, you want the process to end too.

To check that I simply made my game launch a process (when you enter a specific command in the game console). It works well but there is an exception thrown followed by a crash (not resolved yet but not important) in the game when I want to end the process (by using another command).

So I just killed the debug mode and started looking at the code throwing exception.

The later I found that the launched process was still going on.

How do you manage this kind of case where you need to be sure that the child process never live after the death of it's parent process, even after a crash of the parent process?

[syedhs]

How do you manage this kind of case where you need to be sure that the child process never live after the death of it's parent process, even after a crash of the parent process?

If you want it to be really bullet-proof, then the usual design I often encountered is to develop another app (lets call it supervisor) which monitors the live of the processes involved. If one app is crashed and the other is not properly notified, then supervisor will inform the other app, etc. Therefore, the responsibility of spawning/destroying processes now rest on the supervisor's shoulder.

On the other hand, if you want to go for simpler design, probably an inter-process message should be sent each other every say 1 second. If this message is not responded, it most probably mean that the other app has crashed.

[Klaim]

Ah yes, some kind of ping. Didn't thought about that.

[madmarx]

If you are french, this also might interest you :
http://cpp.developpez.com/faq/vc/?page=ProcessThread

For your question :
You create a child processes (not a side process), and, in some kind of destructor of an object of your main app, you call the end of the childprocess.
Even if parent crashes, it will call the destructor of the object of your main app, and killing the child processes is possible.
As a side note, if each of your process is a copy of same exe, you can use the same type_id through processes.

[Klaim]

[french] I posted there on the same site : http://www.developpez.net/forums/d88054 ... sus-parent [/french]

You create a child processes (not a side process), and, in some kind of destructor of an object of your main app, you call the end of the childprocess.
Even if parent crashes, it will call the destructor of the object of your main app, and killing the child processes is possible.

That's what I set up the first time but it can't work well because the exception is launched when I want to terminate the child process...It's a right access problem apparently (I'm using boost::process and it throw an exception if the OpenProcess(...) call failed on windows) Anyway I guess it's not a good idea to abruptly kill the process, just make it end itself is a better solution (like whit threads i guess).

[madmarx]

I never used Boost.process. I had too much pain with boost.thread before that .

In pure win32 API call + boost.interprocess this works correctly, but, as you said, it's cleaner to make a call to destruction to it.
What I did previously was simply to set up a bool 'endRequired' in the shared memory space, and test it at each frame of the child process.
That way I never had difficulties to manage the processes.
This is close to what the guy 3DArchi said on developpez.com, but I would'nt use an auto-mutex : I think you will need to call the release explicitely in your main app, before its end.

Example of what I think :
1/something happens.
2/ in the main app destructor.
for each child still alive set the 'endRequired' to false (access is protected by a mutex).
wait sometime
kill those who are still alive after a certain time (display a message..).
end destructor

EDIT
I just checked one of my demo (simultaneous directx /opengl ogre3d).
I coded it like a pork, and it seems fine^^.

Ending code :

Code: Select all

// we check that the window is still opened
			lFini = lFini || mWindow->isClosed() || lSharedData->mStopEverything;
			if(lIsChildProcess)
			{
				lSharedData->mStopEverything = lFini;
			}
		}
	}

	if(!lIsChildProcess)
	{
		TerminateProcess(piProcessInfo.hProcess, 0); 
		//PostThreadMessage(piProcessInfo.dwThreadId, WM_QUIT, 0, 0);
		//WaitForSingleObject(piProcessInfo.hProcess, INFINITE);// si on veut attendre la fin du fils ...
		::CloseHandle(piProcessInfo.hThread);
		::CloseHandle(piProcessInfo.hProcess);
	}
	// quit
	return;
}

The creation was much more tricky:

Code: Select all

// teste si programme de base
	bool lIsChildProcess = __argc>1;

	using namespace boost::interprocess;

	MySharedInfos* lSharedData = NULL;

	if(!lIsChildProcess)
	{
		//mWindow->reposition(205,15);

		// creation de la zone memoire partage. Pour l'instant en READ_WRITE.
		// et lors de la destruction (a la fin de la parenthese), 
		// c'est l'instance suivante en READONLY (qq lignes plus bas) qui va prendre le relai
		// de cette maniere, le process fils controle le pere.
		windows_shared_memory lSharedMemoryCreate(create_only, "My_Shared_Memory", read_write, 1000);
		mapped_region regionCreate(lSharedMemoryCreate, read_write);
		lSharedData = static_cast<MySharedInfos*>(regionCreate.get_address());

		// initialisation du contenu
		MySharedInfos lInfos;
		lInfos.updateFrom(lNodeCamera,!lIsChildProcess);
		memcpy(lSharedData,&lInfos,sizeof(MySharedInfos));
		lSharedData->mChildIsReady = false;
		// je le rouvre en READ ONLY, et je passe mes pointeurs 
		// en STATIC pour qu'ils tiennent jusqu'a la creation du childprocess!
		static windows_shared_memory lSharedMemoryOpen(open_only, "My_Shared_Memory", read_only);
		static mapped_region region(lSharedMemoryOpen, read_only);
		lSharedData = static_cast<MySharedInfos*>(region.get_address());

	}else{
		//mWindow->reposition(204+mWindow->getWidth(),20);
		// trouver ma shared memory (READ_WRITE)
		static windows_shared_memory lSharedMemoryOpen(open_only, "My_Shared_Memory", read_write);
		static mapped_region region(lSharedMemoryOpen, read_write);
		lSharedData = static_cast<MySharedInfos*>(region.get_address());
		lSharedData->mChildIsReady = true;
	}

	// cela va contenir le processus fils.
	// - inutile s'il n'y a pas de fils- mais ca ne coute (presque) rien.
	STARTUPINFO         siStartupInfo;
	PROCESS_INFORMATION piProcessInfo;

	if(!lIsChildProcess)
	{	
		// initialisation de la structure windows.
		memset(&siStartupInfo, 0, sizeof(siStartupInfo));
		memset(&piProcessInfo, 0, sizeof(piProcessInfo));
		siStartupInfo.cb = sizeof(siStartupInfo);
		// je lance le programme "fils".
		//system("extern_OglDirectx.exe en direct x!");

		if(CreateProcess("extern_OglDirectx.exe",
			"en direct x!",0,0,FALSE,
			CREATE_DEFAULT_ERROR_MODE,0,0,                              
			&siStartupInfo,&piProcessInfo) == FALSE)
		{
			// erreur
		}else{
			// attente (je l'ai laisse pour l'exemple)
			//WaitForSingleObject(piProcessInfo.hProcess, INFINITE);
			// destruction
			//::CloseHandle(piProcessInfo.hThread);
			//::CloseHandle(piProcessInfo.hProcess);
		}
	}

It is not production code, but something I wrote quickly for fun...
So the destruction code might not be that perfect. I am pretty confident for the creation part though.
You are lucky comments are in french ^^.

[Klaim]

Yes, I'll try to do something similar by using boost::interprocess when I find time.

[Klaim]

So, I've been "playing" with multi-process several nights, more precisely by making my game launch a child process that don't do much, just to see how Visual Studio behaves in this specific case with debug mode. What I wanted to achieve is simply using a breakpoint on the child process.

What I've found until now is that there don't seem to be any way to make VS Debugger automatically attach to the child process. You can do it manually but you have to redo it each time you debug.
You can use thedebug functions from the win api but it's clearly made to build a debugger with, not to debug with.
I've tried to add an entry in the register to make the debugger launch when the child process launches. It works but when the prompt windows appear it don't let you choose the currently running VS debugger (that was used to debug the parent application). So it just let you open a new Visual Studio with almost no link to the source code.

I've tried the google macro that they uses to debug Chrome, modified a bit for my case, but it don't seem to work well as you need to have the focus on Visual Studio to be able to use it.

The best way I've found so far is to use a debug break at the start of the child process, coupled with a breakpoint in the parent process just after having launched, just to make sure everything is almost in sync -- almost the same than executing all that without debug mode.
The debugbreak have the same effects than having the entry in the register (a windows asks for a debugger to be used to debug the application) but I can now see the source code. I don't have the whole "project" as it's in another Visual Studio instance.

Now, you see how much noob I am on the subject but I really want to understand what practices you all uses to debug child processes? Any trick to make debugging easier than logging?


edit> By the way madmarx I did implement a similar system than the one you showed me, using boost::process (that has not even have been reviewed yet but is really easy to understand/read) and boost::interprocess but it don't work well for now (an "access right" problem it seems). I was so concentrated on finding ways to debug easily that I didn't fixed that yet.

[madmarx]

I really want to understand what practices you all uses to debug child processes?

I develop most of my separate functionnalities in single thread.
I do unit testing, for each of my 'functionnalities' (which can encompass sometimes only one function, sometimes 7 classes). I make sure all the tests are redoable. As I have lots of such tests, most of my data is generated in the tests (which means also the ogre meshes, textures, skeletons, tracking data). The sole time I had to correct a class after its release (<=>production-ready) was for a getter function that I had badly copy/pasted, and not tested... Just reading the code gave me the solution, but I could have added more tests.
After the release of the class, I can integrate it in a multi-thread / multi process system, there is never a single problem (apart from the one I described just above).

Then for debugging the threading / processing system itself, paper and pen is the only real solution. You need to do something that is theorically clean, otherwise you program can work 2 months without a problem and then suddenly fail. Of course you will do some tests too, but that will not prove anything more than 'yet not broken'. Same goes for trying to debug that IMO.

Apart from that, I can switch on/off multithread/process in my programs to check the behaviours in a single thread, but I don't use it to check the threading system itself. I have relatively few 'multiprocessed points' in my programs. For example, each functions of my module can lead to remote code (if the real-module is running on another PC), but the remote call procedure will sequence the call, the module itself is always single thread (and was tested previously).

As my multithreaded/process aspects are homogeneous for all classes (I have several modules, that are purely single thread), I could log the calls automatically, and then redo corresponding calls in a single thread. That would only debug behaviour, and not the treading system itself. But in reality, the whole behaviour/game engin of my project is a module, which makes blocking/ non blocking (if returns void) calls to other modules. As a consequence, I can already test its behaviour (and debug it) in a single thread.

This is how I do. So no multiprocess debug session by me. And robust code from the beginning of the project. And simple code too (simple because tests are simples, a little like a 'one function ogre setup').

Then the real world comes and force me to do that abberation that is called multi process and multithread debugging. I happen to debug in multithread though more than I would like, but that is for code/projects that I did not write (cough .. ogre ... cough). And if you really really want to know how to do in multiprocess... then you can press CTRL + ALT + P during your debugging session and get multiprocess debugging in a single visual studio (with values etc...).

I am aware my first post goes completely against that. Sorry I should not have tell that debuging was the number one priority. But in fact it is if it is your beginnings with multi thread / process. The real world is always near us, and as a consequence this happens to be handy at least one time for each project (less worst solution ).

I guess my post is not readable, lol ?

[Klaim]

I think I've understood your talking: you're saying that currently you're debugging processes separately to be sure they work together later because you eliminate the problems by using specific logics (avoiding locks etc.)?

For the moment I'll just finish my experimentation with the child process to make it exit when the parent process is killed. Once I make that work I'll get back on finishing the game-specific server code that is a separate module and then compare launching this module as a separate thread or process, see the differences in my specific case.

[madmarx]

Well, it's somewhat the idea. But I do it even in monothread programs. I develop functionnalities separately, with simulated inputs. This allow extremely robust code production. Then, I encapsulate my functionnalities in processes/thread, and it works very well if the threading system is stable.

If I really want to do a multiprocess debug session, I just do F5 and then ctr + alt + P to select the other processes to follow.

Concerning the threading logic, there are a few guideline to follow when doing multi-things. E.g : never do a mutex from inside another mutex (unless you have mentally tested all possibilities, and checked that none of them is going to give you a deadlock).

[Zonder]

I would just use TCP/IP regrdless if it's solo game or not it will simplify the internals.

Didn't read all the rest of the thread but to debug the other process you need to configure your soltion to use multiple startup (properties on the solution) this will then attach 2 debuggers (or as many as you want) to the processes