Opinion: Unit Testing

[RedEyeCoder]

I am looking for opinions and asking a lot of questions about unit testing. Up until this point I have remained quite ignorant of unit testing, simply because given what I know of it, think that is is a) heaps of work and b) an anti-agile practice(simply because I think it will harden code, someone have something to say about that?). It also occurs to me that unit tests are just another place for bugs to occur...

The topic has come up numerous times in my work place whether unit testing is appropriate thing for us to do. I am interested to know how many of you do unit testing either at home or in your work place. How rigorous you are with your testing and do you think it is worth while (I assume there are people out there who are forced by their workplace to write unit tests)?

I am also interested in how people run tests on functions that require the application to be in a specific state before they are even usable - do you write a whole bunch of code to get the application into the required state? That seems like a lot of work.

What unit testing framework/api are you using? Do you integrate unit tests into your source control or automatic build systems?

[xavier]

All good code architectures are hierarchical. Code at higher level depends on code at lower levels. The farther down you go (the lower in the dependency stack), the easier it is to "white-box" test. So, your lower-level code is probably developed in a test-driven fashion. At some point in the architecture, you cannot white-box test any longer and have to black-box test, and only for expected conditions. This is fine because your lower-level code has been tested thoroughly, and the code that uses those lower-level systems as a foundation can depend on the fact that the lower level code is "correct" (as far as the term "correct" can be applied to any systems developed outside of formal methods). Therefore, you do not need to test exhaustively this higher-level code.

So to answer your question about having to get the app in a particular state -- you don't. At some point it's no longer "unit testing", it's "system testing", and at some point above that, it's "black box testing" that is done by QA personnel, who hopefully are employing a repeatable suite of functionality and regression tests.

In other words, there is only so much you can automate, and most of that occurs at the lower levels in the code stack.

Besides, "unit testing" an application is an oxymoron.

John Lakos talks about this at length in "Large-Scale C++ Software Design" (a book I highly recommend).

[RedEyeCoder]

Thanks for your fast response. What you say makes good sense.

Using Ogre as an example, what kind of things would you say are ideal for unit testing, and are there unit tests for Ogre? If there are none, why not?

[Klaim]

Things like Vector2,3,4, Quaternion etc are ideal candidates to unit testing. I guess some other basic objects (Node maybe?) would be nice too.

[PolyVox]

For actual unit testing I agree with what has been said. It's easy to test low level libraires such as maths because they have a clearly defined expected behaviour. As you move to higher levels you might not know what the correct result should be, so you just end up checking it did the same as when you last tested it. For example, I wrote a pathfinding algorithm for a medical application. When it produced a path which looked correct we saved it to disk, and the test just reran the algorithm and made sure the results hadn't changed.

At the highest level you may want to trest the behaviour of you whle application. I implemented a system for this as well. I used AutoIT to automate the gui interaction and then ImageMagick to take and compare screenshots with the reference images. The whole thing was driven by some WSH written in JavaScript.

[rogma]

Using unit testing, going Test Driven is merely a matter of "faith".
It has a lot of known advantages and drawbacks. I personally see two main disadvantages :
* it looks like a lost of time (we should put it in regard to debugging time to decide). It surely is on throw-away-after-usage shell script.
* it seems sometimes impossible (3 months ago I could not see any way to test my GUI code) : which is something that smell from the previous discussion by saying that testing math is easier (true) but higher level is hard.

To PolyVox : for testing your pathfinding, why not "interpret" the pathfinder result : move an element according the path and check that
1. no blocking occurs
2. it reaches destination ?
--> it would prove that your pathfinder algorithm do the job whatever solution it finds.

That's the kind of thing I do to test AI : build a situation (generally I give invincibility to some agents) give orders and check that the expected result happens (generally some specific agents are destroyed).

An other example: I have a calculus that determine where you should shoot to hit a moving spaceship. The test is a complete in usage test :
1. build a target ship that moves linear
2. build a firing ship that has the calculator
3. make the calculator tell where to shoot
4. shoot in that direction
5. check that a hit occured

For my GUI testing (I use CeGUI on Ogre), one told me "we are not here to test third parties libraries", which means that all I focus on during GUI testing is checking that :
* I corretly build the required windows
* performing a GUI action by directly calling the callback method (again, I do not need to check that CEGUI works), it is just like I press a button, then I check that correct behaviour is done (moving windows, performing functionnal modifications...). My latest menu was completely done test driven and I found no bug while manually testing.

Now I am using same principle for the sound : for example, I move my space ship and then check that the openal associated source has also moved position etc...

So my answer to the question is : I use unit test at home and at work and I love it.

Regards

[PolyVox]

To PolyVox : for testing your pathfinding, why not "interpret" the pathfinder result : move an element according the path and check that
1. no blocking occurs
2. it reaches destination ?
--> it would prove that your pathfinder algorithm do the job whatever solution it finds.

Thanks, but it was actually a little more complex. The application is question was a medical app and the center line had to give a 'good' view of the stuctures being examined when the camera was flown along it. It's not impossible to quantify (you could track of how much had and hadn't been seen) but at some point it just becase too much effort for minimal gain