Login | Register
My pages Projects Community openCollabNet

Wiki: GettingStartedWithJHammer

Edit this page | Links to this page | Page information | Attachments | Refresh page


Before Starting

JHammer is an extension to JUnit. If you have no experience with JUnit or with unit testing it is suggested that you start with some of the great JUnit documentation available on the net, such as the JUnit Cookbook.

Why Psuedorandom Testing?

A unit test is typically created to help a programmer gain confidence that a certain 'unit' of code meets the software requirements. Good unit tests will cover all code paths through a given unit, as well as all states and state transitions associated with that unit. Creating a test suite that meets these requirements is a very difficult problem for non-trivial software projects. This is where pseudorandom testing steps in to assist the testers in creating interesting stimuli for unit tests.

For example, take a typical 2d vector implementation with an x and a y coordinate. The programmer implements a simple add function to add two Vectors, and wishes to write a unit test to ensure that this functionality works as intended. Using a straightforward implementation, one could make up a quick unit test (in psuedocode):

void testAdd()
  assert(<0,0> + <0,0> == <0,0>);
  assert(<0,0> + <0,1> == <0,1>);
  assert(<3,2> + <4,3> == <7,5>);
  assert(<MAX_VALUE,0> + <-MAX_VALUE,0> == <0,0>);

This looks like a fairly decent test, it tests the endpoints at the origin and the maximum values (MAX_VALUE) and then picks a few points in the middle to cover. If we imagine the 'coverage' space as a square, the covered points would look something like this:

Now, we could increase the coverage fairly easily by simply adding more 'asserts' into the unit test to cover a larger portion of values. But this is tedious, and not a good use of the programmers time! This is where random testing makes the job much easier. Here's the same test with some added unpredictability:

void testAdd()
  Vector2D startVector = <randInt,randInt>;
  Vector2D addVector = <randInt,randInt>;
  Vector2D endVector = startVector + addVector;
  assert(endVector == <startVector.x + addVector.x, startVector.y + addVector.y>);

Instead of having the tester write a large number of test cases, the random test can be run many (hundreds of thousands) of times, covering more execution paths and state transitions with very little effort. There are some obvious problems with the above test (what about overflow?), but it was created to illustrate a simple point. Typical random tests do not check for absolute correctness (as this example test did), but instead ensure that the unit under test (UUT) is following a set of rules determined by a specification.

An important thing to note is that random testing is not trying to achieve 100% coverage - most non-trivial software projects simply have too much state and too many possible code branches to fully test every possibility. Instead, the unpredictability of random testing is intended to generate interesting test stimulus - some of which may not have been thought of by the programmer or the test writer!

Test Structure and Tips

JHammer constructs a single test 'iteration' by randomly choosing a series of test Methods to run, which implicitly assumes that each class that extends ?RandomTest is associated with a single unit under test. What I mean is that if Vector2DTest extends ?RandomTest and is intended to test a class called Vector2D, then Vector2D test should create a single instance of Vector2D as a unit under test when it is constructed and all Test methods should act on that instance of uut (See the Simple Integer Accumulator Test for an example).

Here's the basic flow of JHammer to have in mind when writing tests: # Create an instance of the test class. # Find those methods of the test class that are annotated with Test. # Randomly choose a series of methods found in 2. # Repeat 1-3 for a specified number of iterations.

Tool Setup

Once you have downloaded JHammer (click 'Documents and Files' on the left navbar), simply add the jhammer.jar file to your project's classpath. That's all!

Basic Test

This is the most basic test that is possible - it simply prints out the methods that are being run by JHammer. Please look at the code in the JHammer tree at http://jhammer.tigris.org/source/browse/jhammer/trunk/src/jhammer/test/BasicTest.java. Lets walk through this.

   1 // import JHammer classes
   2 import jhammer.RandomRunner;
   3 import jhammer.RandomTest;

These are the only two classes you will need to import from JHammer. ?RandomRunner is the custom Runner used by JHammer to run your tests, and ?RandomRunner is the class that random tests must all inherit from. The inheritance from jhammer.?RandomTest is simply a safety check to help maintain repeatable tests.

   1 // run with jhammer.RandomRunner
   2 @RunWith(RandomRunner.class)
   3 public class BasicTest extends RandomTest

These are probably the only two lines that will look foreign to you, and just tell JUnit to run (?RunWith) these tests with jhammers ?RandomRunner. You MUST extend your test from jhammer.?RandomTest. The only thing that jhammer.?RandomTest does is creates an instance of Math.Random for each test, and provides wrapper methods to access it (?RandomTest.nextInt, ?RandomTest.nextDouble, etc.).

Next there are a series of these 'tests':

   2 @Test
   3 public void basicTest0()
   4 {
   5   System.out.println("Started basicTest0");
   6 }

When you run this as a JUnit test (either via the command line or via any JUnit 4 GUI), you should see output similar to the following:

Starting iteration 0 : -Dmethods=5 -Dseed=0x1dfe9f6327203b10
Started basicTest1
Started basicTest0
Started basicTest1
Started basicTest2
Started basicTest1
Starting iteration 1 : -Dmethods=5 -Dseed=0xdd8085dfcbd7f642
Started basicTest1
Started basicTest2
Started basicTest0
Started basicTest2
Started basicTest0

The lines that look like this:

Starting iteration 0 : -Dmethods=5 -Dseed=0x1dfe9f6327203b10

are called 'seed lines', and are used for reproducing failures. Since there are no assertions, this Basic Test will never fail! We'll need a slightly more advanced test to explore failures and how to repeat them.

Simple Integer Accumulator Test

This is another really simple test that is simply testing an accumulator (sum+=i) for overflow and underflow. Please look at the code in the JHammer tree at http://jhammer.tigris.org/source/browse/jhammer/trunk/src/jhammer/test/SimpleIntegerAccumTest.java.

I'll skip the walkthrough for this code, as the comments should be self-explanatory. Run the ?SimpleIntegerAccumTest as a JUnit test, and you should get output similar to the following:

Starting iteration 0 : -Dmethods=5 -Dseed=0xc6e114d8a71a168c
testSub : 0 + -306157684 = -306157684
testSub : -306157684 + -255211966 = -561369650
testAdd : -561369650 + 139327049 = -422042601
testAdd : -422042601 + 439169419 = 17126818
testAdd : 17126818 + 462960878 = 480087696

It is unlikely that your test will fail - I've run 1000's of iterations and never seen a failure. So lets get a little more aggressive and tweak some of the JHammer parameters. Add this to your jvm arguments (see http://junit.sourceforge.net/doc/faq/faq.htm#running_7 for why these can't be passed through directly):

-Dmethods=10 -Diterations=100

This will tell JHammer to run 10 methods (instead of the default 5) per iteration, and to run 100 iterations (instead of the default 10).

You should now get a failure - if not, simply rerun tests until JHammer generates a random stimulus that fails. Notice that JHammer (unlike JUnit) does not stop when a test iteration fails.

If you are using a JUnit GUI that displays the test descriptions (like Eclipse, you will see something resembling a seed line in the test description:

Iteration 10: -Dmethods=10 -Dseed=0x4785fb5ddfc9b18d

Otherwise you can just find the corresponding seed line in the test output:

Starting iteration 10 : -Dmethods=10 -Dseed=0x4785fb5ddfc9b18d

Simply copy and paste the part of the seed line after the colon ':' into your JVM arguments, and tell JHammer to only run one iteration:

-Dmethods=10 -Dseed=0x4785fb5ddfc9b18d -Diterations=1

Execute this test, and JHammer should reproduce the exact same test (check the output) that failed. You can also reproduce passing tests, but this is typically less interesting.

GettingStartedWithJHammer (last edited 2009-03-05 05:42:28 -0700 by ?jrdittmann)