The Maven Cookbook

7.6. Running TestNG Tests in Parallel

7.6. Running TestNG Tests in Parallel

7.6.1. Task

You would like to run your unit tests in parallel to speed up your builds.

7.6.2. Action

To illustrate executing tests in parallel, we can create a component to be tests that sleeps for 5 seconds. The following listing is the SeriousComponent.java class which would be stored in ${basedir}/src/main/java under the org.sonatype.mcookbook package:

package org.sonatype.mcookbook;

public class SeriousComponent {

 /**
  * This function tests the seriousness of a String.
  * Returns false if the string contains the word
  * "FUNNY", returns truje otherwise.
  */
  public static boolean testSeriousness( String text ) {
    try {
      Thread.sleep( 5000 );
    } catch (InterruptedException e) {
    }
    return !text.toUpperCase().contains( "FUNNY" );
  }
}

This class has a single static method which sleeps for five seconds and then tests a string for the presence of the word "funny". To test this class, there is a SeriousComponentTest.java class in ${basedir}/src/test/java under the org.sonatype.mcookbook package which contains the following four test methods:

package org.sonatype.mcookbook;

import org.testng.annotations.Test;

@Test
public class SeriousComponentTest {

  @Test
  public void testSad() throws Exception {
    assert SeriousComponent.testSeriousness("SAD");
  }

  @Test
  public void testSerious() throws Exception {
    assert SeriousComponent.testSeriousness("SERIOUS");
  }

  @Test
  public void testCrazy() throws Exception {
    assert SeriousComponent.testSeriousness("CRAZY");
  }

  @Test
  public void testFunny() throws Exception {
    assert !SeriousComponent.testSeriousness("FUNNY");
  }
}

If you ran this test in a single thread, it would take approximately 20 seconds to execute. To speed up your build, configure Maven to execute each test method in parallel and set the available thread count for parallel test execution to 4. To do this, add the following plugin configuration to your project's POM.

<project xmlns="http://maven.apache.org/POM/4.0.0" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
           http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.sonatype.mcookbook</groupId>
  <artifactId>testng-parallel</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>testng-tests</name>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <parallel>methods</parallel>
          <threadCount>4</threadCount>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <dependencies>
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>5.10</version>
      <classifier>jdk15</classifier>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

Execute the TestNG test in parallel by running mvn test and take note of the time it takes to execute all four tests.

$ mvn test
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building testng-tests
[INFO]    task-segment: [test]
[INFO] ------------------------------------------------------------------------
[INFO] [resources:resources {execution: default-resources}]
[WARNING] Using platform encoding (MacRoman actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources {execution: default-testResources}]
[WARNING] Using platform encoding (MacRoman actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory: /Users/Tim/Library/Code/sonatype/
maven-cookbook/mcookbook-examples/unit/testng-parallel/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.342 sec

Results :

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

You can see that the execution of four tests took a total of 5.342 sec. This is a clear indication that the tests were being run in parallel given the fact that the function being tested has a 5 second sleep built-in. Now, change the parallel configuration parameter to "classes" and rerun the test. Find the Maven Surefire plugin configuration in the project's POM and change the configuration to this:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <parallel>classes</parallel>
          <threadCount>4</threadCount>
        </configuration>
      </plugin>

Now, run mvn test and notice how long it takes to run the unit tests - 20 seconds. When you run the tests in parallel using classes, TestNG is going to create an independent test execution thread for each class. The four test methods in SeriousComponentTest are going to be executed in one, single thread instead of the four simultaneous threads that were used when the parallel configuration parameter was set to "methods".

$ mvn test
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building testng-tests
[INFO]    task-segment: [test]
[INFO] ------------------------------------------------------------------------
[INFO] [resources:resources {execution: default-resources}]
[WARNING] Using platform encoding (MacRoman actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources {execution: default-testResources}]
[WARNING] Using platform encoding (MacRoman actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory: /Users/Tim/Library/Code/sonatype/
maven-cookbook/mcookbook-examples/unit/testng-parallel/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.411 sec

Results :

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

7.6.3. Detail

If waiting around for your unit tests is eating up too much of your time, this is a great way to make your builds a bit faster.

You use the following values in the parallel configuration parmeter:

classes

Each test class will be executed in an independent execution thread.

methods

Each test method will be run as an independent test in a separate thread. Note that if one method states and explicit dependency on another method, TestNG will note the dependency and execute both methods in the same Thread.

tests

If you use a TestNG Suite XML file and you list a test element that groups a number of related tests, listing test in the parallel configuration element will configure TestNG to use a separate thread for each <test> element. This option is only relevant if you have configured the Maven Surefire plugin to use a custom TestNG Suite XML file with one or more <test> elements.