Testing on the JVM is a rich subject matter. There are many different testing libraries and frameworks, as well as many different types of test. All need to be part of the build, whether they are executed frequently or infrequently. This chapter is dedicated to explaining how Gradle handles differing requirements between and within builds, with significant coverage of how it integrates with the two most common testing frameworks: JUnit and TestNG.

It explains:

But first, let’s look at the basics of JVM testing in Gradle.

A new configuration DSL for modeling test execution phases is available via the incubating JVM Test Suite plugin.

The basics

All JVM testing revolves around a single task type: Test. This runs a collection of test cases using any supported test library — JUnit, JUnit Platform or TestNG — and collates the results. You can then turn those results into a report via an instance of the TestReport task type.

In order to operate, the Test task type requires just two pieces of information:

  • Where to find the compiled test classes (property: Test.getTestClassesDirs())

  • The execution classpath, which should include the classes under test as well as the test library that you’re using (property: Test.getClasspath())

When you’re using a JVM language plugin — such as the Java Plugin — you will automatically get the following:

  • A dedicated test source set for unit tests

  • A test task of type Test that runs those unit tests

The JVM language plugins use the source set to configure the task with the appropriate execution classpath and the directory containing the compiled test classes. In addition, they attach the test task to the check lifecycle task.

It’s also worth bearing in mind that the test source set automatically creates corresponding dependency configurations — of which the most useful are testImplementation and testRuntimeOnly — that the plugins tie into the test task’s classpath.

All you need to do in most cases is configure the appropriate compilation and runtime dependencies and add any necessary configuration to the test task. The following example shows a simple setup that uses JUnit Platform and changes the maximum heap size for the tests' JVM to 1 gigabyte:

Example 1. A basic configuration for the 'test' task
build.gradle.kts
dependencies {
    testImplementation("org.junit.jupiter:junit-jupiter:5.7.1")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}

tasks.named<Test>("test") {
    useJUnitPlatform()

    maxHeapSize = "1G"

    testLogging {
        events("passed")
    }
}
build.gradle
dependencies {
    testImplementation 'org.junit.jupiter:junit-jupiter:5.7.1'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

tasks.named('test', Test) {
    useJUnitPlatform()

    maxHeapSize = '1G'

    testLogging {
        events "passed"
    }
}

The Test task has many generic configuration options as well as several framework-specific ones that you can find described in JUnitOptions, JUnitPlatformOptions and TestNGOptions. We cover a significant number of them in the rest of the chapter.

If you want to set up your own Test task with its own set of test classes, then the easiest approach is to create your own source set and Test task instance, as shown in Configuring integration tests.

Test execution

Gradle executes tests in a separate ('forked') JVM, isolated from the main build process. This prevents classpath pollution and excessive memory consumption for the build process. It also allows you to run the tests with different JVM arguments than the build is using.

You can control how the test process is launched via several properties on the Test task, including the following:

maxParallelForks — default: 1

You can run your tests in parallel by setting this property to a value greater than 1. This may make your test suites complete faster, particularly if you run them on a multi-core CPU. When using parallel test execution, make sure your tests are properly isolated from one another. Tests that interact with the filesystem are particularly prone to conflict, causing intermittent test failures.

Your tests can distinguish between parallel test processes by using the value of the org.gradle.test.worker property, which is unique for each process. You can use this for anything you want, but it’s particularly useful for filenames and other resource identifiers to prevent the kind of conflict we just mentioned.

forkEvery — default: 0 (no maximum)

This property specifies the maximum number of test classes that Gradle should run on a test process before its disposed of and a fresh one created. This is mainly used as a way to manage leaky tests or frameworks that have static state that can’t be cleared or reset between tests.

Warning: a low value (other than 0) can severely hurt the performance of the tests

ignoreFailures — default: false

If this property is true, Gradle will continue with the project’s build once the tests have completed, even if some of them have failed. Note that, by default, the Test task always executes every test that it detects, irrespective of this setting.

failFast —  (since Gradle 4.6) default: false

Set this to true if you want the build to fail and finish as soon as one of your tests fails. This can save a lot of time when you have a long-running test suite and is particularly useful when running the build on continuous integration servers. When a build fails before all tests have run, the test reports only include the results of the tests that have completed, successfully or not.

You can also enable this behavior by using the --fail-fast command line option, or disable it respectively with --no-fail-fast.

testLogging — default: not set

This property represents a set of options that control which test events are logged and at what level. You can also configure other logging behavior via this property. See TestLoggingContainer for more detail.

See Test for details on all the available configuration options.

The test process can exit unexpectedly if configured incorrectly. For instance, if the Java executable does not exist or an invalid JVM argument is provided, the test process will fail to start. Similarly, if a test makes programmatic changes to the test process, this can also cause unexpected failures.

For example, issues may occur if a SecurityManager is modified in a test because Gradle’s internal messaging depends on reflection and socket communication, which may be disrupted if the permissions on the security manager change. In this particular case, you should restore the original SecurityManager after the test so that the gradle test worker process can continue to function.

Test filtering

It’s a common requirement to run subsets of a test suite, such as when you’re fixing a bug or developing a new test case. Gradle provides two mechanisms to do this:

  • Filtering (the preferred option)

  • Test inclusion/exclusion

Filtering supersedes the inclusion/exclusion mechanism, but you may still come across the latter in the wild.

With Gradle’s test filtering you can select tests to run based on:

  • A fully-qualified class name or fully qualified method name, e.g. org.gradle.SomeTest, org.gradle.SomeTest.someMethod

  • A simple class name or method name if the pattern starts with an upper-case letter, e.g. SomeTest, SomeTest.someMethod (since Gradle 4.7)

  • '*' wildcard matching

You can enable filtering either in the build script or via the --tests command-line option. Here’s an example of some filters that are applied every time the build runs:

Example 2. Filtering tests in the build script
build.gradle.kts
tasks.test {
    filter {
        //include specific method in any of the tests
        includeTestsMatching("*UiCheck")

        //include all tests from package
        includeTestsMatching("org.gradle.internal.*")

        //include all integration tests
        includeTestsMatching("*IntegTest")
    }
}
build.gradle
test {
    filter {
        //include specific method in any of the tests
        includeTestsMatching "*UiCheck"

        //include all tests from package
        includeTestsMatching "org.gradle.internal.*"

        //include all integration tests
        includeTestsMatching "*IntegTest"
    }
}

For more details and examples of declaring filters in the build script, please see the TestFilter reference.

The command-line option is especially useful to execute a single test method. When you use --tests, be aware that the inclusions declared in the build script are still honored. It is also possible to supply multiple --tests options, all of whose patterns will take effect. The following sections have several examples of using the command-line option.

Not all test frameworks play well with filtering. Some advanced, synthetic tests may not be fully compatible. However, the vast majority of tests and use cases work perfectly well with Gradle’s filtering mechanism.

The following two sections look at the specific cases of simple class/method names and fully-qualified names.

Simple name pattern

Since 4.7, Gradle has treated a pattern starting with an uppercase letter as a simple class name, or a class name + method name. For example, the following command lines run either all or exactly one of the tests in the SomeTestClass test case, regardless of what package it’s in:

# Executes all tests in SomeTestClass
gradle test --tests SomeTestClass

# Executes a single specified test in SomeTestClass
gradle test --tests SomeTestClass.someSpecificMethod

gradle test --tests SomeTestClass.*someMethod*

Fully-qualified name pattern

Prior to 4.7 or if the pattern doesn’t start with an uppercase letter, Gradle treats the pattern as fully-qualified. So if you want to use the test class name irrespective of its package, you would use --tests *.SomeTestClass. Here are some more examples:

# specific class
gradle test --tests org.gradle.SomeTestClass

# specific class and method
gradle test --tests org.gradle.SomeTestClass.someSpecificMethod

# method name containing spaces
gradle test --tests "org.gradle.SomeTestClass.some method containing spaces"

# all classes at specific package (recursively)
gradle test --tests 'all.in.specific.package*'

# specific method at specific package (recursively)
gradle test --tests 'all.in.specific.package*.someSpecificMethod'

gradle test --tests '*IntegTest'

gradle test --tests '*IntegTest*ui*'

gradle test --tests '*ParameterizedTest.foo*'

# the second iteration of a parameterized test
gradle test --tests '*ParameterizedTest.*[2]'

Note that the wildcard '*' has no special understanding of the '.' package separator. It’s purely text based. So --tests *.SomeTestClass will match any package, regardless of its 'depth'.

You can also combine filters defined at the command line with continuous build to re-execute a subset of tests immediately after every change to a production or test source file. The following executes all tests in the 'com.mypackage.foo' package or subpackages whenever a change triggers the tests to run:

gradle test --continuous --tests "com.mypackage.foo.*"

Test reporting

The Test task generates the following results by default:

  • An HTML test report

  • XML test results in a format compatible with the Ant JUnit report task — one that is supported by many other tools, such as CI servers

  • An efficient binary format of the results used by the Test task to generate the other formats

In most cases, you’ll work with the standard HTML report, which automatically includes the results from all your Test tasks, even the ones you explicitly add to the build yourself. For example, if you add a Test task for integration tests, the report will include the results of both the unit tests and the integration tests if both tasks are run.

To aggregate test results across multiple subprojects, see the Test Report Aggregation Plugin.

Unlike with many of the testing configuration options, there are several project-level convention properties that affect the test reports. For example, you can change the destination of the test results and reports like so:

Example 3. Changing the default test report and results directories
build.gradle.kts
reporting.baseDir = file("my-reports")
java.testResultsDir.set(layout.buildDirectory.dir("my-test-results"))

tasks.register("showDirs") {
    val rootDir = project.rootDir
    val reportsDir = project.reporting.baseDirectory
    val testResultsDir = project.java.testResultsDir

    doLast {
        logger.quiet(rootDir.toPath().relativize(reportsDir.get().asFile.toPath()).toString())
        logger.quiet(rootDir.toPath().relativize(testResultsDir.get().asFile.toPath()).toString())
    }
}
build.gradle
reporting.baseDir = "my-reports"
java.testResultsDir = layout.buildDirectory.dir("my-test-results")

tasks.register('showDirs') {
    def rootDir = project.rootDir
    def reportsDir = project.reporting.baseDirectory
    def testResultsDir = project.java.testResultsDir

    doLast {
        logger.quiet(rootDir.toPath().relativize(reportsDir.get().asFile.toPath()).toString())
        logger.quiet(rootDir.toPath().relativize(testResultsDir.get().asFile.toPath()).toString())
    }
}
Output of gradle -q showDirs
> gradle -q showDirs
my-reports
build/my-test-results

Follow the link to the convention properties for more details.

There is also a standalone TestReport task type that you can use to generate a custom HTML test report. All it requires are a value for destinationDir and the test results you want included in the report. Here is a sample which generates a combined report for the unit tests from all subprojects:

Example 4. Creating a unit test report for subprojects
buildSrc/src/main/kotlin/myproject.java-conventions.gradle.kts
plugins {
    id("java")
}

// Disable the test report for the individual test task
tasks.named<Test>("test") {
    reports.html.required.set(false)
}

// Share the test report data to be aggregated for the whole project
configurations.create("binaryTestResultsElements") {
    isCanBeResolved = false
    isCanBeConsumed = true
    attributes {
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
        attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named("test-report-data"))
    }
    outgoing.artifact(tasks.test.map { task -> task.getBinaryResultsDirectory().get() })
}
build.gradle.kts
val testReportData by configurations.creating {
    isCanBeConsumed = false
    attributes {
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
        attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named("test-report-data"))
    }
}

dependencies {
    testReportData(project(":core"))
    testReportData(project(":util"))
}

tasks.register<TestReport>("testReport") {
    destinationDirectory.set(reporting.baseDirectory.dir("allTests"))
    // Use test results from testReportData configuration
    testResults.from(testReportData)
}
buildSrc/src/main/groovy/myproject.java-conventions.gradle
plugins {
    id 'java'
}

// Disable the test report for the individual test task
test {
    reports.html.required = false
}

// Share the test report data to be aggregated for the whole project
configurations {
    binaryTestResultsElements {
        canBeResolved = false
        canBeConsumed = true
        attributes {
            attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
            attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'test-report-data'))
        }
        outgoing.artifact(test.binaryResultsDirectory)
    }
}
build.gradle
// A resolvable configuration to collect test reports data
configurations {
    testReportData {
        canBeConsumed = false
        attributes {
            attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
            attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'test-report-data'))
        }
    }
}

dependencies {
    testReportData project(':core')
    testReportData project(':util')
}

tasks.register('testReport', TestReport) {
    destinationDirectory = reporting.baseDirectory.dir('allTests')
    // Use test results from testReportData configuration
    testResults.from(configurations.testReportData)
}

In this example, we use a convention plugin myproject.java-conventions to expose the test results from a project to Gradle’s variant aware dependency management engine.

The plugin declares a consumable binaryTestResultsElements configuration that represents the binary test results of the test task. In the aggregation project’s build file, we declare the testReportData configuration and depend on all of the projects that we want to aggregate the results from. Gradle will automatically select the binary test result variant from each of the subprojects instead of the project’s jar file. Lastly, we add a testReport task that aggregates the test results from the testResultsDirs property, which contains all of the binary test results resolved from the testReportData configuration.

You should note that the TestReport type combines the results from multiple test tasks and needs to aggregate the results of individual test classes. This means that if a given test class is executed by multiple test tasks, then the test report will include executions of that class, but it can be hard to distinguish individual executions of that class and their output.

Communicating test results to CI servers and other tools via XML files

The Test tasks creates XML files describing the test results, in the “JUnit XML” pseudo standard. It is common for CI servers and other tooling to observe test results via these XML files.

By default, the files are written to $buildDir/test-results/$testTaskName with a file per test class. The location can be changed for all test tasks of a project, or individually per test task.

Example 5. Changing JUnit XML results location for all test tasks
build.gradle.kts
java.testResultsDir.set(layout.buildDirectory.dir("junit-xml"))
build.gradle
java.testResultsDir = layout.buildDirectory.dir("junit-xml")

With the above configuration, the XML files will be written to $buildDir/junit-xml/$testTaskName.

Example 6. Changing JUnit XML results location for a particular test task
build.gradle.kts
tasks.test {
    reports {
        junitXml.outputLocation.set(layout.buildDirectory.dir("test-junit-xml"))
    }
}
build.gradle
test {
    reports {
        junitXml.outputLocation.set(layout.buildDirectory.dir("test-junit-xml"))
    }
}

With the above configuration, the XML files for the test task will be written to $buildDir/test-results/test-junit-xml. The location of the XML files for other test tasks will be unchanged.

Configuration options

The content of the XML files can also be configured to convey the results differently, by configuring the JUnitXmlReport options.

Example 7. Configuring how the results are conveyed
build.gradle.kts
tasks.test {
    reports {
        junitXml.apply {
            isOutputPerTestCase = true // defaults to false
            mergeReruns.set(true) // defaults to false
        }
    }
}
build.gradle
test {
    reports {
        junitXml {
            outputPerTestCase = true // defaults to false
            mergeReruns = true // defaults to false
        }
    }
}
outputPerTestCase

The outputPerTestCase option, when enabled, associates any output logging generated during a test case to that test case in the results. When disabled (the default) output is associated with the test class as whole and not the individual test cases (e.g. test methods) that produced the logging output. Most modern tools that observe JUnit XML files support the “output per test case” format.

If you are using the XML files to communicate test results, it is recommended to enable this option as it provides more useful reporting.

mergeReruns

When mergeReruns is enabled, if a test fails but is then retried and succeeds, its failures will be recorded as <flakyFailure> instead of <failure>, within one <testcase>. This is effectively the reporting produced by the surefire plugin of Apache Maven™ when enabling reruns. If your CI server understands this format, it will indicate that the test was flaky. If it does not, it will indicate that the test succeeded as it will ignore the <flakyFailure> information. If the test does not succeed (i.e. it fails for every retry), it will be indicated as having failed whether your tool understands this format or not.

When mergeReruns is disabled (the default), each execution of a test will be listed as a separate test case.

If you are using build scans or Gradle Enterprise, flaky tests will be detected regardless of this setting.

Enabling this option is especially useful when using a CI tool that uses the XML test results to determine build failure instead of relying on Gradle’s determination of whether the build failed or not, and you wish to not consider the build failed if all failed tests passed when retried. This is the case for the Jenkins CI server and its JUnit plugin. With mergeReruns enabled, tests that pass-on-retry will no longer cause this Jenkins plugin to consider the build to have failed. However, failed test executions will be omitted from the Jenkins test result visualizations as it does not consider <flakyFailure> information. The separate Flaky Test Handler Jenkins plugin can be used in addition to the JUnit Jenkins plugin to have such “flaky failures” also be visualized.

Tests are grouped and merged based on their reported name. When using any kind of test parameterization that affects the reported test name, or any other kind of mechanism that produces a potentially dynamic test name, care should be taken to ensure that the test name is stable and does not unnecessarily change.

Enabling the mergeReruns option does not add any retry/rerun functionality to test execution. Rerunning can be enabled by the test execution framework (e.g. JUnit’s @RepeatedTest), or via the separate Test Retry Gradle plugin.

Test detection

By default, Gradle will run all tests that it detects, which it does by inspecting the compiled test classes. This detection uses different criteria depending on the test framework used.

For JUnit, Gradle scans for both JUnit 3 and 4 test classes. A class is considered to be a JUnit test if it:

  • Ultimately inherits from TestCase or GroovyTestCase

  • Is annotated with @RunWith

  • Contains a method annotated with @Test or a super class does

For TestNG, Gradle scans for methods annotated with @Test.

Note that abstract classes are not executed. In addition, be aware that Gradle scans up the inheritance tree into jar files on the test classpath. So if those JARs contain test classes, they will also be run.

If you don’t want to use test class detection, you can disable it by setting the scanForTestClasses property on Test to false. When you do that, the test task uses only the includes and excludes properties to find test classes.

If scanForTestClasses is false and no include or exclude patterns are specified, Gradle defaults to running any class that matches the patterns **/*Tests.class and **/*Test.class, excluding those that match **/Abstract*.class.

With JUnit Platform, only includes and excludes are used to filter test classes — scanForTestClasses has no effect.

Test grouping

JUnit, JUnit Platform and TestNG allow sophisticated groupings of test methods.

This section applies to grouping individual test classes or methods within a collection of tests that serve the same testing purpose (unit tests, integration tests, acceptance tests, etc.). For dividing test classes based upon their purpose, see the incubating JVM Test Suite plugin.

JUnit 4.8 introduced the concept of categories for grouping JUnit 4 tests classes and methods.[1] Test.useJUnit(org.gradle.api.Action) allows you to specify the JUnit categories you want to include and exclude. For example, the following configuration includes tests in CategoryA and excludes those in CategoryB for the test task:

Example 8. JUnit Categories