Running Tests¶

Quickstart¶

To execute the LLVM test-suite using LNT, use the lnt runtest command. The information below should be enough to get you started, but see the sections below for more complete documentation.

Install lnt as explained in the installation section.
Make sure lit is installed, for example with pip install lit or via a monorepo installation accessible in your $PATH. By default, lnt will look for a binary named llvm-lit in your $PATH. Depending on how you install lit, you may have to point lnt to the right binary by using the --use-lit <path> flag in the command below.
Checkout the LLVM test-suite, if you haven’t already:
```
git clone https://github.com/llvm/llvm-test-suite.git llvm-test-suite
```
You should always keep the test-suite directory itself clean (that is, never do a configure inside your test suite). Make sure not to check it out into the LLVM projects directory, as LLVM’s configure/make build will then want to automatically configure it for you.
Execute the lnt runtest test-suite test producer, point it at the test suite and the compiler you want to test:
```
lnt runtest test-suite --sandbox $PWD/sandbox            \
                       --cc clang                        \
                       --cxx clang++                     \
                       --test-suite $PWD/llvm-test-suite \
                       --cmake-cache Release
```
The --sandbox argument is a path to where the test suite build products and results will be stored (inside a timestamped directory, by default).

We recommend adding --build-tool-options "-k" (if you are using make) or --build-tool-options "-k 0" (if you are using ninja). This ensures that the build tool carries on building even if there is a compilation failure in one of the tests. Without these options, every test after the compilation failure will not be compiled and will be reported as a missing executable.
If you already have a LNT server instance running, you can submit these results to it by passing --submit <path-or-URL-of-instance>.
On most systems, the execution time results will be a bit noisy. There are a range of things you can do to reduce noise:
- Only build the benchmarks in parallel, but do the actual running of the benchmark code at most one at a time (use --threads 1 --build-threads 6). Of course, when you’re also interested in the measured compile time, you should also build sequentially (use --threads 1 --build-threads 1).
- When running on linux: Make lnt use perf to get more accurate timing for short-running benchmarks (use --use-perf=time).
- Pin the running benchmark to a specific core, so the OS doesn’t move the benchmark process from core to core (on linux, use --make-param="RUNUNDER=taskset -c 1").
- Only run the programs that are marked as a benchmark; some of the tests in the test-suite are not intended to be used as a benchmark (use --benchmarking-only).
- Make sure each program gets run multiple times, so that LNT has a higher chance of recognizing which programs are inherently noisy (use --multisample=5).
- Disable frequency scaling / turbo boost. In case of thermal throttling it can skew the results.
- Disable as many processes or services as possible on the target system.

Viewing Results¶

By default, lnt runtest test-suite will show the passes and failures after doing a run, but if you are interested in viewing the result data in more detail you should install a local LNT instance to submit the results to. See the sections on running a server and importing data for instructions on how to do that.

Test Producers¶

On the client-side, LNT comes with a number of built-in test data producers. This documentation focuses on the LLVM test-suite (aka nightly test) generator, since it is the primary test run using the LNT infrastructure, but note that LNT also includes tests for other interesting pieces of data, for example Clang compile-time performance.

LNT also makes it easy to add new test data producers and includes examples of custom data importers (e.g., to import buildbot build information into) and dynamic test data generators (e.g., abusing the infrastructure to plot graphs, for example).

Built-in Tests¶

The built-in tests are designed to be run via the lnt tool. The following tools for working with built-in tests are available:

lnt showtests
List the available tests. Tests are defined with an extensible architecture. FIXME: Point at docs on how to add a new test.

lnt runtest [<run options>] <test name> ... test arguments ...
Run the named test. The run tool itself accepts a number of options which are common to all tests. The most common option is --submit=<url> which specifies the server to submit the results to after testing is complete. See lnt runtest --help for more information on the available options.

The remainder of the options are passed to the test tool itself. The options are specific to the test, but well behaved tests should respond to