Testing Erlang/OTP
==================

Before you start testing you need to have the Erlang release which you
are going to test in your path. See [$ERL_TOP/HOWTO/INSTALL.md][] for
instructions on how to build an Erlang release.

Short version
-------------
Move to the top directory of the Erlang release you want to test, i.e.
cd /ldisk/work/otp

    export ERL_TOP=`pwd`
	./otp_build setup -a
	export PATH=`pwd`/bin:$PATH
	./otp_build tests
	cd release/tests/test_server
	erl -s ts install -s ts run all_tests -s init stop

Where are the tests
-------------------

There are a lot of tests which test Erlang/OTP (as of 2012 about 12000) and
they are located in different places throughout the source tree. Below is a list
of the places where you can expect to find tests:

* $ERL_TOP/lib/AppName/test/
* $ERL_TOP/erts/test/
* $ERL_TOP/erts/emulator/test/
* $ERL_TOP/erts/epmd/test/

Writing tests
-------------

All tests are [common_test][] suites and follow the same pattern as all
[common_test][] suites. However, a couple of corner cases are
handled by a test wrapper called `ts`. `ts` allows the test writer to put
`Makefile.src` and `Makefile.first` files in the [data_dir][] and a special
directory called `all_SUITE_data`.

`Makefile.first` is run before any other Makefile and is typically used to
generate .hrl files which are needed by other test suites. At the moment only
the erl_interface tests use this feature and it should remain that way.

`Makefile.src` is configured to a `Makefile` using the `variables` created when
[configuring the tests][]. These `Makefile`s are later run by the test suite
to compile whatever platform specific code the tests need to run.

Releasing tests
---------------

If you cannot use [ct_run][] in the source tree you have to release the tests
into a common test directory. The easiest way to do this is to use `otp_build`
like this:

    export ERL_TOP=`pwd`; ./otp_build tests

This will release all tests in Erlang/OTP to `$ERL_TOP/release/tests/`. If you
want to change the directory where the tests are released to use the `TESTROOT`
environmental variable.

In the `$TESTROOT` you should now see *_test folders. These folders contain
everything needed to test Erlang/OTP and are platform independent; if you are
testing Erlang on multiple platforms you just have to release on one and copy
the tests to all other platforms.

### Releasing cross tests

For releasing tests in a cross compilation environment see [$ERL_TOP/HOWTO/INSTALL-CROSS.md][].

Configuring and Running tests
-----------------------------

Running tests is done by first navigating to the `$TESTROOT/test_server` folder
created when you released the tests and then start `erl` in that directory. The
emulator flags specified will be used in the test runs. For example, if you want
to test using async threads you have to supply `+A 10` to `erl` when you start it.

To configure and run the tests `ts` is used. `ts` is a wrapper module to
[common_test][] which takes care of configuration and build issues before
[common_test][] is started.

`ts` has a lot of special options and functions which can be useful when
testing Erlang/OTP. For a full listing issue `ts:help()` in the erlang shell.

### Configuring the test environment

Before running released tests you have to install them on the target system.
Installing the tests is done by calling `ts:install().` in the Erlang shell
which you intend to test from. `ts:install()` is basically a wrapper to a
configure script and some Erlang code which figures out what your system looks
like and what kind of emulator you are testing with. `ts:install()` can also
take some arguments if necessary, see `ts:help()` for details.

All variables created by `ts:install()` are found in
`$TESTROOT/test_server/variables`.

### Running the tests

To run all test suites go to `$TESTROOT/test_server` fire up an Erlang shell and type:

    ts:run().

Note that running all tests will require several hours, so you may want to run
the test cases for a single application

    ts:run(Application, [batch]).

or even part of the test suite for an application, for example

    ts:run(emulator, bs, [batch]).

to run all test suite modules starting with `bs` (i.e. all modules that test
the bit syntax).

To run a specific test case in a module, the full name of the module and test
case must be specified:

    ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]).

Run `ts:help().` for more information.

As of R14B02 it is also possibly to start all tests but the erl_interface tests
by invoking Common Test directly from the released applications test directory,
i.e.

    cd $TESTROOT/test_server
    $ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse

Running [ct_run][] from the command line still requires you to do the
`ts:install()` step above.

### Convenience for running tests without the release and configuration steps

It can be convenient to run tests with a single command. This way, one
do not need to worry about missing to run `make release_tests` after
changing a test suite. The `make test` command can be used for this
purpose. The `make test` command works when the current directory
contains a directory called test and in the root directory of the
source code tree.

*(Warning)* Some test cases do not run correctly or cannot be run at
all through the `make test` command (typically test cases that require
test specific C code to be compiled) because `make test` runs tests
directly by invoking the `ct_run` command instead of using the `ts`
wrapper. One has to follow the procedure described above to run test
cases that do not work with `make test`.

Below are some examples that illustrate how `make test` can be
used:

    # ERL_TOP needs to be set correctly
    cd /path/to/otp
    export ERL_TOP=`pwd`

    # Build Erlang/OTP
    #
    # Note that make test will only compile test code except when
    # make test is executed from $ERL_TOP.
    ./otp_build setup -a

    # Run a test case (The ARGS variable is passed to ct_run)
    (cd $ERL_TOP/erts/emulator && make ARGS="-suite binary_SUITE -case deep_bitstr_lists" test)

    # Run a test suite
    (cd $ERL_TOP/lib/stdlib && make ARGS="-suite ets_SUITE" test)

    # Run all test suites for an application
    (cd $ERL_TOP/lib/asn1 && make test)

    # Run all tests
    #
    # When executed from $ERL_TOP, "make test" will first release and
    # configure all tests and then attempt to run all tests with `ts:run`.
    # This will take several hours.
    (cd $ERL_TOP && make test)


Examining the results
---------------------

Open the file `release/tests/test_server/index.html` in a web browser. Or open
`release/tests/test_server/last_test.html` when a test suite is running to
examine the results so far for the currently executing test suite (in R14B02 and
later you want to open the `release/tests/test_server/all_runs.html` file to
get to the currently running test)


Run tests with Address Sanitizer
--------------------------------

First build emulator with `asan` build target.
See [$ERL_TOP/HOWTO/INSTALL.md][].

Set environment variable `ASAN_LOG_DIR` to the directory
where the error logs will be generated.

    export ASAN_LOG_DIR=$TESTROOT/test_server/asan_logs
    mkdir $ASAN_LOG_DIR

Set environment variable `TS_RUN_EMU` to `asan`.

    export TS_RUN_EMU=asan

Then run the tests you want with `ts:run` as described above. Either
inspect the log files directly or use the script at
`$ERL_TOP/erts/emulator/asan/asan_logs_to_html` to read all log files
in `$ASAN_LOG_DIR` and distill them into one html page
`asan_summary.html`. Repeated reports from the same memory leak will
for example be ignored by the script and make it easier to analyze.


Run tests with Valgrind
-----------------------

First make sure [valgrind][] is installed, then build OTP from source
and build the emulator with `valgrind` build target. See
[$ERL_TOP/HOWTO/INSTALL.md][].

Set environment variable `VALGRIND_LOG_DIR` to the directory
where the valgrind error logs will be generated.

    export VALGRIND_LOG_DIR=$TESTROOT/test_server/vg_logs
    mkdir $VALGRIND_LOG_DIR

Set environment variable `TS_RUN_EMU` to `valgrind`.

    export TS_RUN_EMU=valgrind

Then run the tests you want with `ts:run` as described above and
inspect the log file(s) in `$VALGRIND_LOG_DIR`.


   [ct_run]: http://www.erlang.org/doc/man/ct_run.html
   [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html
   [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md
   [$ERL_TOP/HOWTO/INSTALL-CROSS.md]: INSTALL-CROSS.md#testing-the-cross-compiled-system
   [common_test]: http://www.erlang.org/doc/man/ct.html
   [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir
   [configuring the tests]: #configuring-the-test-environment
   [valgrind]: https://valgrind.org

   [?TOC]: true
