Testing

Structuring tests

The goal is to be able to run subsets of tests and to encapsulate the code of each test.

My current approach:

1. Place each testset inside a function.
2. Call these functions from within other testsets.
3. One can now include each file and run the tests independently.
4. The function provide some isolation (similar to using modules).

# runtests.jl

@testset "All" begin
    include("test_one.jl");
end


# test_one.jl

using Test

function test_one()
    @testset "A" begin
        @test 1 == 1
    end
end

@testset "One" begin
    test_one();
end

Now Pkg.test() runs everything, but include("test/test_one.jl") only runs the subset. SafeTests.jl goes further by wrapping tests in modules.

Test helpers can be put into a separate file and conditionally included, as in (taken from MPVerify.jl):

@isdefined(TestHelpers) || include("../TestHelpers.jl")

This checks whether the module TestHelpers exists. But one may not need a module for the helpers.

Module approach:

1. Place each group of tests into a module, so the tests are independent of each other and can be run independently. SafeTestsets.jl has a similar idea, but I find it cleaner to explicitly write out the modules. Though modules have the benefit that they can include setup code that is used repeatedly in different tests.
2. runtests.jl simply contains a list of include statements; one for each test module. Those are wrapped in a @testset for nice display and to ensure that errors don't stop the tests.
3. Each test module also contains a @testset.
4. When runtests is run, it displays a single success summary. But when there are errors, they are nicely broken down by testset.
5. To run tests selectively, simply include the file that contains the @testset at the REPL.

Test specific dependencies

Test dependencies now need to be added to the Project.toml file in ./test:

pkg> activate ./test
pkg> add MyPkg
pkg> activate .

All dependencies used in tests now have to be manually added. They do not "carry over" from the main package.

Developing test set dependencies seems to cause problems ("error: cannot merge projects"). They need to be added.

Important note: For now (1.5) test dependencies are still considered "beta" and buggy. Do not use. Instead, use the 1.0 method of extras in Project.toml.

Useful packages

• Aqua.jl
• checks for method ambiguities, invalid exports, stale dependencies, and more.
• TestSetExtensions.jl
• mainly provides nicer display of test progress and failures
• UnitTestDesign
• generates combinations of arguments that are passed to tests
• the goal is to generate coverage without having to run all parameter combinations
• Jet.jl can be used to test type stability (using @test_opt foo()).
• TestPicker.jl can run individual tests based on fuzzy testset search. Requires no modification of test code.

TestItems.jl

Can place tests anywhere in code.

Can run tests selectively (also from within VS Code). That happens without much compilation lag.

Setup code

• Put in @testmodule if to be run only once per test run.
• Need to qualify access to all objects from the testmodule as in MyTestModule.foo. Even if exported.

Issues

1. Need to add using MyPackage, Test to each testitem. Otherwise, tests cannot be run from the REPL using @run_package_tests. (2025)
2. Tests stop when errors are encountered (unlike the standard Pkg.test framework).

Travis CI (1.2)

Travis can automatically test all branches uploaded to github.

Need to customize travis.yml to only build for the current Julia version.

Building with unregistered dependencies is tricky. Probably ok if the dependencies are added (so they point to a github url), but not if they are developed.

Miscellaneous

Errors in the code to be tested (but not caught by @test) cause the entire test run to crash. Preventing this requires all tests to be enclosed in a @testset. A sequence of @testset does not do the trick. An error in one prevents all others from being run. Nested @testsets produce nested error reports (nice).

@test statements can be placed inside functions. To preserve result reporting, the function should contain a @testset and return its result.

---