Facility Testing of E4S via E4S Testsuite, Spack Test, and Buildtest

September 14, 2021, 11:00 am – 1:00 pm (CT)

Extreme-Scale Scientific Software Stack (E4S) is a collection of open-source software packages for running scientific application typically run on HPC systems. E4S is a collection of Spack packages that is built collectively with a fixed version of spack on a quarterly basis. So far, we have deployed E4S on Cori and Summit, and in the coming months we will have E4S deployed on Perlmutter, Spock, and Aurora. As part of the facility deployment, we need a mechanism to test the software stack and increase test coverage that properly tests the software at the facility.

E4S Testsuite is a validation testsuite for E4S products, which is a collection of shell scripts, makefiles, source code that runs tests on a facility deployed spack stack. Spack recently added support to run standalone tests for spack packages via spack test command which allows one to specify tests in their spack packages that can be run post-deployment.

Buildtest is a HPC testing framework designed to help facilities develop and run tests to validate their system and software stack. In buildtest, tests are written in YAML template called buildspecs which are processed by buildtest into shell-scripts. Buildtest has suppport for job submission to Slurm, LSF, PBS and Cobalt scheduler. Buildtest supports a rich YAML structure for writing buildspecs that is validated by a JSON Schema. Furthermore, buildtest provides numerous commands to query test results, inspect a particular test, validate buildspecs and find buildspecs through its buildspec cache. In v0.10.0, buildtest added support for spack schema to allow one to write buildspecs that can leverage spack to install specs, manage spack environments, and run spack test.

The presentation will provide a brief overview of buildtest commands and how to write tests in buildspecs, followed by a demo. The presentation will include an overview of Cori testsuite, which is a repository that contains sanity test for the Cori system including E4S tests using E4S tests. Gitlab is used to help automate execution of tests which are pushed to CDASH for post-processing. The presentation will end with a summary of E4S tests that are run on Cori, as well as current challenges.