EasyBuild

Scientific software installation may be a difficult and tedious process. Not all software rely on "standard" building tools like the traditional configure/make/make install or CMake. EasyBuild is one of the tools which automates the potentially time-consuming and error-prone process of correctly building and installing scientific softwares. It performs the high-level steps of obtaining source code, setting necessary build options and installing dependencies.

🧭 Full documentation about EasyBuild can be found here: https://docs.easybuild.io/

General concepts

Toolchains

EasyBuild is based on the concept of toolchains. A toolchain is generally a set of compiler and scientific libraries. It can be loaded to achieve compilations of homemade or open-source scientific applications - this is typically how scientific libraries and software are built and installed within EasyBuild. One of the most widely used toolchain is the foss toolchain which provides the GNU compiler together with OpenMPI (for MPI support), OpenBLAS (for Blas/Lapack support), FFTW and ScaLAPACK - all of these libraries being compiled and installed with the associated GNU compiler.

   greg@frontal02:~ # ml EasyBuild/2024a 
   greg@frontal02:~ # module whatis foss/2024a 
   foss/2024a          : Description: GNU Compiler Collection (GCC) based compiler toolchain, including
                         OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
   foss/2024a          : Homepage: https://easybuild.readthedocs.io/en/master/Common-toolchains.html#foss-toolchain
   foss/2024a          : URL: https://easybuild.readthedocs.io/en/master/Common-toolchains.html#foss-toolchain

The two most common toolchains are the foss (free and open source software - based on GCC) and intel (Intel compilers and tools) toolchains. A list of all available toolchains can be obtained via the eb --list-toolchains command.

Note

The system toolchain is quite specific and a little bit against EasyBuild original philosophy. It allows users (or systems administrators) to rely on the Operating System compilers, libraries ans development headers (instead of the same tools usually provided by EasyBuild itself).

Easyconfigs

The easyconfig files (or just easyconfigs) can be considered as the recipes of the installations of scientific softwares and libraries. From a user perspective, these are the files of main interest for building an application.

An easyconfig describes all the required actions to download, build, install, check and create the associated module of a given scientific application. To illustrate the contents of an easyconfig, let's take the example of OpenMPI 5.0.3 in EasyBuild/2024a (which relies on GCC 13.3). The easyconfig is thus name OpenMPI-5.0.3-GCC-13.3.0.eb:

   name = 'OpenMPI'
   version = '5.0.3'

   homepage = 'https://www.open-mpi.org/'
   description = """The Open MPI Project is an open source MPI-3 implementation."""

   toolchain = {'name': 'GCC', 'version': '13.3.0'}

   source_urls = ['https://www.open-mpi.org/software/ompi/v%(version_major_minor)s/downloads']
   sources = [SOURCELOWER_TAR_BZ2]
   patches = [('OpenMPI-5.0.2_build-with-internal-cuda-header.patch', 1)]
   checksums = [
       {'openmpi-5.0.3.tar.bz2': '990582f206b3ab32e938aa31bbf07c639368e4405dca196fabe7f0f76eeda90b'},
       {'OpenMPI-5.0.2_build-with-internal-cuda-header.patch':
        'f52dc470543f35efef10d651dd159c771ae25f8f76a420d20d87abf4dc769ed7'},
   ]

   builddependencies = [
       ('pkgconf', '2.2.0'),
      ('Autotools', '20231222'),
   ]

   dependencies = [
       ('zlib', '1.3.1'),
       ('hwloc', '2.10.0'),
       ('libevent', '2.1.12'),
       ('UCX', '1.16.0'),
       ('libfabric', '1.21.0'),
       ('PMIx', '5.0.2'),
       ('PRRTE', '3.0.5'),
       ('UCC', '1.3.0'),
   ]

   # CUDA related patches and custom configure option can be removed if CUDA support isn't wanted.
   preconfigopts = 'gcc -Iopal/mca/cuda/include -shared opal/mca/cuda/lib/cuda.c -o opal/mca/cuda/lib/libcuda.so && '
   configopts = '--with-slurm --with-cuda=%(start_dir)s/opal/mca/cuda --with-show-load-errors=no '

   # Add knem support
   configopts += '--with-knem=/opt/knem-1.1.4.90mlnx1'

   moduleclass = 'mpi'

The table below explains all the different parameters included in this file:

Command	Description
`name`	Name of the current package to be installed
`version`	Version of the package
`source_urls`	URLs provided to download the source file
`source`	Source file and its extension
`patches`	Patch(es) to apply to the sources of the package (if applicable)
`checksums`	Check the integrity of the source and patch(es) files
`builddependencies`	Required dependencies for building the application
`dependencies`	Required dependencies for building and running the application
`preconfigopts`	Options used as prefix for `configure` command
`configopts`	Options to the `configure` command
`moduleclass`	Module category to which the application belongs

Note

There are currently around 50 supported parameters that can be used in a easyconfig. See https://docs.easybuild.io/version-specific/easyconfig-parameters/ for details.

EasyBuild in user's space

System administrators use EasyBuild to install most of the software layer. And users can also use EasyBuild to build their own. Here is a quick user guide for beginners.

Load the EasyBuild framework of your choice

   module load EasyBuild/2023a
   module load EasyBuild-user/4.9.0

Change installation path (if wanted)

By default, EasyBuild writes all build, sources, softwares and modules directories in $HOME/.local/easybuild. This default value can be changed using the installpath option as follows:
```
   eb --installpath=/path/to/my/eb/directory
```
Search available recipes

Use the -S option to search for easyconfigs providing recipes for the package you want to install.
```
   eb -S name_of_application
```
If no easyconfig is available, the output of this command will be empty. Otherwise, a list of available easyconfigs will be displayed and users can choose which package they want to install.

Dry-run

We advise users to check in the first place what EasyBuild is going to do for the installation of the selected easyconfig. Use either option --dry-run or -x (extended dry run): this will print the build overview - including dependencies and whether the dependencies are already available or not. For example:

   user@frontal02:~ # eb name_of_easyconfig.eb --dry-run
   == Temporary log file in case of crash /tmp/eb-_ujw73sq/easybuild-snzbm2ri.log
   == found valid index for /gpfs/softs/easybuild/2024a/software/EasyBuild/4.9.4/easybuild/easyconfigs, so using it...
   Dry run: printing build status of easyconfigs and dependencies
   ... 
   Skipped for convenience
   ...
   * [x] /gpfs/softs/easybuild/2024a/software/EasyBuild/4.9.4/easybuild/easyconfigs/l/libfabric/libfabric-1.18.0-GCCcore-12.3.0.eb (module: libfabric/1.18.0-GCCcore-12.3.0)
   *[ ] /gpfs/softs/easybuild/2024a/software/EasyBuild/4.9.4/easybuild/easyconfigs/l/libpciaccess/libpciaccess-0.17-GCCcore-12.3.0.eb (module: libpciaccess/0.17-GCCcore-12.3.0)
   ...
   == Temporary log file(s) /tmp/eb-_ujw73sq/easybuild-snzbm2ri.log* have been removed.
   == Temporary directory /tmp/eb-_ujw73sq has been removed.
   user@frontal02:~ #

In the above example, EasyBuild found that the libfabric dependency is already available and it will not need to be installed. On the contrary, the libpciaccess dependency does not seem to be available and will be installed.

Note

Using the dry run option, EasyBuild checks the availability of a dependency based on the presence of the associated module file. Therefore, a dependency may appear as unavailable although it has been already installed but its associated module file is hidden (ie. the module filename is preceded by a dot). you can verify the presence of a hidden module using the --hide-deps option. In the above example, it will then return:

  user@frontal02:~ # eb name_of_easyconfig.eb --dry-run --hide-deps=libpciacces
  == Temporary log file in case of crash /tmp/eb-_ujw73sq/easybuild-snzbm2ri.log
  == found valid index for /gpfs/softs/easybuild/2024a/software/EasyBuild/4.9.4/easybuild/easyconfigs, so using it...
  Dry run: printing build status of easyconfigs and dependencies
  ... 
  Skipped for convenience
  ...
  * [x] /gpfs/softs/easybuild/2024a/software/EasyBuild/4.9.4/easybuild/easyconfigs/l/libfabric/libfabric-1.18.0-GCCcore-12.3.0.eb (module: libfabric/1.18.0-GCCcore-12.3.0)
  * [x] /gpfs/softs/easybuild/2024a/software/EasyBuild/4.9.4/easybuild/easyconfigs/l/libpciaccess/libpciaccess-0.17-GCCcore-12.3.0.eb (module: libpciaccess/.0.17-GCCcore-12.3.0)
  ...
  == Temporary log file(s) /tmp/eb-_ujw73sq/easybuild-snzbm2ri.log* have been removed.
  == Temporary directory /tmp/eb-_ujw73sq has been removed.
  user@frontal02:~ #

Compilation and installation

Once these settings and verifications are complete, the user can launch the compilation of the package using the --robot option in order to instruct EasyBuild to also compile and install the required dependencies.
```
   user@frontal02:~ # eb --robot name_of_easyconfig.eb --hide-deps=(list of dependencies to hide separated by commas)
```
The package and its module will be installed in the installpath provided by the user. To see the newly installed modules, it is required to add the path where they were installed to the user's MODULEPATH envrionment variable.

Note

No manual pages are available for EasyBuild but the command eb --help prints all EasyBuild options and their purpose.

External resources

📚 You can refer to the following URLs if you need additional information: