Building LBANN in containers
We currently support Docker and Singularity.
Warning
The container builds are not regularly tested. If problems are encountered, please open an issue.
Singularity
First build a Singularity container with the lbann.def file:
sudo singularity build --writable lbann.img lbann.def
Note
Building the image requires root access.
Note
--writable
allows users to make changes inside the
container. This is required for LLNL’s LC systems.
This will create a container called lbann.img which can be used to invoke lbann on any system with Singularity and Open-MPI installed.
Customizing Configuration in lbann.def
Singularity is designed to take advantage of underlying HPC resources. The lbann.def file in this directory specifically installs packages necessary for infiniband interconnects (lines 15-19). It builds openmpi outside of the spack step to ensure it is built with infiniband support (lines 37-55). Experienced users should modify these sections to match with the underlying resources they intend to run on. This defintion file also builds GCC version 4.9.3, and uses it to build Open-MPI and LBANN (lines 33-35). This is also customized to run on specific LLNL LC resources, and can be modified depending on the users system.
Running LBANN with Singualrity
To run LBANN use mpirun and singularity’s execute command:
salloc -N2
mpirun -np 4 singularity exec -B /p:/p lbann.img \
/lbann/spack_builds/singularity/model_zoo/lbann \
--model=/lbann/model_zoo/models/lenet_mnist/model_lenet_mnist.prototext \
--reader=/lbann/model_zoo/data_readers/data_reader_mnist.prototext \
--optimizer=/lbann/model_zoo/optimizers/opt_adagrad.prototext
Note
The -B
Singularity command binds directories from
the surrounding filesystem to the container. Be sure to
include any necessary files using this command (e.g., model
prototext files, datasets, etc). Alternatively, system
administrators are capable of allowing a Singularity
container to utilize the host’s filesystem. This is done by
changing MOUNT HOSTFS in the Singularity config file.
Docker
First build a Docker image with the Dockerfile. From whichever directory contains the Dockerfile:
docker build -t dockban .
Note
The -t
flag specifies an identifying tag for this
image. “dockban” can be changed to any desired tag.
Customizing Configuration in Dockerfile
The Dockerfile container defintion is less complicated than its
Singularity counterpart. GCC 7.1.0 is built and registered with spack
in lines 19-21. Users can change this, as well as LBANN-specific build
options in spack (line 22). For example, to add gpu support, a user can
add +gpu
to this line.
Running LBANN with Docker
This LBANN build also uses Open-MPI. Thus, LBANN can be launched with
mpirun
here as well. However, this example will just show the
single-process invocation.
Start a docker container from the previously created image, and attach
to it. Make sure to bind any necessary directories using -v
:
docker run -it -v $HOME/MNIST:/MNIST dockban
Run LBANN as you would outside of a container:
./spack_build/docker_build/model_zoo/lbann \
--model=model_zoo/models/lenet_mnist/model_lenet_mnist.prototext \
--reader=model_zoo/data_readers/data_reader_mnist.prototext \
--optimizer=model_zoo/optimizers/opt_sgd.prototext