Building LBANN with CMake

LBANN uses CMake for its build system and a version newer than or equal to 3.12.0 is required. LBANN development is done primarily on UNIX-based platforms. As such, the build is tested regularly on Linux-based machines, occasionally on OSX, and never on Windows machines.

The CMake build system is available to any users or developers who need a more fine-grained level of control over dependency resolution and/or features of LBANN. The LBANN team has made an effort to expose as many knobs as possible through the Spack package but if something is missing, please open an issue.

It is required that LBANN be built out-of-source. That is, CMake must not be invoked in a directory containing a CMakeLists.

Dependencies

The following packages and tools are required to build LBANN. All packages listed below may be installed using Spack. See the Spack installation instructions for more details on using Spack to build a complete LBANN environment.

The following basic tools are required.

  • A C++11-compliant compiler.

  • OpenMP, version 3.0 or newer.

  • An MPI-3.0 implementation.

  • CEREAL is used to handle complex serialization tasks.

  • CMake, version 3.12 or newer.

The following LLNL-maintained packages are required.

  • Hydrogen is a fork of the Elemental distributed dense linear-algebra library and it may be installed via Spack using the package name “hydrogen”. If CUDA support is enabled in Hydrogen, LBANN will inherit this support.

The following third-party packages are required.

  • CNPY is used to ingest data in NumPy format. In principle this should be optional, but at time of writing, LBANN will not build without it.

  • OpenCV is used to preprocess image data. For performance reasons, it is recommend to build OpenCV with JPEG-turbo for JPEG format support.

  • ProtoBuf is used to express models in a portable format.

The following LLNL-maintained packages are optional.

  • Aluminum is a communication library optimized for machine learning and interaction with GPUs. We cannot recommend its use strongly enough. It can be built using Spack.

  • CONDUIT is used to ingest structured data produced by scientific simulations.

  • DiHydrogen is going to become the linear algebra interface; currently, it can be used to manage metaprogramming and some utilities.

The following third-party packages are optional.

  • CUDA. The development team currently uses CUDA version 9.2. Building with CUDA support requires that Hydrogen has been built with CUDA support (see below).

  • cuDNN is required if building LBANN with CUDA support. It is freely available as a binary distribution from NVIDIA.

  • HWLOC. HWLOC enables LBANN to make certain optimizations based on the hardware topology. Its use is strongly recommended.

  • NVTX. LBANN supports some improved annotations for NVPROF using NVTX. NVTX is provided as part of the CUDA toolkit.

  • VTune. LBANN supports some improved annotations for VTune.

LBANN CMake options

The following options are exposed in the CMake build system.

  • LBANN_WITH_ALUMINUM (Default: OFF): Use the Aluminum communication package. This will be set to ON automatically if Hydrogen was built with Aluminum.

  • LBANN_WITH_CNPY (Default: ON): Build with support for CNPY for reading Numpy data.

  • LBANN_WITH_CONDUIT (Default: OFF): Build with support for CONDUIT.

  • LBANN_WITH_DIHYDROGEN (Default: OFF): Build with DiHydrogen support. This will replace temporary implementations in LBANN with permanent implementations from DiHydrogen.

  • LBANN_WITH_NVPROF (Default: OFF): Build with extra annotations for NVPROF.

  • LBANN_WITH_TOPO_AWARE (Default: ON): Use HWLOC for topology-aware choices.

  • LBANN_WITH_TBINF (Default: ON): Enable the Tensorboard interace.

  • LBANN_WITH_VTUNE (Default: OFF): Build with extra annotations for VTune.

  • LBANN_DETERMINISTIC (Default: OFF): Force as much of the code as possible to be deterministic. This is not a guarantee as certain operations in third-party libraries cannot be forced into a deterministic mode, especially for CUDA-enabled builds.

  • LBANN_SEQUENTIAL_INITIALIZATION (Default: OFF): Force sequentially consistent initialization of data structures.

  • LBANN_WARNINGS_AS_ERRORS (Default: OFF): Promote compiler warnings to errors. This should be used by developers only. Developers are encouraged to build with this ON prior to merging any code into the repository.

  • LBANN_USE_PROTOBUF_MODULE (Default: OFF): Search for Protobuf using CMake’s FindProtobuf.cmake module instead of the Protobuf config file. This is useful on platforms with differently architected compute nodes or when the config method is inexplicably failing.

The following variables may also be set:

  • LBANN_DATATYPE (Default: float): The datatype to use for training. Currently this must be float or double.

The following variable has been deprecated and removed:

  • LBANN_WITH_CUDA. The “CUDA-ness” of LBANN is now tied 1:1 with the “CUDA-ness” of Hydrogen. At present, it seems like unnecessary overhead to support the situation in which Hydrogen has CUDA support but LBANN doesn’t want to use it until a compelling use-case reveals itself.

Controlling dependency resolution

The following variables may be set with CMake to identify dependencies that are not installed into the “typical” locations that CMake searches by default. They may be either exported into the environment used by CMake using whatever mechanisms are allowed by the shell or passed to CMake as a cache variable (e.g., cmake -DPKG_DIR=/path/to/pkg). The latter option is recommended.

  • Aluminum_DIR or ALUMINUM_DIR or AL_DIR: The path to either the Aluminum installation prefix or the AluminumConfig.cmake file. If Hydrogen has not been built with Aluminum support, set LBANN_WITH_ALUMINUM=ON to enable Aluminum support.

  • CEREAL_DIR: The path to either the CEREAL installation prefix or the cereal-config.cmake file.

  • CNPY_DIR: The path to the CNPY installation prefix. Must set LBANN_WITH_CNPY=ON to enable CNPY support.

  • CONDUIT_DIR or CONDUIT_DIR: The path to either the CONDUIT installation prefix or the ConduitConfig.cmake file. Must set LBANN_WITH_CONDUIT=ON to enable CONDUIT support.

  • DIHYDROGEN_DIR or H2_DIR: The path to either the DiHydrogen installation prefix or the DiHydrogenConfig.cmake file. Alternatively, DiHydrogen_DIR can be set to the path of the DiHydrogenConfig.cmake file.

  • HDF5_DIR: The path to either the HDF5 installation prefix or the hdf5_config.cmake file. There is a known issue with CONDUIT that it may link to HDF5 but not properly export that dependency.

  • HWLOC_DIR: The path to the HWLOC installation prefix. Must set LBANN_WITH_HWLOC=ON to enable HWLOC support.

  • Hydrogen_DIR or HYDROGEN_DIR: The path to either the Hydrogen installation prefix or the HydrogenConfig.cmake file.

  • NVTX_DIR: The path the the prefix of NVTX. This should not be used except in circumstances in which one might want to link to a different NVTX installation than the CUDA toolkit. Under normal circumstances, if CUDA was found without issue, NVTX should be as well.

  • OpenCV_DIR or OPENCV_DIR: The path to either the OpenCV installation prefix or the OpenCVConfig.cmake file.

  • Protobuf_DIR or PROTOBUF_DIR: The path to either the Protobuf installation prefix or the protobuf-config.cmake file.

  • VTUNE_DIR: The path to the prefix of the VTune (or Intel compiler suite) installation.

Compilers, include CUDA compilers, are found using the default CMake mechanisms, as are OpenMP and MPI. Thus, the process of finding these tools can be manipulated using the usual CMake mechanisms and/or cache variables as documented by CMake.

Except where otherwise noted, this list attempts to address the first level of dependencies of LBANN, that is, those that are one edge away in the DAG. If deeper dependency issues appear, please consult the documentation of the packages that are causing the issues as they may require additional CMake/environment flags to be set before properly resolving.

Building JAG utilities

The JAG utility executables are not part of the all target. In order to use or install them, they must be built using the jag-utils target. In order to install them, this must be done before installing.

# Configure LBANN
cmake <see below... or above> /path/to/lbann

# Build main LBANN library and front-ends
cmake --build .

# If JAG utilities are required, build them
cmake --build . --target jag-utils

# Install all (built) targets
cmake --build . --target install

Example CMake invocation

A sample CMake build for LBANN might look like the following.

cmake \
  -D LBANN_WITH_CUDA:BOOL=ON \
  -D LBANN_WITH_NVPROF:BOOL=ON \
  -D LBANN_DATATYPE:STRING=float \
  -D Hydrogen_DIR:PATH=/path/to/hydrogen \
  -D HWLOC_DIR:PATH=/path/to/hwloc \
  /path/to/lbann