Callbacks

LBANN has numerous callbacks that can be used to collect data about an experiment, such as scalars, metrics, weights, memory usage, images, etc. The documentation of many of these is pending; see the list of Available Callbacks for the documented ones.

The callbacks are set to execute at various times, and can be used to display images according to either a boolean output or their global sample index.

For a complete listing of callbacks and details about their functionality, please see Available Callbacks.

Using Callbacks

Callbacks are used by adding them to the python front end with the appropriate arguments and passing them as a list into the model. For example, the callbacks timer, print_statistics, and save model could be included with the following:

Python Front End

timer = lbann.CallbackTimer()
print_stats = lbann.CallbackPrintStatistics(
              batch_interval=5)
save_model = lbann.CallbackSaveModel(
             dir=".",
             disable_save_after_training=True)

callbacks = [timer,
             print_stats,
             save_model]

model = lbann.Model(num_epochs,
                    layers,
                    objective_function,
                    metrics,
                    callbacks)

Profobuf (Advanced)

callback {
  timer {
  }
  print_statistics {
    batch_interval: 5
  }
  save_model {
    dir: "."
    disable_save_after_training: true
  }
}

Available Callbacks