Base class for callbacks during training/testing. More...

#include <callback.hpp>

Inheritance diagram for lbann::callback_base:

Public Member Functions
Constructors and destructor
	callback_base (int batch_interval=1)
	Initialize a callback with an optional batch interval. More...

	callback_base (const callback_base &)=default

virtual	~callback_base ()=default

Polymorphic copy
virtual callback_base *	copy () const =0

Modifiers
virtual void	setup (trainer *t)
	Called once to set up the callback on the trainer. More...

virtual void	setup (model *m)
	Called once to set up the callback on the model (after all layers are set up). More...

Callback hooks
virtual void	on_setup_end (model *m)
	Called at the end of setup. More...

virtual void	on_train_begin (model *m)
	Called at the beginning of training. More...

virtual void	on_train_end (model *m)
	Called at the end of training. More...

virtual void	on_phase_end (model *m)
	Called at the end of every phase (multiple epochs) in a layer-wise model training. More...

virtual void	on_epoch_begin (model *m)
	Called at the beginning of each epoch. More...

virtual void	on_epoch_end (model *m)
	Called immediate after the end of each epoch. More...

virtual void	on_batch_begin (model *m)
	Called at the beginning of a (mini-)batch. More...

virtual void	on_batch_end (model *m)
	Called immediately after the end of a (mini-)batch. More...

virtual void	on_test_begin (model *m)
	Called at the beginning of testing. More...

virtual void	on_test_end (model *m)
	Called immediately after the end of testing. More...

virtual void	on_validation_begin (model *m)
	Called at the beginning of validation. More...

virtual void	on_validation_end (model *m)
	Called immediately after the end of validation. More...

virtual void	on_forward_prop_begin (model *m)
	Called when a model begins forward propagation. More...

virtual void	on_forward_prop_begin (model m, Layer l)
	Called when a layer begins forward propagation. More...

virtual void	on_forward_prop_end (model *m)
	Called when a model ends forward propagation. More...

virtual void	on_forward_prop_end (model m, Layer l)
	Called when a layer ends forward propagation. More...

virtual void	on_backward_prop_begin (model *m)
	Called when a model begins backward propagation. More...

virtual void	on_backward_prop_begin (model m, Layer l)
	Called when a layer begins backward propagation. More...

virtual void	on_backward_prop_end (model *m)
	Called when a model ends backward propagation. More...

virtual void	on_backward_prop_end (model m, Layer l)
	Called when a layer ends backward propagation. More...

virtual void	on_optimize_begin (model *m)
	Called when a model begins optimization. More...

virtual void	on_optimize_begin (model m, weights w)
	Called when weights begins optimization. More...

virtual void	on_optimize_end (model *m)
	Called when a model ends optimization. More...

virtual void	on_optimize_end (model m, weights w)
	Called when weights ends optimization. More...

virtual void	on_batch_evaluate_begin (model *m)
	Called at the beginning of a (mini-)batch evaluation (validation / testing). More...

virtual void	on_batch_evaluate_end (model *m)
	Called at the end of a (mini-)batch evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_begin (model *m)
	Called when a model begins forward propagation for evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_begin (model m, Layer l)
	Called when a layer begins forward propagation for evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_end (model *m)
	Called when a model ends forward propagation for evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_end (model m, Layer l)
	Called when a layer ends forward propagation for evaluation (validation / testing). More...

Queries
int	get_batch_interval () const
	Return the batch interval. More...

virtual std::string	name () const =0
	Return this callback's name. More...

virtual description	get_description () const
	Human-readable description. More...

Protected Member Functions
std::string	get_multi_trainer_path (const model &m, const std::string &root_dir)
	Build a standard directory hierarchy including trainer ID. More...

std::string	get_multi_trainer_ec_model_path (const model &m, const std::string &root_dir)
	Build a standard directory hierachy including trainer, execution context, and model information (in that order). More...

std::string	get_multi_trainer_model_path (const model &m, const std::string &root_dir)
	Build a standard directory hierachy including trainer, model information in that order. More...

callback_base &	operator= (const callback_base &)=default
	Copy-assignment operator. More...

Protected Attributes
int	m_batch_interval
	Batch methods should once every this many steps. More...

Serialization
template<class Archive >
void	serialize (Archive &ar)
	Store state to archive for checkpoint and restart. More...

void	write_proto (lbann_data::Callback &proto) const
	Write a protobuf description of the callback. More...

virtual void	write_specific_proto (lbann_data::Callback &proto) const =0
	Add callback specific data to prototext. More...

Detailed Description

Base class for callbacks during training/testing.

The method of each callback is called at a given point during training or testing by the model. Implement whichever ones you care about. Callbacks may be passed a lbann_summary instance, which they can use to log any relevant information.

Definition at line 76 of file callback.hpp.

Constructor & Destructor Documentation

◆ callback_base() [1/2]

lbann::callback_base::callback_base ( int batch_interval = 1 )

inline

Initialize a callback with an optional batch interval.

Definition at line 84 of file callback.hpp.

◆ callback_base() [2/2]

lbann::callback_base::callback_base ( const callback_base & )

default

◆ ~callback_base()

virtual lbann::callback_base::~callback_base ( )

virtualdefault

Member Function Documentation

◆ copy()

virtual callback_base* lbann::callback_base::copy ( ) const

pure virtual

◆ get_batch_interval()

int lbann::callback_base::get_batch_interval ( ) const

inline

Return the batch interval.

Definition at line 194 of file callback.hpp.

Here is the call graph for this function:

◆ get_description()

virtual description lbann::callback_base::get_description ( ) const

virtual

Human-readable description.

◆ get_multi_trainer_ec_model_path()

std::string lbann::callback_base::get_multi_trainer_ec_model_path	(	const model &	m,
		const std::string &	root_dir
	)

protected

Build a standard directory hierachy including trainer, execution context, and model information (in that order).

◆ get_multi_trainer_model_path()

std::string lbann::callback_base::get_multi_trainer_model_path	(	const model &	m,
		const std::string &	root_dir
	)

protected

Build a standard directory hierachy including trainer, model information in that order.

◆ get_multi_trainer_path()

std::string lbann::callback_base::get_multi_trainer_path	(	const model &	m,
		const std::string &	root_dir
	)

protected

Build a standard directory hierarchy including trainer ID.

◆ name()

virtual std::string lbann::callback_base::name ( ) const

pure virtual

Return this callback's name.

Implemented in lbann::callback::cosine_decay_learning_rate, lbann::callback::optimizerwise_adaptive_learning_rate, lbann::callback::poly_learning_rate, lbann::callback::linear_growth_learning_rate, lbann::callback::drop_fixed_learning_rate, lbann::callback::adaptive_learning_rate, lbann::callback::summarize_images, lbann::callback::checkpoint, lbann::callback::set_learning_rate, lbann::callback::minibatch_schedule, lbann::callback::step_learning_rate, lbann::callback::step_minibatch, lbann::callback::debug_io, lbann::callback::ltfb, lbann::callback::perturb_adam, lbann::callback::dump_outputs, lbann::callback::timer, lbann::callback::perturb_learning_rate, lbann::callback::profiler, lbann::callback::dump_minibatch_sample_indices, lbann::callback::dump_weights, lbann::callback::load_model, lbann::callback::save_model, lbann::callback::check_gradients, lbann::callback::export_onnx, lbann::callback::save_topk_models, lbann::callback::alternate_updates, lbann::callback::mixup, lbann::callback::dump_gradients, lbann::callback::perturb_weights, lbann::callback::progress_bar, lbann::callback::sync_layers, lbann::callback::check_small, lbann::callback::monitor_io, lbann::callback::summary, lbann::callback::check_dataset, lbann::callback::check_nan, lbann::callback::confusion_matrix, lbann::callback::debug, lbann::callback::print_statistics, lbann::callback::hang, lbann::callback::replace_weights, lbann::callback::save_images, lbann::callback::set_weights_value, lbann::callback::timeline, lbann::callback::compute_model_size, lbann::callback::early_stopping, lbann::callback::dump_error_signals, lbann::callback::dump_model_graph, lbann::callback::perturb_dropout, lbann::callback::print_model_description, lbann::callback::gpu_memory_usage, lbann::callback::check_init, and lbann::callback::check_metric.

◆ on_backward_prop_begin() [1/2]

virtual void lbann::callback_base::on_backward_prop_begin ( model * m )

inlinevirtual

Called when a model begins backward propagation.

Reimplemented in lbann::callback::profiler.

Definition at line 148 of file callback.hpp.

Here is the caller graph for this function:

◆ on_backward_prop_begin() [2/2]

virtual void lbann::callback_base::on_backward_prop_begin	(	model *	m,
		Layer *	l
	)

inlinevirtual

Called when a layer begins backward propagation.

Reimplemented in lbann::callback::debug, lbann::callback::timeline, and lbann::callback::profiler.

Definition at line 150 of file callback.hpp.

◆ on_backward_prop_end() [1/2]

virtual void lbann::callback_base::on_backward_prop_end ( model * m )

inlinevirtual

Called when a model ends backward propagation.

Reimplemented in lbann::callback::learning_rate, lbann::callback::dump_gradients, lbann::callback::profiler, lbann::callback::check_small, and lbann::callback::check_nan.

Definition at line 152 of file callback.hpp.

Here is the caller graph for this function:

◆ on_backward_prop_end() [2/2]

virtual void lbann::callback_base::on_backward_prop_end	(	model *	m,
		Layer *	l
	)

inlinevirtual

Called when a layer ends backward propagation.

Reimplemented in lbann::callback::debug, lbann::callback::timeline, lbann::callback::profiler, lbann::callback::sync_layers, lbann::callback::dump_error_signals, and lbann::callback::check_nan.

Definition at line 154 of file callback.hpp.

◆ on_batch_begin()

virtual void lbann::callback_base::on_batch_begin ( model * m )

inlinevirtual

Called at the beginning of a (mini-)batch.

Reimplemented in lbann::callback::checkpoint, lbann::callback::ltfb, lbann::callback::perturb_adam, lbann::callback::perturb_learning_rate, lbann::callback::timer, lbann::callback::perturb_weights, lbann::callback::alternate_updates, lbann::callback::debug, lbann::callback::compute_model_size, lbann::callback::set_weights_value, and lbann::callback::profiler.

Definition at line 128 of file callback.hpp.

◆ on_batch_end()

virtual void lbann::callback_base::on_batch_end ( model * m )

inlinevirtual

Called immediately after the end of a (mini-)batch.

Reimplemented in lbann::callback::confusion_matrix, lbann::callback::timer, lbann::callback::debug, lbann::callback::check_small, lbann::callback::check_nan, lbann::callback::summary, lbann::callback::replace_weights, and lbann::callback::profiler.

Definition at line 130 of file callback.hpp.

◆ on_batch_evaluate_begin()

virtual void lbann::callback_base::on_batch_evaluate_begin ( model * m )

inlinevirtual

Called at the beginning of a (mini-)batch evaluation (validation / testing).

Reimplemented in lbann::callback::timer, lbann::callback::debug, and lbann::callback::profiler.

Definition at line 167 of file callback.hpp.

◆ on_batch_evaluate_end()

virtual void lbann::callback_base::on_batch_evaluate_end ( model * m )

inlinevirtual

Called at the end of a (mini-)batch evaluation (validation / testing).

Reimplemented in lbann::callback::summarize_images, lbann::callback::timer, lbann::callback::confusion_matrix, lbann::callback::debug, and lbann::callback::profiler.

Definition at line 171 of file callback.hpp.

◆ on_epoch_begin()

virtual void lbann::callback_base::on_epoch_begin ( model * m )

inlinevirtual

Called at the beginning of each epoch.

Reimplemented in lbann::callback::checkpoint, lbann::callback::confusion_matrix, lbann::callback::debug_io, lbann::callback::progress_bar, lbann::callback::print_statistics, lbann::callback::timer, lbann::callback::gpu_memory_usage, and lbann::callback::profiler.

Definition at line 124 of file callback.hpp.

◆ on_epoch_end()

virtual void lbann::callback_base::on_epoch_end ( model * m )

inlinevirtual

Called immediate after the end of each epoch.

Reimplemented in lbann::callback::dump_weights, lbann::callback::confusion_matrix, lbann::callback::learning_rate, lbann::callback::monitor_io, lbann::callback::summary, lbann::callback::timer, lbann::callback::print_statistics, lbann::callback::save_images, lbann::callback::check_dataset, lbann::callback::variable_minibatch, lbann::callback::check_metric, and lbann::callback::profiler.

Definition at line 126 of file callback.hpp.

◆ on_evaluate_forward_prop_begin() [1/2]

virtual void lbann::callback_base::on_evaluate_forward_prop_begin ( model * m )

inlinevirtual

Called when a model begins forward propagation for evaluation (validation / testing).

Reimplemented in lbann::callback::profiler.

Definition at line 175 of file callback.hpp.

Here is the caller graph for this function:

◆ on_evaluate_forward_prop_begin() [2/2]

virtual void lbann::callback_base::on_evaluate_forward_prop_begin	(	model *	m,
		Layer *	l
	)

inlinevirtual

Called when a layer begins forward propagation for evaluation (validation / testing).

Reimplemented in lbann::callback::debug, lbann::callback::debug_io, and lbann::callback::profiler.

Definition at line 179 of file callback.hpp.

◆ on_evaluate_forward_prop_end() [1/2]

virtual void lbann::callback_base::on_evaluate_forward_prop_end ( model * m )

inlinevirtual

Called when a model ends forward propagation for evaluation (validation / testing).

Reimplemented in lbann::callback::profiler.

Definition at line 183 of file callback.hpp.

Here is the caller graph for this function:

◆ on_evaluate_forward_prop_end() [2/2]

virtual void lbann::callback_base::on_evaluate_forward_prop_end	(	model *	m,
		Layer *	l
	)

inlinevirtual

Called when a layer ends forward propagation for evaluation (validation / testing).

Reimplemented in lbann::callback::debug, lbann::callback::dump_outputs, lbann::callback::dump_minibatch_sample_indices, lbann::callback::profiler, and lbann::callback::check_dataset.

Definition at line 187 of file callback.hpp.

◆ on_forward_prop_begin() [1/2]

virtual void lbann::callback_base::on_forward_prop_begin ( model * m )

inlinevirtual

Called when a model begins forward propagation.

Reimplemented in lbann::callback::progress_bar, and lbann::callback::profiler.

Definition at line 140 of file callback.hpp.

Here is the caller graph for this function:

◆ on_forward_prop_begin() [2/2]

virtual void lbann::callback_base::on_forward_prop_begin	(	model *	m,
		Layer *	l
	)

inlinevirtual

Called when a layer begins forward propagation.

Reimplemented in lbann::callback::debug, lbann::callback::timeline, lbann::callback::debug_io, and lbann::callback::profiler.

Definition at line 142 of file callback.hpp.

◆ on_forward_prop_end() [1/2]

virtual void lbann::callback_base::on_forward_prop_end ( model * m )

inlinevirtual

Called when a model ends forward propagation.

Reimplemented in lbann::callback::profiler.

Definition at line 144 of file callback.hpp.

Here is the caller graph for this function:

◆ on_forward_prop_end() [2/2]

virtual void lbann::callback_base::on_forward_prop_end	(	model *	m,
		Layer *	l
	)

inlinevirtual

Called when a layer ends forward propagation.

Reimplemented in lbann::callback::debug, lbann::callback::dump_outputs, lbann::callback::dump_minibatch_sample_indices, lbann::callback::mixup, lbann::callback::sync_layers, lbann::callback::timeline, lbann::callback::profiler, lbann::callback::check_small, lbann::callback::check_nan, and lbann::callback::check_dataset.

Definition at line 146 of file callback.hpp.

◆ on_optimize_begin() [1/2]

virtual void lbann::callback_base::on_optimize_begin ( model * m )

inlinevirtual

Called when a model begins optimization.

Reimplemented in lbann::callback::profiler.

Definition at line 156 of file callback.hpp.

Here is the caller graph for this function:

◆ on_optimize_begin() [2/2]

virtual void lbann::callback_base::on_optimize_begin	(	model *	m,
		weights *	w
	)

inlinevirtual

Called when weights begins optimization.

Reimplemented in lbann::callback::debug, lbann::callback::profiler, and lbann::callback::timeline.

Definition at line 158 of file callback.hpp.

◆ on_optimize_end() [1/2]

virtual void lbann::callback_base::on_optimize_end ( model * m )

inlinevirtual

Called when a model ends optimization.

Reimplemented in lbann::callback::profiler.

Definition at line 160 of file callback.hpp.

Here is the caller graph for this function:

◆ on_optimize_end() [2/2]

virtual void lbann::callback_base::on_optimize_end	(	model *	m,
		weights *	w
	)

inlinevirtual

Called when weights ends optimization.

Reimplemented in lbann::callback::debug, lbann::callback::profiler, and lbann::callback::timeline.

Definition at line 162 of file callback.hpp.

◆ on_phase_end()

virtual void lbann::callback_base::on_phase_end ( model * m )

inlinevirtual

Called at the end of every phase (multiple epochs) in a layer-wise model training.

Definition at line 122 of file callback.hpp.

◆ on_setup_end()

virtual void lbann::callback_base::on_setup_end ( model * m )

inlinevirtual

Called at the end of setup.

Reimplemented in lbann::callback::dump_model_graph, and lbann::callback::print_model_description.

Definition at line 114 of file callback.hpp.

◆ on_test_begin()

virtual void lbann::callback_base::on_test_begin ( model * m )

inlinevirtual

Called at the beginning of testing.

Reimplemented in lbann::callback::debug_io, lbann::callback::load_model, lbann::callback::confusion_matrix, lbann::callback::timer, and lbann::callback::profiler.

Definition at line 132 of file callback.hpp.

◆ on_test_end()

virtual void lbann::callback_base::on_test_end ( model * m )

inlinevirtual

Called immediately after the end of testing.

Reimplemented in lbann::callback::check_gradients, lbann::callback::confusion_matrix, lbann::callback::save_topk_models, lbann::callback::timer, lbann::callback::monitor_io, lbann::callback::summary, lbann::callback::print_statistics, lbann::callback::save_images, lbann::callback::check_dataset, lbann::callback::check_metric, and lbann::callback::profiler.

Definition at line 134 of file callback.hpp.

◆ on_train_begin()

virtual void lbann::callback_base::on_train_begin ( model * m )

inlinevirtual

Called at the beginning of training.

Reimplemented in lbann::callback::checkpoint, lbann::callback::ltfb, lbann::callback::dump_weights, lbann::callback::load_model, lbann::callback::hang, lbann::callback::summary, lbann::callback::timeline, lbann::callback::variable_minibatch, and lbann::callback::check_init.

Definition at line 116 of file callback.hpp.

◆ on_train_end()

virtual void lbann::callback_base::on_train_end ( model * m )

inlinevirtual

Called at the end of training.

Reimplemented in lbann::callback::checkpoint, lbann::callback::export_onnx, lbann::callback::check_gradients, lbann::callback::save_model, and lbann::callback::timeline.

Definition at line 118 of file callback.hpp.

◆ on_validation_begin()

virtual void lbann::callback_base::on_validation_begin ( model * m )

inlinevirtual

Called at the beginning of validation.

Reimplemented in lbann::callback::checkpoint, lbann::callback::debug_io, lbann::callback::confusion_matrix, lbann::callback::timer, and lbann::callback::profiler.

Definition at line 136 of file callback.hpp.

◆ on_validation_end()

virtual void lbann::callback_base::on_validation_end ( model * m )

inlinevirtual

Called immediately after the end of validation.

Reimplemented in lbann::callback::check_gradients, lbann::callback::confusion_matrix, lbann::callback::timer, lbann::callback::print_statistics, lbann::callback::check_dataset, lbann::callback::early_stopping, lbann::callback::check_metric, and lbann::callback::profiler.

Definition at line 138 of file callback.hpp.

◆ operator=()

callback_base& lbann::callback_base::operator= ( const callback_base & )

protecteddefault

Copy-assignment operator.

Performs a shallow (pointer) copy of the summarizer.

◆ serialize()

template<class Archive >

void lbann::callback_base::serialize ( Archive & ar )

Store state to archive for checkpoint and restart.

◆ setup() [1/2]

virtual void lbann::callback_base::setup ( trainer * t )

inlinevirtual

Called once to set up the callback on the trainer.

Reimplemented in lbann::callback::checkpoint.

Definition at line 102 of file callback.hpp.

◆ setup() [2/2]

virtual void lbann::callback_base::setup ( model * m )

inlinevirtual

Called once to set up the callback on the model (after all layers are set up).

Reimplemented in lbann::callback::cosine_decay_learning_rate, lbann::callback::poly_learning_rate, lbann::callback::linear_growth_learning_rate, lbann::callback::checkpoint, lbann::callback::perturb_adam, lbann::callback::perturb_learning_rate, lbann::callback::perturb_weights, lbann::callback::alternate_updates, lbann::callback::confusion_matrix, lbann::callback::learning_rate, lbann::callback::compute_model_size, lbann::callback::perturb_dropout, lbann::callback::replace_weights, lbann::callback::hang, and lbann::callback::print_statistics.

Definition at line 107 of file callback.hpp.

◆ write_proto()

void lbann::callback_base::write_proto ( lbann_data::Callback & proto ) const

Write a protobuf description of the callback.

◆ write_specific_proto()

virtual void lbann::callback_base::write_specific_proto ( lbann_data::Callback & proto ) const

protectedpure virtual

Add callback specific data to prototext.

Here is the caller graph for this function:

Member Data Documentation

◆ m_batch_interval

int lbann::callback_base::m_batch_interval

protected

Batch methods should once every this many steps.

Todo:: Make callback data private

Definition at line 247 of file callback.hpp.

The documentation for this class was generated from the following file:

callback.hpp

Public Member Functions

Protected Member Functions

Protected Attributes

Serialization

Detailed Description

Constructor & Destructor Documentation

◆ callback_base() [1/2]

◆ callback_base() [2/2]

◆ ~callback_base()

Member Function Documentation

◆ copy()

◆ get_batch_interval()

◆ get_description()

◆ get_multi_trainer_ec_model_path()

◆ get_multi_trainer_model_path()

◆ get_multi_trainer_path()

◆ name()

◆ on_backward_prop_begin() [1/2]

◆ on_backward_prop_begin() [2/2]

◆ on_backward_prop_end() [1/2]

◆ on_backward_prop_end() [2/2]

◆ on_batch_begin()

◆ on_batch_end()

◆ on_batch_evaluate_begin()

◆ on_batch_evaluate_end()

◆ on_epoch_begin()

◆ on_epoch_end()

◆ on_evaluate_forward_prop_begin() [1/2]

◆ on_evaluate_forward_prop_begin() [2/2]

◆ on_evaluate_forward_prop_end() [1/2]

◆ on_evaluate_forward_prop_end() [2/2]

◆ on_forward_prop_begin() [1/2]

◆ on_forward_prop_begin() [2/2]

◆ on_forward_prop_end() [1/2]

◆ on_forward_prop_end() [2/2]

◆ on_optimize_begin() [1/2]

◆ on_optimize_begin() [2/2]

◆ on_optimize_end() [1/2]

◆ on_optimize_end() [2/2]

◆ on_phase_end()

◆ on_setup_end()

◆ on_test_begin()

◆ on_test_end()

◆ on_train_begin()

◆ on_train_end()

◆ on_validation_begin()

◆ on_validation_end()

◆ operator=()

◆ serialize()

◆ setup() [1/2]

◆ setup() [2/2]

◆ write_proto()

◆ write_specific_proto()

Member Data Documentation

◆ m_batch_interval