Gradient checking callback. More...

#include <check_gradients.hpp>

Inheritance diagram for lbann::callback::check_gradients:

Collaboration diagram for lbann::callback::check_gradients:

Public Member Functions
	check_gradients (std::set< execution_mode > modes={}, DataType step_size=DataType(0), bool verbose=false, bool error_on_failure=false)

check_gradients *	copy () const override

std::string	name () const override
	Return this callback's name. More...

void	on_train_end (model *m) override
	Called at the end of training. More...

void	on_validation_end (model *m) override
	Called immediately after the end of validation. More...

void	on_test_end (model *m) override
	Called immediately after the end of testing. More...

Serialization
template<class Archive >
void	serialize (Archive &ar)
	Store state to archive for checkpoint and restart. More...

Public Member Functions inherited from lbann::callback_base
	callback_base (int batch_interval=1)
	Initialize a callback with an optional batch interval. More...

	callback_base (const callback_base &)=default

virtual	~callback_base ()=default

virtual void	setup (trainer *t)
	Called once to set up the callback on the trainer. More...

virtual void	setup (model *m)
	Called once to set up the callback on the model (after all layers are set up). More...

virtual void	on_setup_end (model *m)
	Called at the end of setup. More...

virtual void	on_train_begin (model *m)
	Called at the beginning of training. More...

virtual void	on_phase_end (model *m)
	Called at the end of every phase (multiple epochs) in a layer-wise model training. More...

virtual void	on_epoch_begin (model *m)
	Called at the beginning of each epoch. More...

virtual void	on_epoch_end (model *m)
	Called immediate after the end of each epoch. More...

virtual void	on_batch_begin (model *m)
	Called at the beginning of a (mini-)batch. More...

virtual void	on_batch_end (model *m)
	Called immediately after the end of a (mini-)batch. More...

virtual void	on_test_begin (model *m)
	Called at the beginning of testing. More...

virtual void	on_validation_begin (model *m)
	Called at the beginning of validation. More...

virtual void	on_forward_prop_begin (model *m)
	Called when a model begins forward propagation. More...

virtual void	on_forward_prop_begin (model m, Layer l)
	Called when a layer begins forward propagation. More...

virtual void	on_forward_prop_end (model *m)
	Called when a model ends forward propagation. More...

virtual void	on_forward_prop_end (model m, Layer l)
	Called when a layer ends forward propagation. More...

virtual void	on_backward_prop_begin (model *m)
	Called when a model begins backward propagation. More...

virtual void	on_backward_prop_begin (model m, Layer l)
	Called when a layer begins backward propagation. More...

virtual void	on_backward_prop_end (model *m)
	Called when a model ends backward propagation. More...

virtual void	on_backward_prop_end (model m, Layer l)
	Called when a layer ends backward propagation. More...

virtual void	on_optimize_begin (model *m)
	Called when a model begins optimization. More...

virtual void	on_optimize_begin (model m, weights w)
	Called when weights begins optimization. More...

virtual void	on_optimize_end (model *m)
	Called when a model ends optimization. More...

virtual void	on_optimize_end (model m, weights w)
	Called when weights ends optimization. More...

virtual void	on_batch_evaluate_begin (model *m)
	Called at the beginning of a (mini-)batch evaluation (validation / testing). More...

virtual void	on_batch_evaluate_end (model *m)
	Called at the end of a (mini-)batch evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_begin (model *m)
	Called when a model begins forward propagation for evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_begin (model m, Layer l)
	Called when a layer begins forward propagation for evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_end (model *m)
	Called when a model ends forward propagation for evaluation (validation / testing). More...

virtual void	on_evaluate_forward_prop_end (model m, Layer l)
	Called when a layer ends forward propagation for evaluation (validation / testing). More...

int	get_batch_interval () const
	Return the batch interval. More...

virtual description	get_description () const
	Human-readable description. More...

template<class Archive >
void	serialize (Archive &ar)
	Store state to archive for checkpoint and restart. More...

void	write_proto (lbann_data::Callback &proto) const
	Write a protobuf description of the callback. More...

Private Member Functions
void	write_specific_proto (lbann_data::Callback &proto) const final

void	do_check_gradients (model &m) const

Private Attributes
std::set< execution_mode >	m_modes

EvalType	m_step_size

bool	m_verbose

bool	m_error_on_failure

Additional Inherited Members
Protected Member Functions inherited from lbann::callback_base
std::string	get_multi_trainer_path (const model &m, const std::string &root_dir)
	Build a standard directory hierarchy including trainer ID. More...

std::string	get_multi_trainer_ec_model_path (const model &m, const std::string &root_dir)
	Build a standard directory hierachy including trainer, execution context, and model information (in that order). More...

std::string	get_multi_trainer_model_path (const model &m, const std::string &root_dir)
	Build a standard directory hierachy including trainer, model information in that order. More...

callback_base &	operator= (const callback_base &)=default
	Copy-assignment operator. More...

Protected Attributes inherited from lbann::callback_base
int	m_batch_interval
	Batch methods should once every this many steps. More...

Detailed Description

Gradient checking callback.

Gradient checking is performed at the end of each execution mode phase. Using a fourth-order finite difference scheme, a numerical partial derivative is computed for every weight parameter. If the numerical derivative differs signifcantly from the analytical derivative computed during backprop, the gradient check has failed.

Definition at line 48 of file check_gradients.hpp.

Constructor & Destructor Documentation

◆ check_gradients()

lbann::callback::check_gradients::check_gradients	(	std::set< execution_mode >	modes = `{}`,
		DataType	step_size = `DataType(0)`,
		bool	verbose = `false`,
		bool	error_on_failure = `false`
	)

Parameters

modes	Execution modes with gradient checks. If none are provided, gradient checking is performed for every execution mode.
step_size	Step size for numerical differentiation (with a step size of zero, the step size is estimated to minimize the numerical error).
verbose	Whether to print results for each parameter.
error_on_failure	Whether to throw an exception for large gradient errors.

Here is the caller graph for this function:

Member Function Documentation

◆ copy()

check_gradients* lbann::callback::check_gradients::copy ( ) const

inlineoverridevirtual

Implements lbann::callback_base.

Definition at line 68 of file check_gradients.hpp.

Here is the call graph for this function:

◆ do_check_gradients()

void lbann::callback::check_gradients::do_check_gradients ( model & m ) const

private

Does nothing if current execution mode is not in m_modes.

Here is the caller graph for this function:

◆ name()

std::string lbann::callback::check_gradients::name ( ) const

inlineoverridevirtual

Return this callback's name.

Implements lbann::callback_base.

Definition at line 69 of file check_gradients.hpp.

◆ on_test_end()

void lbann::callback::check_gradients::on_test_end ( model * m )

inlineoverridevirtual

Called immediately after the end of testing.

Reimplemented from lbann::callback_base.

Definition at line 72 of file check_gradients.hpp.

Here is the call graph for this function:

◆ on_train_end()

void lbann::callback::check_gradients::on_train_end ( model * m )

inlineoverridevirtual

Called at the end of training.

Reimplemented from lbann::callback_base.

Definition at line 70 of file check_gradients.hpp.

Here is the call graph for this function:

◆ on_validation_end()

void lbann::callback::check_gradients::on_validation_end ( model * m )

inlineoverridevirtual

Called immediately after the end of validation.

Reimplemented from lbann::callback_base.

Definition at line 71 of file check_gradients.hpp.

Here is the call graph for this function:

◆ serialize()

template<class Archive >

void lbann::callback::check_gradients::serialize ( Archive & ar )

Store state to archive for checkpoint and restart.

Here is the caller graph for this function:

◆ write_specific_proto()

void lbann::callback::check_gradients::write_specific_proto ( lbann_data::Callback & proto ) const

finalprivatevirtual

Add callback specific data to prototext

Implements lbann::callback_base.

Here is the caller graph for this function:

Member Data Documentation

◆ m_error_on_failure

bool lbann::callback::check_gradients::m_error_on_failure

private

Whether to throw an exception for large gradient errors.

Definition at line 94 of file check_gradients.hpp.

◆ m_modes

std::set<execution_mode> lbann::callback::check_gradients::m_modes

private

Execution modes with gradient checks.

Definition at line 88 of file check_gradients.hpp.

◆ m_step_size

EvalType lbann::callback::check_gradients::m_step_size

private

Step size for numerical differentiation.

Definition at line 90 of file check_gradients.hpp.

◆ m_verbose

bool lbann::callback::check_gradients::m_verbose

private

Whether to print results for each parameter.

Definition at line 92 of file check_gradients.hpp.

The documentation for this class was generated from the following file:

check_gradients.hpp

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ check_gradients()

Member Function Documentation

◆ copy()

◆ do_check_gradients()

◆ name()

◆ on_test_end()

◆ on_train_end()

◆ on_validation_end()

◆ serialize()

◆ write_specific_proto()

Member Data Documentation

◆ m_error_on_failure

◆ m_modes

◆ m_step_size

◆ m_verbose