Apply L2 regularization to a set of weights. More...

#include <l2.hpp>

Inheritance diagram for lbann::l2_weight_regularization:

Collaboration diagram for lbann::l2_weight_regularization:

Public Types
using	AccumulateDataType = DataType

using	OptimizerType = data_type_optimizer< DataType >

using	WeightsType = data_type_weights< DataType >

template<El::Device D>
using	DMatType = El::Matrix< AccumulateDataType, D >

using	CPUMatType = DMatType< El::Device::CPU >

Public Member Functions
	l2_weight_regularization (EvalType scale_factor=1)

l2_weight_regularization *	copy () const override

template<typename ArchiveT >
void	serialize (ArchiveT &ar)

std::string	name () const override

void	setup (model &m) override

void	start_evaluation () override

EvalType	finish_evaluation () override

void	differentiate () override

void	compute_weight_regularization () override

Public Member Functions inherited from lbann::objective_function_term
	objective_function_term (EvalType scale_factor=EvalType(1))

	objective_function_term (const objective_function_term &other)=default

objective_function_term &	operator= (const objective_function_term &other)=default

virtual	~objective_function_term ()=default

template<class Archive >
void	serialize (Archive &ar)

std::vector< ViewingLayerPtr >	get_layer_pointers () const

void	set_layer_pointers (std::vector< ViewingLayerPtr > layers)

std::vector< ViewingWeightsPtr >	get_weights_pointers () const

void	set_weights_pointers (std::vector< ViewingWeightsPtr > w)

Private Member Functions
void	write_specific_proto (lbann_data::ObjectiveFunction &proto) const final
	Add objective function data to prototext. More...

Static Private Member Functions
template<El::Device Device>
static void	accumulate_contribution (const DMatType< Device > &vals, DMatType< Device > &contribution)

Private Attributes
std::map< El::Device, CPUMatType >	m_contributions

Al::request	m_allreduce_req

Additional Inherited Members
Protected Member Functions inherited from lbann::objective_function_term
lbann_comm &	get_comm ()

Protected Attributes inherited from lbann::objective_function_term
EvalType	m_scale_factor

std::vector< ViewingLayerPtr >	m_layers

std::vector< ViewingWeightsPtr >	m_weights

Detailed Description

Apply L2 regularization to a set of weights.

Given a weights tensor $ w $ ,

$L2(w) = \frac{1}{2} \sum\limits_{i} w(i)^2$

Note the $ 1/2 $ scaling factor.

Definition at line 50 of file l2.hpp.

Member Typedef Documentation

◆ AccumulateDataType

using lbann::l2_weight_regularization::AccumulateDataType = DataType

Definition at line 53 of file l2.hpp.

◆ CPUMatType

using lbann::l2_weight_regularization::CPUMatType = DMatType<El::Device::CPU>

Definition at line 62 of file l2.hpp.

◆ DMatType

template<El::Device D>

using lbann::l2_weight_regularization::DMatType = El::Matrix<AccumulateDataType, D>

Definition at line 60 of file l2.hpp.

◆ OptimizerType

using lbann::l2_weight_regularization::OptimizerType = data_type_optimizer<DataType>

Definition at line 55 of file l2.hpp.

◆ WeightsType

using lbann::l2_weight_regularization::WeightsType = data_type_weights<DataType>

Definition at line 57 of file l2.hpp.

Constructor & Destructor Documentation

◆ l2_weight_regularization()

lbann::l2_weight_regularization::l2_weight_regularization ( EvalType scale_factor = 1 )

Parameters

scale_factor The objective function term is $\text{scale\_factor} \times \sum L2(w_i)$

Here is the caller graph for this function:

Member Function Documentation

◆ accumulate_contribution()

template<El::Device Device>

static void lbann::l2_weight_regularization::accumulate_contribution	(	const DMatType< Device > &	vals,
		DMatType< Device > &	contribution
	)

staticprivate

Add the sum of squares of vals to contribution.

Parameters

vals	The values to accumulate
contribution	$1 \times 1$ matrix. Used as an accumulation variable.

◆ compute_weight_regularization()

void lbann::l2_weight_regularization::compute_weight_regularization ( )

overridevirtual

Compute the gradient w.r.t. the weights.

$\nabla_w L2(w) = w$

Implements lbann::objective_function_term.

Here is the caller graph for this function:

◆ copy()

l2_weight_regularization* lbann::l2_weight_regularization::copy ( ) const

inlineoverridevirtual

Copy function.

Implements lbann::objective_function_term.

Definition at line 69 of file l2.hpp.

Here is the call graph for this function:

◆ differentiate()

void lbann::l2_weight_regularization::differentiate ( )

inlineoverridevirtual

Compute the gradient w.r.t. the activations.

L2 regularization is independent of forward prop output, so nothing needs to be done here.

Todo:: Come up with a better function name in the base class.

Implements lbann::objective_function_term.

Definition at line 90 of file l2.hpp.

Here is the call graph for this function:

◆ finish_evaluation()

EvalType lbann::l2_weight_regularization::finish_evaluation ( )

overridevirtual

Complete evaluation of the objective function term.

Implements lbann::objective_function_term.

Here is the caller graph for this function:

◆ name()

std::string lbann::l2_weight_regularization::name ( ) const

inlineoverridevirtual

Get the name of the objective function term.

Implements lbann::objective_function_term.

Definition at line 78 of file l2.hpp.

Here is the call graph for this function:

◆ serialize()

template<typename ArchiveT >

void lbann::l2_weight_regularization::serialize ( ArchiveT & ar )

Archive for checkpoint and restart

Here is the caller graph for this function:

◆ setup()

void lbann::l2_weight_regularization::setup ( model & m )

overridevirtual

Setup objective function term.

Reimplemented from lbann::objective_function_term.

Here is the caller graph for this function:

◆ start_evaluation()

void lbann::l2_weight_regularization::start_evaluation ( )

overridevirtual

Start evaluation of the objective function term. This should include the scaling factor. The result is not available until finish_evaluation has been called.

Implements lbann::objective_function_term.

Here is the caller graph for this function:

◆ write_specific_proto()

void lbann::l2_weight_regularization::write_specific_proto ( lbann_data::ObjectiveFunction & proto ) const

finalprivatevirtual

Add objective function data to prototext.

Implements lbann::objective_function_term.

Here is the caller graph for this function:

Member Data Documentation

◆ m_allreduce_req

Al::request lbann::l2_weight_regularization::m_allreduce_req

private

For non-blocking allreduces.

Definition at line 106 of file l2.hpp.

◆ m_contributions

std::map<El::Device, CPUMatType> lbann::l2_weight_regularization::m_contributions

private

Contributions to evaluated value.

Definition at line 103 of file l2.hpp.

The documentation for this class was generated from the following file:

l2.hpp

Public Types

Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Member Typedef Documentation

◆ AccumulateDataType

◆ CPUMatType

◆ DMatType

◆ OptimizerType

◆ WeightsType

Constructor & Destructor Documentation

◆ l2_weight_regularization()

Member Function Documentation

◆ accumulate_contribution()

◆ compute_weight_regularization()

◆ copy()

◆ differentiate()

◆ finish_evaluation()

◆ name()

◆ serialize()

◆ setup()

◆ start_evaluation()

◆ write_specific_proto()

Member Data Documentation

◆ m_allreduce_req

◆ m_contributions