Hypergradient Adam optimizer. More...

#include <hypergradient_adam.hpp>

Inheritance diagram for lbann::hypergradient_adam< TensorDataType >:

Collaboration diagram for lbann::hypergradient_adam< TensorDataType >:

Public Types
Public Types
using	AbsDistMatrixType = El::AbstractDistMatrix< TensorDataType >
	The tensor type expected in this object. More...

using	WeightsType = data_type_weights< TensorDataType >
	The concrete weights type used by this object. More...

using	OptimizerType = data_type_optimizer< TensorDataType >
	The base optimizer type for this class. More...

Public Member Functions
	hypergradient_adam (TensorDataType init_learning_rate=El::To< TensorDataType >(1e-3), TensorDataType hyper_learning_rate=El::To< TensorDataType >(1e-7), TensorDataType beta1=El::To< TensorDataType >(0.9), TensorDataType beta2=El::To< TensorDataType >(0.99), TensorDataType eps=El::To< TensorDataType >(1e-8))
	Construct a Hypergradient Adam optimizer object. More...

	hypergradient_adam (const hypergradient_adam &other)

hypergradient_adam &	operator= (const hypergradient_adam &other)

	~hypergradient_adam () override=default

template<class Archive >
void	serialize (Archive &ar)

std::string	get_type () const override
	Human-readable type name. More...

description	get_description () const override
	Human-readable description. More...

void	setup (WeightsType *w=nullptr) override

void	write_proto (lbann_data::Optimizer &opt) const final

Public Member Functions inherited from lbann::Cloneable< hypergradient_adam< TensorDataType >, data_type_optimizer< TensorDataType > >
std::unique_ptr< hypergradient_adam< TensorDataType > >	clone () const
	Return an exception-safe, memory-safe copy of this object. More...

Protected Member Functions
void	step_compute (AbsDistMatrixType &values, const AbsDistMatrixType &gradient) override
	Computation for an optimization step. More...

Private Types
using	BaseType = Cloneable< hypergradient_adam< TensorDataType >, data_type_optimizer< TensorDataType > >

Private Attributes
TensorDataType	m_hyper_learning_rate
	Hypergradient learning rate. More...

TensorDataType	m_beta1
	Update factor for first moment estimate. More...

TensorDataType	m_beta2
	Update factor for second moment estimate. More...

TensorDataType	m_eps
	Small factor to avoid division by zero. More...

TensorDataType	m_current_beta1
	beta1 ^ iteration. More...

TensorDataType	m_current_beta2
	beta2 ^ iteration. More...

std::unique_ptr< AbsDistMatrixType >	m_moment1
	First moment estimates. More...

std::unique_ptr< AbsDistMatrixType >	m_moment2
	Second moment estimates. More...

std::unique_ptr< AbsDistMatrixType >	m_old_gradient
	Gradient estimate from the prior step (for hypergradient). More...

Detailed Description

template<typename TensorDataType>
class lbann::hypergradient_adam< TensorDataType >

Hypergradient Adam optimizer.

Reference:

Baydin et al. "Online Learning Rate Adaptation with Hypergradient Descent", 2017.

Definition at line 45 of file hypergradient_adam.hpp.

Member Typedef Documentation

◆ AbsDistMatrixType

template<typename TensorDataType >

using lbann::hypergradient_adam< TensorDataType >::AbsDistMatrixType = El::AbstractDistMatrix<TensorDataType>

The tensor type expected in this object.

Definition at line 56 of file hypergradient_adam.hpp.

◆ BaseType

template<typename TensorDataType >

using lbann::hypergradient_adam< TensorDataType >::BaseType = Cloneable<hypergradient_adam<TensorDataType>, data_type_optimizer<TensorDataType> >

private

Definition at line 49 of file hypergradient_adam.hpp.

◆ OptimizerType

template<typename TensorDataType >

using lbann::hypergradient_adam< TensorDataType >::OptimizerType = data_type_optimizer<TensorDataType>

The base optimizer type for this class.

Definition at line 62 of file hypergradient_adam.hpp.

◆ WeightsType

template<typename TensorDataType >

using lbann::hypergradient_adam< TensorDataType >::WeightsType = data_type_weights<TensorDataType>

The concrete weights type used by this object.

Definition at line 59 of file hypergradient_adam.hpp.

Constructor & Destructor Documentation

◆ hypergradient_adam() [1/2]

template<typename TensorDataType >

lbann::hypergradient_adam< TensorDataType >::hypergradient_adam	(	TensorDataType	init_learning_rate = `El::To< TensorDataType >(1e-3)`,
		TensorDataType	hyper_learning_rate = `El::To< TensorDataType >(1e-7)`,
		TensorDataType	beta1 = `El::To< TensorDataType >(0.9)`,
		TensorDataType	beta2 = `El::To< TensorDataType >(0.99)`,
		TensorDataType	eps = `El::To< TensorDataType >(1e-8)`
	)

Construct a Hypergradient Adam optimizer object.

Parameters

init_learning_rate	Initial Adam learning rate (0.001 is reasonable).
hyper_learning_rate	Hypergradient learning rate.
beta1	Decay rate for the first moment moving average.
beta2	Decay rate for the second moment moving average.
eps	Small factor to avoid division by zero.

◆ hypergradient_adam() [2/2]

template<typename TensorDataType >

lbann::hypergradient_adam< TensorDataType >::hypergradient_adam ( const hypergradient_adam< TensorDataType > & other )

◆ ~hypergradient_adam()

template<typename TensorDataType >

lbann::hypergradient_adam< TensorDataType >::~hypergradient_adam ( )

overridedefault

Member Function Documentation

◆ get_description()

template<typename TensorDataType >

description lbann::hypergradient_adam< TensorDataType >::get_description ( ) const

override

Human-readable description.

Here is the caller graph for this function:

◆ get_type()

template<typename TensorDataType >

std::string lbann::hypergradient_adam< TensorDataType >::get_type ( ) const

inlineoverride

Human-readable type name.

Definition at line 94 of file hypergradient_adam.hpp.

Here is the call graph for this function:

◆ operator=()

template<typename TensorDataType >

hypergradient_adam& lbann::hypergradient_adam< TensorDataType >::operator= ( const hypergradient_adam< TensorDataType > & other )

◆ serialize()

template<typename TensorDataType >

template<class Archive >

void lbann::hypergradient_adam< TensorDataType >::serialize ( Archive & ar )

Archive for checkpoint and restart

Definition at line 37 of file hypergradient_adam_impl.hpp.

◆ setup()

template<typename TensorDataType >

void lbann::hypergradient_adam< TensorDataType >::setup ( WeightsType * w = nullptr )

override

Here is the caller graph for this function:

◆ step_compute()

template<typename TensorDataType >

void lbann::hypergradient_adam< TensorDataType >::step_compute	(	AbsDistMatrixType &	values,
		const AbsDistMatrixType &	gradient
	)

overrideprotected

Computation for an optimization step.

Here is the caller graph for this function:

◆ write_proto()

template<typename TensorDataType >

void lbann::hypergradient_adam< TensorDataType >::write_proto ( lbann_data::Optimizer & opt ) const

final

Add optimizer data to prototext

Here is the caller graph for this function:

Member Data Documentation

◆ m_beta1

template<typename TensorDataType >

TensorDataType lbann::hypergradient_adam< TensorDataType >::m_beta1

private

Update factor for first moment estimate.

Definition at line 113 of file hypergradient_adam.hpp.

◆ m_beta2

template<typename TensorDataType >

TensorDataType lbann::hypergradient_adam< TensorDataType >::m_beta2

private

Update factor for second moment estimate.

Definition at line 115 of file hypergradient_adam.hpp.

◆ m_current_beta1

template<typename TensorDataType >

TensorDataType lbann::hypergradient_adam< TensorDataType >::m_current_beta1

private

beta1 ^ iteration.

Definition at line 119 of file hypergradient_adam.hpp.

◆ m_current_beta2

template<typename TensorDataType >

TensorDataType lbann::hypergradient_adam< TensorDataType >::m_current_beta2

private

beta2 ^ iteration.

Definition at line 121 of file hypergradient_adam.hpp.

◆ m_eps

template<typename TensorDataType >

TensorDataType lbann::hypergradient_adam< TensorDataType >::m_eps

private

Small factor to avoid division by zero.

Definition at line 117 of file hypergradient_adam.hpp.

◆ m_hyper_learning_rate

template<typename TensorDataType >

TensorDataType lbann::hypergradient_adam< TensorDataType >::m_hyper_learning_rate

private

Hypergradient learning rate.

Definition at line 111 of file hypergradient_adam.hpp.

◆ m_moment1

template<typename TensorDataType >

std::unique_ptr<AbsDistMatrixType> lbann::hypergradient_adam< TensorDataType >::m_moment1

private

First moment estimates.

Definition at line 123 of file hypergradient_adam.hpp.

◆ m_moment2

template<typename TensorDataType >

std::unique_ptr<AbsDistMatrixType> lbann::hypergradient_adam< TensorDataType >::m_moment2

private

Second moment estimates.

Definition at line 125 of file hypergradient_adam.hpp.

◆ m_old_gradient

template<typename TensorDataType >

std::unique_ptr<AbsDistMatrixType> lbann::hypergradient_adam< TensorDataType >::m_old_gradient

private

Gradient estimate from the prior step (for hypergradient).

Definition at line 127 of file hypergradient_adam.hpp.

The documentation for this class was generated from the following files:

Public Types

Public Member Functions

Protected Member Functions

Private Types

Private Attributes

Detailed Description

template<typename TensorDataType> class lbann::hypergradient_adam< TensorDataType >

Member Typedef Documentation

◆ AbsDistMatrixType

◆ BaseType

◆ OptimizerType

◆ WeightsType

Constructor & Destructor Documentation

◆ hypergradient_adam() [1/2]

◆ hypergradient_adam() [2/2]

◆ ~hypergradient_adam()

Member Function Documentation

◆ get_description()

◆ get_type()

◆ operator=()

◆ serialize()

◆ setup()

◆ step_compute()

◆ write_proto()

Member Data Documentation

◆ m_beta1

◆ m_beta2

◆ m_current_beta1

◆ m_current_beta2

◆ m_eps

◆ m_hyper_learning_rate

◆ m_moment1

◆ m_moment2

◆ m_old_gradient

template<typename TensorDataType>
class lbann::hypergradient_adam< TensorDataType >