The original LTFB algorithm. More...

#include <random_pairwise_exchange.hpp>

Inheritance diagram for lbann::ltfb::RandomPairwiseExchange:

Collaboration diagram for lbann::ltfb::RandomPairwiseExchange:

Classes
class	ExchangeStrategy
	A method for exchanging models with a partner trainer. More...

Public Types
enum	metric_strategy { metric_strategy::LOWER_IS_BETTER, metric_strategy::HIGHER_IS_BETTER }

Public Member Functions
void	select_next (model &m, ltfb::LTFBExecutionContext &ctxt, data_coordinator &dc) const final
	Engage in a tournament with a partner trainer. More...

Life-cycle management
	RandomPairwiseExchange (std::string metric_name, metric_strategy winner_strategy, std::unique_ptr< ExchangeStrategy > comm_algo, std::unique_ptr< MutationStrategy > mutate_algo)
	Constructor. More...

	RandomPairwiseExchange (std::unordered_map< std::string, metric_strategy > metrics, std::unique_ptr< ExchangeStrategy > comm_algo, std::unique_ptr< MutationStrategy > mutate_algo)
	Constructor. More...

	~RandomPairwiseExchange ()=default

	RandomPairwiseExchange (RandomPairwiseExchange const &other)

Public Member Functions inherited from lbann::Cloneable< RandomPairwiseExchange, MetaLearningStrategy >
std::unique_ptr< RandomPairwiseExchange >	clone () const
	Return an exception-safe, memory-safe copy of this object. More...

Private Member Functions
std::unordered_map< std::string, EvalType >	evaluate_model (model &m, LTFBExecutionContext &ctxt, data_coordinator &dc) const
	Get the value of the given metric from the model. More...

int	get_partner_trainer (lbann_comm const &c) const noexcept
	Generate a new trainer partner from the comm. More...

bool	local_is_better (std::unordered_map< std::string, EvalType > const &local_scores, std::unordered_map< std::string, EvalType > const &partner_scores) const
	Evaluate the output of two models according to the input metric strategies. More...

Private Attributes
std::unordered_map< std::string, metric_strategy >	m_metrics
	The list of metric/strategy pairs. More...

std::unique_ptr< ExchangeStrategy >	m_comm_algo
	The strategy for exchanging two models. More...

std::unique_ptr< MutationStrategy >	m_mutate_algo
	The strategy for mutation of a model. More...

Detailed Description

The original LTFB algorithm.

In this case, a collection of metrics are provided (by name) and the winner of the tournament is chosen based on the metric strategy associated with each of them. The assumption is that the metric outputs a single scalar and the winner is chosen by a simple ">" or "<" operation.

The tournament partners are chosen according to an internal algorithm. Partners exchange their models with each other, according to the selected communication scheme, and the winner at each rank is chosen based on evaluating the metrics on the local trainer's data (one consequence being that each partner might select a different winner). The partner model must win ALL of the metrics to be declared the tournament winner.

Definition at line 60 of file random_pairwise_exchange.hpp.

Member Enumeration Documentation

◆ metric_strategy

enum lbann::ltfb::RandomPairwiseExchange::metric_strategy

strong

Enumerator
LOWER_IS_BETTER
HIGHER_IS_BETTER

Definition at line 111 of file random_pairwise_exchange.hpp.

Constructor & Destructor Documentation

◆ RandomPairwiseExchange() [1/3]

lbann::ltfb::RandomPairwiseExchange::RandomPairwiseExchange	(	std::string	metric_name,
		metric_strategy	winner_strategy,
		std::unique_ptr< ExchangeStrategy >	comm_algo,
		std::unique_ptr< MutationStrategy >	mutate_algo
	)

Constructor.

Parameters

[in]	metric_name	The name of the metric to use for evaluation. A metric with this name must exist in the model passed to apply().
[in]	winner_strategy	Strategy for determining the winner of a tournament.
[in]	comm_algo	Algorithm for exchanging models.
[in]	mutate_algo	Algorithm for mutating models.

◆ RandomPairwiseExchange() [2/3]

lbann::ltfb::RandomPairwiseExchange::RandomPairwiseExchange	(	std::unordered_map< std::string, metric_strategy >	metrics,
		std::unique_ptr< ExchangeStrategy >	comm_algo,
		std::unique_ptr< MutationStrategy >	mutate_algo
	)

Constructor.

Parameters

[in]	metrics	The list of metric/strategy pairs. A metric with each given name must exist in the model passed to apply(). The local model is favored. The partner model must win ALL of the metric comparisons to be declared the winner.
[in]	comm_algo	Algorithm for exchanging models.
[in]	mutate_algo	Algorithm for mutating models.

◆ ~RandomPairwiseExchange()

lbann::ltfb::RandomPairwiseExchange::~RandomPairwiseExchange ( )

default

◆ RandomPairwiseExchange() [3/3]

lbann::ltfb::RandomPairwiseExchange::RandomPairwiseExchange ( RandomPairwiseExchange const & other )

Member Function Documentation

◆ evaluate_model()

std::unordered_map<std::string, EvalType> lbann::ltfb::RandomPairwiseExchange::evaluate_model	(	model &	m,
		LTFBExecutionContext &	ctxt,
		data_coordinator &	dc
	)		const

private

Get the value of the given metric from the model.

◆ get_partner_trainer()

int lbann::ltfb::RandomPairwiseExchange::get_partner_trainer ( lbann_comm const & c ) const

privatenoexcept

Generate a new trainer partner from the comm.

◆ local_is_better()

bool lbann::ltfb::RandomPairwiseExchange::local_is_better	(	std::unordered_map< std::string, EvalType > const &	local_scores,
		std::unordered_map< std::string, EvalType > const &	partner_scores
	)		const

private

Evaluate the output of two models according to the input metric strategies.

The local model is preferred if possible (to avoid a model move). That is, "<=" or ">=" is used.

Parameters

[in]	local_scores	The metric outputs of the local model.
[in]	partner_scores	The metric output of the remote model.

Returns: true if the local model is favored.

◆ select_next()

void lbann::ltfb::RandomPairwiseExchange::select_next	(	model &	m,
		ltfb::LTFBExecutionContext &	ctxt,
		data_coordinator &	dc
	)		const

final

Engage in a tournament with a partner trainer.

Parameters

[in,out]	m	On input, the locally computed model. On output, the winning model with respect to the tournament.
[in,out]	ctxt	The execution context for the outer LTFB wrapper.
[in,out]	dc	The data source for the tournament.

Member Data Documentation

◆ m_comm_algo

std::unique_ptr<ExchangeStrategy> lbann::ltfb::RandomPairwiseExchange::m_comm_algo

private

The strategy for exchanging two models.

This is largely an implementation detail of moving models around. It shouldn't be used to alter the algorithmic goings-on of the method. Ideally, it should disappear as we converge on a "universally best method" for moving things around (let's pretend such a thing exists; it makes me feel better, anyway).

Definition at line 203 of file random_pairwise_exchange.hpp.

◆ m_metrics

std::unordered_map<std::string, metric_strategy> lbann::ltfb::RandomPairwiseExchange::m_metrics

private

The list of metric/strategy pairs.

Each metric gets its own strategy. A partner model must win every metric to be declared the tournament winner.

Definition at line 193 of file random_pairwise_exchange.hpp.

◆ m_mutate_algo

std::unique_ptr<MutationStrategy> lbann::ltfb::RandomPairwiseExchange::m_mutate_algo

private

The strategy for mutation of a model.

When a trainer loses in a LTFB tournament, the winning model is copied over to it and this mutation strategy is applied to the copied model to explore a new model. This is relevant to neural architecture search (NAS).

Definition at line 212 of file random_pairwise_exchange.hpp.

The documentation for this class was generated from the following file:

random_pairwise_exchange.hpp

Classes

Public Types

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Member Enumeration Documentation

◆ metric_strategy

Constructor & Destructor Documentation

◆ RandomPairwiseExchange() [1/3]

◆ RandomPairwiseExchange() [2/3]

◆ ~RandomPairwiseExchange()

◆ RandomPairwiseExchange() [3/3]

Member Function Documentation

◆ evaluate_model()

◆ get_partner_trainer()

◆ local_is_better()

◆ select_next()

Member Data Documentation

◆ m_comm_algo

◆ m_metrics

◆ m_mutate_algo