lbann::kfac_block_bn< Device > Class Template Reference
#include <kfac_block_bn.hpp>
Inheritance diagram for lbann::kfac_block_bn< Device >:
Collaboration diagram for lbann::kfac_block_bn< Device >:
Public Member Functions | |
| kfac_block_bn (Layer *layer, kfac::KFACExecutionContext *context, size_t layer_id, size_t inverse_proc_rank, bool enable_copy_errors, bool enable_copy_activations, int input_size, int output_size) | |
| kfac_block_bn (const kfac_block_bn &)=default | |
| kfac_block_bn & | operator= (const kfac_block_bn &)=default |
| int | get_local_memory_consumption () override |
| Get local Memory Consumption. More... | |
| void | compute_local_kronecker_factors (lbann_comm *comm, bool print_matrix, bool print_matrix_summary) override |
| Compute Kronecker factors. More... | |
| const std::vector< El::AbstractMatrix< DataType > * > | get_local_kronecker_buffers () override |
| Get buffers of Kronecker factors for reduce-scatter. More... | |
| void | update_kronecker_average (lbann_comm *comm, DataType kronecker_decay, bool print_matrix, bool print_matrix_summary) override |
| Update the average Kronecker factors. More... | |
| void | update_kronecker_inverse (lbann_comm *comm, bool use_pi, DataType damping_act, DataType damping_err, DataType learning_rate_factor, bool use_eigen_decomposition, bool print_matrix, bool print_matrix_summary, bool print_time) override |
| Compute the inverse of the average Kronecker factors. More... | |
| void | compute_preconditioned_gradients (lbann_comm *comm, DataType learning_rate_factor, bool print_matrix, bool print_matrix_summary, bool print_time) override |
| Compute the inverse of the average Kronecker factors. More... | |
| void | start_communication_forward_end (lbann_comm *comm) override |
| void | end_communication_forward_end (lbann_comm *comm) override |
| void | start_communication_backward_end (lbann_comm *comm) override |
| void | end_communication_backward_end (lbann_comm *comm) override |
| const std::vector< El::AbstractMatrix< DataType > * > | get_preconditioned_grad_buffers () override |
| Get buffers of preconditioned parameter gradients. More... | |
| std::vector< std::tuple< std::string, size_t, size_t > > | get_internal_matrix_info () const override |
| Return the list of internal matrices' (name, height, width) for debugging. All internal matrices should be ready when this function is called. More... | |
| std::string | get_info () const override |
| Get block's information in one line. More... | |
| int | get_inverse_matrices (El::Matrix< DataType, Device > &output, int offset) override |
| Copy inverse matrices to output buffer. More... | |
| int | get_inverse_matrices_size (lbann_comm *comm) override |
| Get inverse matrices size (offset). More... | |
| std::vector< int > | get_inverse_matrices_size_vector (lbann_comm *comm) override |
| Get inverse matrices size vector. More... | |
| void | resize_inverse_matrices_size (El::Matrix< double, El::Device::CPU > &inverse_matrices_size, int block_number) override |
| Get inverse matrices size vector. More... | |
| int | set_inverse_matrices (El::Matrix< DataType, Device > &workspace, int offset, lbann_comm *comm) override |
| Copy inverse matrices from output buffer. More... | |
 Public Member Functions inherited from lbann::kfac_block< Device > | |
| kfac_block (Layer *layer, kfac::KFACExecutionContext *context, size_t layer_id, size_t inverse_proc_rank, bool enable_copy_errors, bool enable_copy_activations, int input_size, int output_size) | |
| virtual | ~kfac_block ()=default |
| virtual void | on_forward_prop_end (lbann_comm *comm) |
| virtual void | initialize_activations_and_errors (lbann_comm *comm, int num_local_activations, int num_local_errors, int num_weights) |
| Copies activations, errors, and weights from model class to private variables to be used in KFAC computation. More... | |
| void | set_current_batch_size (El::Int batch_size) |
| std::string | get_name () const |
| size_t | get_inverse_proc_rank () const |
| DataType * | get_local_activation_buffer (int index) |
| DataType * | get_local_error_buffer (int index) |
| DataType * | get_weight_buffer (int index) |
| DataType * | get_gradient_wrt_weight_buffer (int index) |
| El::Int | get_current_batch_size () |
| El::Int | get_input_size () |
| El::Int | get_output_size () |
Private Attributes | |
| bool | m_is_after_conv |
| Information to perform its computation. More... | |
| size_t | m_num_channels |
| size_t | m_spatial_prod |
| El::Matrix< DataType, Device > | m_fisher_buf |
| Lower triangle buffers of the Fisher block. More... | |
| El::Matrix< DataType, Device > | m_fisher_average |
| Exponential moving average of the Fisher matrix. More... | |
| El::Matrix< DataType, Device > | m_fisher_inverse |
| Inverse of the average Fisher matrix. More... | |
Additional Inherited Members | |
| Protected Member Functions inherited from lbann::kfac_block< Device > | |
| El::Matrix< DataType, Device > & | get_workspace_matrix (const std::string &key, size_t height, size_t width) |
| Gets the Kronecker factor matrix of a FC layer. The same key is tied with the same matrix instance. More... | |
| El::SyncInfo< Device > | get_sync_info () |
| Return the default sync info that may used in update functions. More... | |
| Protected Attributes inherited from lbann::kfac_block< Device > | |
| Layer * | m_layer |
| The target layer. More... | |
| const size_t | m_layer_id |
| The layer ID in the model. TODO: Remove this. More... | |
| const int | m_inverse_proc_rank |
| The process ID which perform inverse on Kronecker. More... | |
| std::vector< std::unique_ptr< AbsDistMat > > | m_parent_local_activations |
| distributed martices for activations and gradients. More... | |
| std::vector< std::unique_ptr< AbsDistMat > > | m_child_local_errors |
| std::vector< std::unique_ptr< AbsDistMat > > | m_weight_gradients |
| std::vector< std::unique_ptr< AbsDistMat > > | m_subset_matrix |
| std::vector< std::unique_ptr< AbsDistMat > > | m_errors_copy |
| std::vector< std::unique_ptr< AbsDistMat > > | m_activations_copy |
| std::vector< std::unique_ptr< AbsDistMat > > | m_weight_values |
| Translatebetweengrid funciton has a basic implementation for STAR,STAR distributed matrices. Therefore, using local matrices for weights. More... | |
| std::vector< kfac::ReqT > | m_requests_forward_end |
| std::vector< kfac::ReqT > | m_requests_backward_end |
| int | m_input_size |
| feature size and batch size (used in primary -> secondary grid communication) More... | |
| int | m_output_size |
| int | m_batch_size |
| bool | m_enable_copy_errors |
| Enable copying of errors to enhance async communication. More... | |
| bool | m_enable_copy_activations |
| Enable copying of activations to enhance async communication. More... | |
| bool | m_has_kronecker_inverse |
| Whether this block already has an inverse history. More... | |
Detailed Description
template<El::Device Device>
class lbann::kfac_block_bn< Device >
A BN building block for K-FAC.
Definition at line 56 of file kfac_block_bn.hpp.
Constructor & Destructor Documentation
◆ kfac_block_bn() [1/2]
template<El::Device Device>
|
inline |
Constructor.
Definition at line 61 of file kfac_block_bn.hpp.
Here is the call graph for this function:
◆ kfac_block_bn() [2/2]
template<El::Device Device>
|
default |
Member Function Documentation
◆ compute_local_kronecker_factors()
template<El::Device Device>
|
overridevirtual |
Compute Kronecker factors.
Reimplemented from lbann::kfac_block< Device >.
◆ compute_preconditioned_gradients()
template<El::Device Device>
|
overridevirtual |
Compute the inverse of the average Kronecker factors.
Reimplemented from lbann::kfac_block< Device >.
◆ end_communication_backward_end()
template<El::Device Device>
|
overridevirtual |
Implements lbann::kfac_block< Device >.
◆ end_communication_forward_end()
template<El::Device Device>
|
overridevirtual |
Implements lbann::kfac_block< Device >.
◆ get_info()
template<El::Device Device>
|
inlineoverridevirtual |
Get block's information in one line.
Reimplemented from lbann::kfac_block< Device >.
Definition at line 168 of file kfac_block_bn.hpp.
◆ get_internal_matrix_info()
template<El::Device Device>
|
overridevirtual |
Return the list of internal matrices' (name, height, width) for debugging. All internal matrices should be ready when this function is called.
Reimplemented from lbann::kfac_block< Device >.
◆ get_inverse_matrices()
template<El::Device Device>
|
overridevirtual |
Copy inverse matrices to output buffer.
Implements lbann::kfac_block< Device >.
◆ get_inverse_matrices_size()
template<El::Device Device>
|
overridevirtual |
Get inverse matrices size (offset).
Implements lbann::kfac_block< Device >.
◆ get_inverse_matrices_size_vector()
template<El::Device Device>
|
inlineoverridevirtual |
Get inverse matrices size vector.
Implements lbann::kfac_block< Device >.
Definition at line 183 of file kfac_block_bn.hpp.
◆ get_local_kronecker_buffers()
template<El::Device Device>
|
inlineoverridevirtual |
Get buffers of Kronecker factors for reduce-scatter.
Reimplemented from lbann::kfac_block< Device >.
Definition at line 130 of file kfac_block_bn.hpp.
Here is the call graph for this function:
◆ get_local_memory_consumption()
template<El::Device Device>
|
inlineoverridevirtual |
Get local Memory Consumption.
Implements lbann::kfac_block< Device >.
Definition at line 116 of file kfac_block_bn.hpp.
◆ get_preconditioned_grad_buffers()
template<El::Device Device>
|
overridevirtual |
Get buffers of preconditioned parameter gradients.
Reimplemented from lbann::kfac_block< Device >.
◆ operator=()
template<El::Device Device>
|
default |
◆ resize_inverse_matrices_size()
template<El::Device Device>
|
inlineoverridevirtual |
Get inverse matrices size vector.
Implements lbann::kfac_block< Device >.
Definition at line 189 of file kfac_block_bn.hpp.
◆ set_inverse_matrices()
template<El::Device Device>
|
overridevirtual |
Copy inverse matrices from output buffer.
Implements lbann::kfac_block< Device >.
◆ start_communication_backward_end()
template<El::Device Device>
|
overridevirtual |
Implements lbann::kfac_block< Device >.
◆ start_communication_forward_end()
template<El::Device Device>
|
overridevirtual |
Implements lbann::kfac_block< Device >.
◆ update_kronecker_average()
template<El::Device Device>
|
overridevirtual |
Update the average Kronecker factors.
Reimplemented from lbann::kfac_block< Device >.
◆ update_kronecker_inverse()
template<El::Device Device>
|
overridevirtual |
Compute the inverse of the average Kronecker factors.
Reimplemented from lbann::kfac_block< Device >.
Member Data Documentation
◆ m_fisher_average
template<El::Device Device>
|
private |
Exponential moving average of the Fisher matrix.
Definition at line 210 of file kfac_block_bn.hpp.
◆ m_fisher_buf
template<El::Device Device>
|
private |
Lower triangle buffers of the Fisher block.
Definition at line 207 of file kfac_block_bn.hpp.
◆ m_fisher_inverse
template<El::Device Device>
|
private |
Inverse of the average Fisher matrix.
Definition at line 213 of file kfac_block_bn.hpp.
◆ m_is_after_conv
template<El::Device Device>
|
private |
Information to perform its computation.
Definition at line 203 of file kfac_block_bn.hpp.
◆ m_num_channels
template<El::Device Device>
|
private |
Definition at line 204 of file kfac_block_bn.hpp.
◆ m_spatial_prod
template<El::Device Device>
|
private |
Definition at line 204 of file kfac_block_bn.hpp.
The documentation for this class was generated from the following file:
