Operators

An operator defines a mathematical function that that supports both forward and possibly backward operations. In the forward direction, it takes a vector of input tensors and produces a vector of output tensors. In the backward direction they implement the differentiation of the forward operation, applying the function to the operator’s forward inputs and gradient with respect to the outputs, to compute the gradient with respect to the input.

Operators only store immutable state; they do not have learnable parameters. An operator should also be able to take objective function gradients w.r.t. the outputs (“previous error signals”) and compute the objective function gradients w.r.t. the inputs (“error signals”). This allows the model to perform automatic differentiation.

Operators are specified for unique input and output data types.

Operator	Description
Operator	Base class for LBANN operators
AbsOperator	Apply the Abs operator entrywise
AcoshOperator	Apply the Acosh operator entrywise
AcosOperator	Apply the Acos operator entrywise
AddOperator	Apply the Add operator entrywise
AddConstantOperator	Add a constant to each input value (x+c)
AsinOperator	Apply the Asin operator entrywise
AsinhOperator	Apply the Asinh operator entrywise
AtanOperator	Apply the Atan operator entrywise
AtanhOperator	Apply the Atanh operator entrywise
BinaryCrossEntropyOperator	Apply the BinaryCrossEntropy operator entrywise
BooleanAccuracyOperator	Apply the BooleanAccuracy operator entrywise
BooleanFalseNegativeOperator	Apply the BooleanFalseNegative operator entrywise
BooleanFalsePositiveOperator	Apply the BooleanFalsePositive operator entrywise
CeilOperator	Apply the Ceil operator entrywise
ClampOperator	Constrain all values in a tensor within a range
ConstantSubtractOperator	Subtract each input value from a constant (c-x)
CosOperator	Apply the Cos operator entrywise
CoshOperator	Apply the Cosh operator entrywise
DivideOperator	Apply the Divide operator entrywise
EqualOperator	Apply the Equal operator entrywise
EqualConstantOperator	Test each value for equality with a constant (x==c)
ErfOperator	Apply the Erf operator entrywise
ErfInvOperator	Apply the ErfInv operator entrywise
ExpOperator	Apply the Exp operator entrywise
Expm1Operator	Apply the Expm1 operator entrywise
FloorOperator	Apply the Floor operator entrywise
GeluOperator	Gaussian Error Linear Unit operator
GreaterOperator	Apply the Greater operator entrywise
GreaterConstantOperator	Test each value for ‘greater-than’ with a constant (x>c)
GreaterEqualOperator	Apply the GreaterEqual operator entrywise
GreaterEqualConstantOperator	Test each value for ‘greater-than-or-equal-to’ with a constant (x>=c)
LessOperator	Apply the Less operator entrywise
LessConstantOperator	Test each value for ‘less-than’ with a constant (x<c)
LessEqualOperator	Apply the LessEqual operator entrywise
LessEqualConstantOperator	Test each value for ‘less-than-or-equal-to’ with a constant (x<=c)
LogOperator	Apply the Log operator entrywise
Log1pOperator	Apply the Log1p operator entrywise
LogSigmoidOperator	Apply the LogSigmoid operator entrywise
LogSoftmaxOperator	Apply the LogSoftmax operator entrywise
LogicalAndOperator	Apply the LogicalAnd operator entrywise
LogicalNotOperator	Apply the LogicalNot operator entrywise
LogicalOrOperator	Apply the LogicalOr operator entrywise
LogicalXorOperator	Apply the LogicalXor operator entrywise
MaxOperator	Apply the Max operator entrywise
MaxConstantOperator	Apply the MaxConstant operator entrywise
MinOperator	Apply the Min operator entrywise
MinConstantOperator	Apply the MinConstant operator entrywise
ModOperator	Apply the Mod operator entrywise
MultiplyOperator	Apply the Multiply operator entrywise
NegativeOperator	Apply the Log Negative entrywise
NotEqualOperator	Apply the NotEqual operator entrywise
NotEqualConstantOperator	Test each value for inequality with a constant (x!=c)
PowOperator	Apply the Pow operator entrywise
ReciprocalOperator	Apply the Reciprocal operator entrywise
RoundOperator	Apply the Round operator entrywise
RsqrtOperator	Apply the Rsqrt operator entrywise
SafeDivideOperator	Apply the SafeDivide operator entrywise
SafeReciprocalOperator	Apply the SafeReciprocal operator entrywise
ScaleOperator	Scale each input value by a constant value (c*x)
SelectOperator	Chooses one input or the other based on the value of a predicate (if a return b, else c)
SeluOperator	Apply the Selu operator entrywise
SigmoidOperator	Apply the Sigmoid operator entrywise
SigmoidBinaryCrossEntropyOperator	Apply the SigmoidBinaryCrossEntropy operator entrywise.
SignOperator	Apply the Sign operator entrywise
SinOperator	Apply the Sin operator entrywise
SinhOperator	Apply the Sinh operator entrywise
SoftplusOperator	Apply the Softplus operator entrywise
SoftsignOperator	Apply the Softsign operator entrywise
SqrtOperator	Apply the Sqrt operator entrywise
SquareOperator	Apply the Square operator entrywise
SquareDifferenceOperator	Apply the SquareDifference operator entrywise
SubtractOperator	Apply the Subtract operator entrywise
SubtractConstantOperator	Apply the SubtractConstant operator entrywise
TanOperator	Apply the Tan operator entrywise
TanhOperator	Apply the Tanh operator entrywise

Operator

Operator is the base class for LBANN operators

Arguments:

input_type

(lbann.DataType) The type expected as input

output_type

(lbann.DataType) The type expected as output

device

(lbann.device_allocation) The device allocation

Methods:

export_proto()

Get a protobuf representation of this object

do_export_proto()

Get a protobuf representation of this object

Must be implemented in derived classes

Back to Top

---

Abs

Perform entrywise absolute value on the input tensor.

\[\text{Abs}(x) = |x|\]

Back to Top

---

Acosh

Apply the inverse hyperbolic cosine entrywise.

Back to Top

---

Acos

Apply the inverse cosine function entrywise.

Back to Top

---

Add

Perform entrywise addition on two input tensors.

Back to Top

---

AddConstant

Add a constant to each input value.

\[\text{AddConstant}(x,c) = x + c\]

Arguments:

constant

(double) The constant to be added

Back to Top

---

Asin

Apply the inverse sine function entrywise.

Back to Top

---

Asinh

Apply the hyperbolic inverse sine function entrywise.

Back to Top

---

Atan

Apply the inverse tangent function entrywise.

Back to Top

---

Atanh

Apply the hyperbolic inverse tangent function entrywise.

Back to Top

---

BinaryCrossEntropy

Apply the BinaryCrossEntropy operator entrywise.

Compare each predicted probability to actual class value, either 0 or 1. Calculate the score that penalizes the probabilities based on the distance from the expected value.

Back to Top

---

BooleanAccuracy

Apply the BooleanAccuracy operator entrywise.

Applies the function:

\[\text{BooleanAccuracy}(x1,x2) = (x1 >= 0.5) == (x2 >= 0.5)\]

Back to Top

---

BooleanFalseNegative

Apply the BooleanFalseNegative operator entrywise.

Back to Top

---

BooleanFalsePositive

Apply the BooleanFalsePositive operator entrywise.

Back to Top

---

Ceil

Apply the ceiling function to an input tensor entrywise.

Back to Top

---

Clamp

Constrain all values in a tensor within a range

\[\begin{split}\text{Clamp}(x; \text{min}, \text{max}) = \begin{cases} \text{min} & x \leq \text{min} \\ x & \text{min} < x < \text{max} \\ \text{max} & x \geq \text{max} \end{cases}\end{split}\]

Arguments:

min

(double) Minimum value in range

max

(double) Maximum value in range

Back to Top

---

ConstantSubtract

Subtract each input value from a constant.

\[\text{ConstantSubtract}(c,x) = c - x\]

Arguments:

constant

(double) The constant to subtract from

Back to Top

---

Cos

Compute the cosine of the input tensor entrywise.

Back to Top

---

Cosh

Compute the hyperbolic cosine of the input tensor entrywise.

Back to Top

---

Divide

Perform entrywise division on two input tensors.

\[\text{Divide}(x,y) = \frac{x}{y}\]

Back to Top

---

Equal

Perform entrywise logical equal on two input tensors.

Back to Top

---

EqualConstant

Perform entrywise logical equal on input tensor and a constant.

\[\text{EqualConstant}(x,c) = x \equiv c\]

Arguments:

constant

(double) The constant used for comparison

Back to Top

---

Erf

Compute the error function of the inpute tensor entrywise.

Back to Top

---

ErfInv

Compute the inverse error function entrywise.

Back to Top

---

Exp

Calculate the exponential of the input tensor entrywise.

\[\text{Exp}(x) = e^x\]

Back to Top

---

Expm1

Calculate the exponential minus one of the input tensor entrywise.

\[\text{Expm1}(x) = e^x - 1\]

Back to Top

---

Floor

Apply the floor function to the input tensor entrywise.

Back to Top

---

Gelu (GELU tanh approximation)

The Gaussian Error Linear Unit (GELU) operator is defined by:

\[\text{GELU}(x) = x\Phi (x),\]

where \(\Phi\) is the Gaussian cumulative distribution function. The hyperbolic tangent-based approximation of the Gaussian Error Linear Unit (GELU) operator, found in the BERT and GPT transformer codebases, is implemented in LBANN and given by:

\[\text{GELU'}(x) = \frac{x}{2} \cdot (1 + \text{tanh}(\sqrt{2 / \pi} \cdot (x + 0.044715 x^3))).\]

For explanation on GELU, see:

Dan Hendrycks and Kevin Gimpel. “Gaussian Error Linear Units (GELUs).” arXiv preprint arXiv:1606.08415 (2016).

Back to Top

---

Greater

Perform entrywise logical ‘greater’ on two input tensors.

\[\text{Greater}(x,y) = x > y\]

Back to Top

---

GreaterConstant

Perform entrywise logical ‘greater-than’ on input tensor and a constant.

\[\text{GreaterConstant}(x,c) = x > c\]

Arguments:

constant

(double) The constant to be used for comparison

Back to Top

---

GreaterEqual

Perform entrywise logical ‘greater-or-equal’ on two input tensors.

\[\text{GreaterEqual}(x,y) = x \geq y\]

Back to Top

---

GreaterEqualConstant

Perform entrywise logical ‘greater-or-equal’ on input tensor and a constant.

\[\text{GreaterEqualConstant}(x,c) = x \geq c\]

Arguments:

constant

(double) The constant to be used for comparison

Back to Top

---

Less

Perform entrywise logical ‘less-than’ on two input tensors.

\[\text{Less}(x,y) = x < y\]

Back to Top

---

LessConstant

Perform entrywise logical ‘less-than’ on input tensor and a constant.

\[\text{LessConstant}(x,y) = x < c\]

Arguments:

constant

(double) The constant to be used for comparison

Back to Top

---

LessEqual

Perform entrywise logical ‘less-equal’ on two input tensors.

\[\text{LessEqual}(x,y) = x \leq y\]

Back to Top

---

LessEqualConstant

Perform entrywise logical ‘less-or-equal’ on input tensor and a constant.

\[\text{LessEqualConstant}(x,c) = x \leq c\]

Arguments:

constant

(double) The constant to be used for comparison

Back to Top

---

Log

Calculate the log of the input tensor entrywise.

Back to Top

---

Log1p

Calculate the log of one plus the input tensor entrywise.

\[\text{Log1p}(x) = \log{1 + x}\]

Back to Top

---

LogSigmoid

Calculate the log of the output from the sigmoid function entrywise.

\[\text{LogSigmoid}(x) = \log \frac{1}{1+e^{-x}}\]

Back to Top

---

LogSoftmax

Calculate the log of the softmax function entrywise.

\[\text{LogSoftmax}(x)_i = x_i - \log \sum_j e^{x_j}\]

Back to Top

---

LogicalAnd

Perform entrywise logical ‘and’ on two input tensors.

Back to Top

---

LogicalNot

Perform entrywise logical ‘not’ on two input tensors.

Back to Top

---

LogicalOr

Perform entrywise logical ‘or’ on two input tensors.

Back to Top

---

LogicalXor

Perform entrywise logical ‘xor’ on two input tensors.

Back to Top

---

Max

Perform entrywise max of input tensors.

Back to Top

---

MaxConstant

Perform entrywise max of input tensor against a constant.

Back to Top

---

Min

Perform entrywise min of input tensors.

Back to Top

---

MinConstant

Perform entrywise min of input tensor against a constant.

Back to Top

---

Mod

Perform entrywise modulus on two input tensors.

Back to Top

---

Multiply

Perform entrywise multiplication on input tensors.

Back to Top

---

Negative

Produce output tensor with flipped sign.

\[\text{Negative}(x) = -x\]

Back to Top

---

NotEqual

Perform entrywise logical ‘not-equal’ on two input tensors.

Back to Top

---

NotEqualConstant

Perform entrywise logical ‘not-equal’ on input tensor and a constant.

\[\text{NotEqualConstant}(x, c) = x \neq c\]

Arguments:

constant

(double) The constant to be used for comparison

Back to Top

---

Pow

Perform entrywise exponent using one input tensor as the base and a second input tensor as the exponent.

\[\text{Pow}(x,y) = x^y\]

Back to Top

---

Reciprocal

Perform entrywise reciprocal function on input tensor.

\[\text{Reciprocal}(x) = \frac{1}{x}\]

Back to Top

---

Round

Round input tensor values to the nearest integer entrywise.

Back to Top

---

Rsqrt

Compute reciprocal of square-root of values in the input tensor entrywise.

\[\text{Rsqrt}(x) = \frac{1}{\sqrt{x}}\]

Back to Top

---

SafeDivide

FIXME: Is this right?

Perform entrywise division on two input tensors. Return zero if the divisor is zero.

Back to Top

---

SafeReciprocal

FIXME: Is this right?

Perform entrywise reciprocal function on input tensor. Return zero if the input value is zero.

\[\text{SafeReciprocal}(x) = \frac{1}{x}\]

Back to Top

---

Scale

Scale each input value by a constant.

\[\text{Scale}(x,c) = c * x\]

Arguments:

constant

(double) The constant to scale by

Back to Top

---

Select

Chooses one input or the other based on the value of a predicate (if a return b, else c). The predicate is given as the first tensor, followed by the true and false results, respectively. To optimize the operator for comparison predicates, the value and epsilon arguments provide a value to compare with.

For further optimization, the true and false tensors can be replaced with a constant value via the arguments (see below). If one of those values (or both) are set, the respective tensor parameters are not necessary. For example, Select(condition, some_tensor, value=1, if_true=2) in the Python frontend will internally set constant_if_true to True and value_if_true to 2. The resulting expression would be (condition == 1) ? 2 : some_tensor for every input element.

In general, the following statement is executed:

\[\begin{split}\text{Select}(pred, trueval, falseval) = \begin{cases} trueval & | pred - \text{value} | < \text{epsilon} \\ falseval & otherwise \end{cases}\end{split}\]

Arguments:

value

(double) The value to compare the predicate with

epsilon

(double) Comparison threshold (default: 1e-5)

constant_if_true

(bool) If true, uses value_if_true as a constant true value

constant_if_false

(bool) If true, uses value_if_false as a constant false value

value_if_true

(double) If set, uses the given value instead of the second parameter

value_if_false

(double) If set, uses the given value instead of the third parameter

Back to Top

---

Selu

Apply scaled exponential linear unit function to input tensor entrywise.

Back to Top

---

Sigmoid

Apply the sigmoid function to the input tensor entrywise.

\[\text{Sigmoid}(x) = \frac{1}{1+e^{-x}}\]

Back to Top

---

SigmoidBinaryCrossEntropy

FIXME: Better description of this?

Apply the SigmoidBinaryCrossEntropy operator entrywise.

Back to Top

---

Sign

FIXME: Is this output right?

Compute the sign of the imput tensor entrywise. If input > 0, output 1. if input < 0, output -1. if input == 0, output 0.

Back to Top

---

Sin

Calculate entrywise sine of the input tensor.

Back to Top

---

Sinh

Calculate entrywise hyperbolic sine of the input tensor.

Back to Top

---

Softplus

Calculate the softplus of the input tensor entrywise.

\[\text{Softplus}(x) = \log{1 + e^x}\]

Back to Top

---

Softsign

Calculate the softsign of the input tensor entrywise.

\[\text{Softsign}(x) = \frac{x}{1+|x|}\]

Back to Top

---

Sqrt

Compute square root of input tensor values entrywise.

Back to Top

---

Square

Compute square of input tensor values entrywise.

Back to Top

---

SquareDifference

FIXME: Better description?

Apply the SquareDifference operator entrywise

Back to Top

---

Subtract

Perform entrywise subtraction on two input tensors.

\[\text{Subtract}(x,y) = x - y\]

Back to Top

---

SubtractConstant

Subtract a constant from from the input tensor entrywise.

\[\text{SubtractConstant}(x,c) = x - c\]

Arguments:

constant

(double) The constant to subtract

Back to Top

---

Tan

Apply the tangent function entrywise.

Back to Top

---

Tanh

Apply the hyperbolic tangent function entrywise.

Back to Top