Activation Layers

Layer	Description
Elu	Exponential linear unit
Identity	Output the input tensor
LeakyRelu	Leaky relu
LogSoftmax	Logarithm of softmax function
Relu	Rectified linear unit
Softmax	Softmax

---

Elu

The Elu layer is similar to Relu but with negative values that cause the mean of the Elu activation function to shift toward 0.

\[\begin{split}\text{ELU}(x; \alpha) = \begin{cases} x & x > 0 \\ \alpha (e^x - 1) & x \leq 0 \end{cases}\end{split}\]

\(\alpha\) should be non-negative. See:

Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. “Fast and accurate deep network learning by exponential linear units (ELUs).” arXiv preprint arXiv:1511.07289 (2015).

Arguments:

alpha

(double, optional) Default: 1. Should be >=0

Back to Top

---

Identity

The Identity layer outputs the input tensor.

This layer is very cheap since it just involves setting up tensor views.

Arguments: None

Back to Top

---

LeakyRelu

LeakyRelu modifies the Relu function to allow for

a small, non-zero gradient when the unit is saturated and not active.

\[\begin{split}\text{LeakyReLU}(x; \alpha) = \begin{cases} x & x > 0 \\ \alpha x & x \leq 0 \end{cases}\end{split}\]

See:

Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. “Rectifier nonlinearities improve neural network acoustic models.” In Proc. ICML, vol. 30, no. 1, p. 3. 2013.

Arguments:

negative_slope

(double, optional) Default: 0.01

Back to Top

---

LogSoftmax

LogSoftmax is the logarithm of the softmax function.

\[\log \text{softmax}(x)_i = x_i - \log \sum_j e^{x_j}\]

Arguments: None

Back to Top

---

Relu

The Relu layer outputs input directly if positive, otherwise outputs zero.

\[\text{ReLU}(x) = \text{max}(x, 0)\]

Arguments: None

Back to Top

---

Softmax

The Softmax layer turns a vector of K real values into a vector of K real values that sum to 1.

\[\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}\]

Arguments:

softmax_mode

(string, optional) Options: instance (default), channel

Back to Top