Activations

Sigmoid

Sigmoid function is a function that has a characteristic S-shaped curve.

logistic function

$$\sigma(x) \doteq \frac{1}{1 + e^{-x}}$$

Properties

$$\begin{align*} \sigma(-x) &= \frac{1}{1 + e^{x}} = \frac{e^{-x}}{e^{-x} + 1} ~~(\because \text{multiplied with } e^{-x} / e^{-x}) \\ &= \frac{{\color{magenta}1} + e^{-x} {\color{magenta}- 1}}{1 + e^{-x}} = 1 - \frac{1}{e^{-x} + 1} \\ &= 1 - \sigma(x) \end{align*}$$

derivative:

$$\begin{align*} \frac{\partial}{\partial x} \sigma(x) &= - \frac{1}{(1 + e^{-x})^2} \cdot (-e^{-x}) \\ &= \frac{e^{-x}}{(1 + e^{-x})^2} = \frac{1 + e^{-x} - 1}{(1 + e^{-x})^2} \\ &= \frac{1}{1 + e^{-x}} - \frac{1}{(1 + e^{-x})^2} \\ &= \frac{1}{1 + e^{-x}} \left( 1 - \frac{1}{1 + e^{-x}} \right) \\ &= \sigma(x)(1 - \sigma(x)) \quad = \sigma(x)\sigma(-x) \end{align*}$$

Hyperbolic tangent

$$f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$