Neural networks
Data
We generally consider data points \(\mathbf x_i = [x_{i,1},\dots,x_{i,j},\dots,x_{i,p}] \in \mathbb R^p\) living in a \(p\)-dimensional space. Each dimension of this \(p\) dimensional space is generally called a variable or a feature or a covariate.
We generally have a collection of \(n\) data points \(\mathbf x_i\) with \(i = 1,\dots,n\), that are called observations or individuals and constitute a sample.
In supervised setting, we also have, for each data point \(\mathbf x_i\), a quantitative or qualitative label, also called a response, which can be univariate \(y_i \in \mathbb R\) or multivariate \(\mathbf y_{i} = [y_{i,1}, \dots, y_{i,q}] \in \mathbb R^q\).
All observations can be gathered in the following matrices:
- covariate matrix \(\mathbf X = [x_{i,j}]_{i=1:n}^{j=1:p}\) of dimension \(n \times p\) with observations in rows and variables in columns;
- response matrix \(\mathbf Y = [y_{i,\ell}]_{i=1:n}^{\ell=1:q}\) of dimension \(n \times q\) with observations in rows and response variables in columns.
Dense neural networks (DNN)
Neural networks are generally represented as in Figure 1.
- This kind of diagram is read from left to right.
- Each node, called an artificial neuron, represents the value of a given feature for a given data point \(i\).
- Nodes are organized in layers.
- The input layer nodes contain the value of each feature \(x_{i,j}\) for data point \(i\).
- Hidden layer nodes or intermediate nodes encode features defining data embedding in a different space (with lower or higher dimensions depending on the number of neurons in the layer).
- An edge connect two nodes if the value stored in the left node is used to compute the value in the right node.
- In supervised setting, the output layer contains predicted values for the response variables \([\hat{y}_{i,1}, \dots, \hat{y}_{i,q}]\).
In dense neural network, we only have fully connected layers, meaning that all nodes (i.e. neurons) from one layer are used to compute the next layer:
For instance, all covariate values in the data point vector \(\mathbf x_i = [x_{i,1},\dots,x_{i,j},\dots,x_{i,p}]\) are used to compute values stored in each node of the first layer, i.e. \(\mathbf a_i^{(1)} = [a_{i,1}^{(1)}, a_{i,2}^{(1)}, \dots, a_{i,h_1}^{(1)}]\).
Similarly, all values stored in each node of the first layer, i.e. \(\mathbf a_i^{(1)} = [a_{i,1}^{(1)}, a_{i,2}^{(1)}, \dots, a_{i,h_1}^{(1)}]\), are used to compute values stored in each node of the second layer, \(\mathbf a_i^{(2)} = [a_{i,1}^{(2)}, a_{i,2}^{(2)}, \dots, a_{i,h_1}^{(2)}]\), etc.
Figure 2 details how each hidden layer node value is computed for each data point \(\mathbf x_i = [x_{i,1},\dots,x_{i,j},\dots,x_{i,p}]\), using a succession of linear and non-linear operations. From the neural network representation in Figure 1, we can derive the matrix representation that is actually used for computations:
\[ \begin{array}{ccl} \mathbf{a}_i^{(1)} & = & \sigma\big( \mathbf{W}^{(1)} \mathbf{x}_i + \mathbf{b}^{(1)} \big)\\ \mathbf{a}_i^{(2)} & = & \sigma\Big( \mathbf{W}^{(2)}\sigma\Big( \mathbf{W}^{(1)} \mathbf{a}_i^{(0)} + \mathbf{b}^{(1)} \Big) + \mathbf{b}^{(2)} \Big)\\ \vdots\\ \hat{\mathbf{y}}_i & = & \sigma\Bigg(\dots \sigma\Big( \mathbf{W}^{(2)}\sigma\big( \mathbf{W}^{(1)} \mathbf{a}_i^{(0)} + \mathbf{b}^{(1)} \big) + \mathbf{b}^{(2)} \Big) \Bigg) \end{array} \] where \(\sigma(.)\) is an element-wise non-linear function (e.g. that is applied to each element of the input vector), also called the activation function.
The weight matrices \(\mathbf W^{(1)}, \mathbf W^{(2)}, \dots\) are also called filters.
An artificial neural network is “just” a sequence of linear transformations followed by element-wise non linear transformations.
Convulotional neural networks (CNN)
The main difference between dense neural network and convolutional neural network is that, in the latter, the values stored in neurons within hidden layers are not computed using all feature values from the previous layer.
In general, a convolution layer is similar to a fully connected layer, with weighted sums of features (matrix product) added by a bias term and passed to an activation function, but the weighted sums are made (for each neuron) using only a small part of the neuron values from the previous layer.
Generally, input data features/covariates or previous layer features are organised in a 1 dimensional (sequence) or 2 dimensional (image) array, and only neighbor features (i.e. neighbor pixels in an image) are used to compute each convolution.
Figure 3 represent a typical convolutional neural network with several convolution layers followed by some fully connecter layers.
Auto-encoders
