Introduction

In this course, we'll discuss several types of neural networks. We'll begin this journey by examining the simplest form of neural network: the multilayer perceptron (MLP), which is a subtype of feedforward neural networks.

Objectives

• describe the components of a simple feedforward neural network
• convert simple functions to computation graphs
• describe how a neural network learns

Getting started

• Before reading further, watch the following video series from Grant Sanderson (3blue1brown)

• If you're rusty on matrix multiplication, review one of the following resources:

• https://en.wikipedia.org/wiki/Matrix_multiplication

• https://www.khanacademy.org/math/precalculus/x9e81a4f98389efdf:matrices/x9e81a4f98389efdf:multiplying-matrices-by-matrices/v/matrix-multiplication-intro

Background resources

The resources linked below provide helpful background on the topics covered in this lesson:

• OPTIONAL: Sections 1-3 of Yoav Goldberg's Primer on Neural Network Models for NLP
• Section 4 of Yoav Goldberg's Primer on Neural Network Models for NLP
• 7.0 - 7.4 of Jurafsky and Martin's Speech and Language Processing

Moving from one neuron to a neural network

In LING 539, we had an introduction to one possible building block of neural networks, logistic regression, which we can think of as a type of artificial neuron with a particular activation function, the sigmoid. Imagine each circle in the following diagram as a logistic regression:

This is one possible design a mulilayer perceptron (MLP) can take. The MLP depicted above has two hidden layers (layers other than the network's input and output). All neural networks are composed of interconnected neurons, though the particular configuration of these connections and the internal structure of each neuron may differ. All MLPs have some fixed-size input layer and one or more hidden layers each comprised of one or more neurons[^neuronaka]. The layers are arranged sequentially ($\text{input} \rightarrow h_{l1} \rightarrow \dots \rightarrow h_{ln} \rightarrow \text{output}$). As the name suggests, the input layer is the first layer in the network. It provides the initial numerical representation of each datapoint.

Properties universal to MLPs

• at least one hidden layer
• fully connected
• every neuron in one layer of the network is connected to every neuron in the next layer. The output of one neuron becomes part of the input to all neurons in the next layer of the network (each of these inputs takes some learnable weight).

Networks with at least two hidden layers (such as the one shown above) are the shallowest form of deep neural networks.

Defining multilayer perceptron

There is some ambiguity about the definition of a multilayer perceptron (MLP).¹ We will take it to mean a fully connected feedforward neural network with neurons exhibiting the following properties:

1. each neuron in the network takes as input a) activations from the immediately previous layer, b) weights, and c) a bias term (might be set to zero).
2. A linear transformation² of the form $\mathbf{Wx} + \mathbf{b}$ (a matrix multiplication and vector addition) is applied to the input.
3. The result is then fed into some activation function $\phi$ that is typically non-linear.

Activation functions

So far we've only discussed the sigmoid, but that's certainly not the only kind of nonlinear activation function. In fact, using the sigmoid can lead to very flat gradients (that's a big problem which we'll cover later).

In later lessons, we'll look at why we might want to use one activation fuction over another. If you're curious about other options, you can interactively compare some commonly used activation functions using Justin Emery's tool:

• https://uazhlt.github.io/activation-functions-visualized

Classifying with MLPs

Recall that logistic regression is a linear model.

Why is logistic regression linear?

Logistic regression includes no interaction between the feature values (ex. $x_{1} * x_{2}$). Of course, one can alter the input ($x$) to include such information (ex. $\left[ x \right] \rightarrow \left[ x, x^{2}, x^{3} \right]$), but $y = \mathbf{w}\mathbf{x}+b$ as a model encodes no such trick. No matter the value of $\mathbf{w}$ or $b$ selected, you're going to end up with a $n-1$ dimensional hyperplane.

Like logistic regression, an MLP can be used to classify linearly separable data by iteratively updating its parameters to minimize prediction errors:

Beyond linear models

Deep MLPs, however, can learn to classify data that is not linearly separable.

More impressive still, under the right conditions, a neural network is a universal approximator. In other words, given enough data (and sufficiently expressive input), we can use a certain class of neural networks to approximate any function to an arbitrary level of precision [2015].

We've looked at simple neural networks at a high level, but how are these implemented in practice? In this course, we won't reinvent the wheel by implementing neural networks from scratch (though this can be an enlightening exercise!). Instead, we'll design and train our networks using a popular, powerful, and efficient library. Let's look at a couple of key components and features of frameworks like PyTorch.

Learning our parameters

Given some objective, one can imagine many ways you might optimize the parameters of neural network. For example, you might repeatedly try random sets of parameters. Alternatively, you could alter a single parameter in the network slightly and observe its effect on the loss. As you might guess, the problem with such methods is that they are extremely inefficient. These approaches quickly become infeasible as you increase the number of parameters. Deep MLPs can have many parameters (each neuron has $\vert h_{l-1}\vert + 1$ weights leading into it), but commonly used models in NLP today have millions (sometime billions and even a trillion) parameters!

Fortunately, there is an efficient method to updating these parameters (and we've already seen a form of it). Just like we saw with logistic regression³, we'll measure our error, compute our derivatives, and use gradient descent to actually update our parameters.

Computation graphs

Frameworks like PyTorch can record the operations we perform as a computation graph. The graph shows how intermediate values depend on earlier values. It also provides a way to work backward from the loss and compute how sensitive that loss is to each of our learnable parameters.

Architecture

Let's return to logistic regression with a single input feature:

$$a = \hat{y} = \sigma(wx + b)$$

Here, $x$ is our input, $w$ is a weight, and $b$ is a bias. All three are scalars. The sigmoid gives us a prediction $a$ between 0 and 1. Our model learns $w$ and $b$; for a given example, we keep $x$ fixed.

At the level of the architecture, we have one input feeding into one neuron:

The computation graph opens up that neuron into its individual operations. We can express those operations in a few lines of Python:

import torch

u = w * x
z = u + b
a = torch.sigmoid(z)
loss = loss_fn(a, y)

The names in this code match the names in the graph below. $y$ is the target label, and loss_fn compares our prediction with that target. For this example, we'll use binary cross entropy, as in logistic regression.

Forward pass

In the forward pass, we compute our output (a prediction) given some input. When training, we use that output (prediction) to compute our loss^†.

Suppose $x = 2$, $w = 0.5$, and $b = -1$. First, we multiply the input by the weight to get $u = 1$. Adding the bias gives $z = 0$, and applying the sigmoid gives $a = 0.5$.

The visual below starts with those values and a target of $y = 1$. Change the weight or bias and watch the intermediate values, prediction, and loss. Which values change when you adjust the bias? Which stay the same?

Logistic regression visual

Each operation uses values computed earlier. For example, sigmoid takes $z$ as its input; it does not need to know whether $z$ came from an addition, a multiplication, or even a more complicated network.

Here's the same dependency structure as a static graph:

The circular nodes supply the input ($x$), parameters ($w$ and $b$), and target ($y$). The rectangular nodes perform operations and pass their results along the arrows. Following the arrows lets us answer questions about dependencies. Changing $b$ can change $z$, $a$, and the loss, but it cannot change $u$.

Backward pass

Once we've computed the loss, we can work backward through the graph to compute gradients. A gradient tells us how sensitive the loss is to a small change in a value. For a single scalar parameter, think of it as the local rate of change of the loss per unit change in that parameter.

Select Backward in the visual above. It shows the gradient of the loss with respect to each intermediate value and, below the graph, the gradients for $w$ and $b$. The red upward arrows indicate the direction of the backward computation. Values computed during the forward pass are still visible.

At the starting settings, the weight gradient is $-1$ and the bias gradient is $-0.5$. Their negative signs tell us that a small increase in either parameter would reduce the loss here.

This process of computing gradients by working backward is called backpropagation. The graph below highlights the path from the loss to our parameters:

In this second diagram, the red edge labels identify local derivatives: how one operation's output changes with one of its inputs. These differ from the gradients displayed in the interactive visual, which describe how the final loss changes. We combine the local derivatives along a path to obtain a gradient of the loss.

The idea is basically this:

1. Loop over the nodes in reverse order starting from the last node in the graph

2. Use each operation's derivative rule to pass the gradient back to its inputs.

3. Combine these contributions using the chain rule until we reach the parameters whose gradients we need.

Let's look at what partial derivative each red edge in our example backward graph represents:

$$d_{0} = \frac{\partial{L}}{\partial{a}} \\[1em] d_{1} = \frac{\partial{a}}{\partial{z}} \\[1em] d_{2} = \frac{\partial{z}}{\partial{b}} \\[1em] d_{3} = \frac{\partial{z}}{\partial{u}} \\[1em] d_{4} = \frac{\partial{u}}{\partial{w}} \\[1em]$$

You can read $\frac{\partial{z}}{\partial{b}}$ as the derivative of $z$ wrt⁴ $b$. In other words, at what rate does $z$ change as $b$ changes?

Looking at our computation graph, we have a composition of functions. In order to figure out how to update our parameters, we'll compute partial derivatives and use the chain rule from calculus:

For example, $w$ affects $z$ through $u$. To compute the derivative of $z$ with respect to $w$, we combine the rates of change along that path:

$$\frac{\partial{z}}{\partial{w}} = \frac{\partial{z}}{\partial{u}}\frac{\partial{u}}{\partial{w}}$$

If you're beginning to freak out from the sight of derivatives, I've got good news: PyTorch supports automatic differentiation, or autodiff, to handle this for us. The key idea is to follow the dependencies: a change in $w$ changes $u$, which changes $z$, which can change the prediction and loss.

From gradients to learning

Computing gradients and updating parameters are separate steps. In PyTorch, loss.backward() computes gradients and accumulates them^† in the parameters' .grad attributes. An optimizer then uses those gradients to update the parameters.

For basic gradient descent, we subtract the gradient multiplied by a learning rate^†.

The simple curve below illustrates this with a different loss, $L(w) = (w - 2)^2$. Its minimum is at $w = 2$. We start at $w = 0.5$, where the gradient is $-3$. Compare learning rates of $0.1$, $0.75$, and $1.1$.

Learning rate and overshooting

Now return to our logistic regression example. The code below supplies the gradients shown in our computation graph visual at its starting settings. Before running it, predict the updated values of $w$ and $b$. Do you expect the loss to increase or decrease?

from math import exp, log

x = 2.0
y = 1.0
w = 0.5
b = -1.0
learning_rate = 0.1

# Gradients computed at these starting settings.
w_gradient = -1.0
b_gradient = -0.5

def loss_for_example(w, b):
    z = w * x + b
    a = 1 / (1 + exp(-z))  # sigmoid
    return -y * log(a) - (1 - y) * log(1 - a)

loss_before = loss_for_example(w, b)
w = w - learning_rate * w_gradient
b = b - learning_rate * b_gradient
loss_after = loss_for_example(w, b)

print(f"Updated parameters: w = {w:.2f}, b = {b:.2f}")
print(f"Loss before: {loss_before:.3f}")
print(f"Loss after: {loss_after:.3f}")

At our starting settings, a learning rate of $0.1$ would change $w$ from $0.5$ to $0.6$ and $b$ from $-1$ to $-0.95$. Try those values in the visual with $y = 1$. Does the loss decrease?

Automatic differentiation

In the practice activity for this unit, we estimate a local rate of change by nudging an input and dividing the resulting output change by the input step. This is a finite difference approximation. It gives us a useful way to interpret a derivative.

PyTorch's automatic differentiation, or autodiff, uses a different approach: it records operations and combines their derivative rules using the chain rule. It does not need to repeatedly nudge each parameter and rerun the model to estimate a gradient. Its computed gradients are still subject to the usual limits of floating point arithmetic.

For this lesson, focus on reading the graph, identifying what depends on what, and interpreting what a gradient tells us. PyTorch's introduction to autograd shows how these ideas appear in code.

Further reading

For a deeper look at backpropagation and automatic differentiation, see the following resources:

• https://colah.github.io/posts/2015-08-Backprop/

• https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html

Next steps

• Learn how to implement, train, and evaluate a neural network using PyTorch

• Want to play around with deep MLPs? See this interactive tool from Tensorflow

References