Overview

This tutorial provides some useful default settings and strategies when using deep learning for NLP.

Resources

Yoav Goldberg's Neural Network Methods for NLP [2017] is full of practical tricks for training various types of neural networks.
For a deeper dive into task-specific tricks, see Sebastian Ruder's more extensive Deep Learning for NLP Best Practices [2017]. This tutorial omits discussion of RNNs and transformers (topics we will cover elsewhere).

Outcomes

After completing this lesson, you'll be able to …

detect vanishing gradients
identify potential solutions for addressing vanishing gradients
detect exploding gradients
identify potential solutions for addressing exploding gradients
implement regularization strategies in PyTorch

Problems

Let's look at some commonly encountered problems in deep learning and techniques that can hep address these problems.

Imbalanced data

Class imbalance (an unequal proportion of each class in training data) is common in datasets. Due to annotation costs or rarity, you may have few examples of a particular class. Training on the data as-is will introduce a label bias (the model will learn to expect one class to be more common than another). To avoid such a bias, you need some way of rebalancing the data or weighting your loss.

Addressing imbalanced data

Undersampling

One option for addressing class imbalance is to find the size of the smallest class and discard items from the other classes until all classes are equally represented (i.e., all have the number of datapoints as the minority class); however, this may toss out useful examples that differ from the included datapoints in important ways that might limit the model's ability to generalize.

Carl Weathers (in Arrested Development): Whoa, whoa, whoa. There's still plenty of meat on that bone. Now you take this home, throw it in a pot, add some broth, a potato. Baby, you've got a stew going.

Another option is to randomly undersample each class in epoch to allow datapoints that were set aside in an earlier epoch another opportunity to appear. In PyTorch, this can be achieved using the torch.utils.data.sampler.WeightedRandomSampler. This sampling can be performed with or without replacement. First, we need to assign a weight to each class. One option to discount the majority class is to weight each class by its inverse frequency:

$w_{c_{n}} = \frac{1}{\text{count}(c_{n})}$

from torch.utils.data import WeightedRandomSampler, Subset
 
# define dataset, model, etc. ...
for epoch in range(1, max_epochs+1):
  # indices of our weighted sample
  indices = WeightedRandomSampler(weights=class_weights, num_samples=len(dataset), replacement=True)
  # select items from dataset by their indices
  subset = Subset(dataset, indices)
  myloader = DataLoader(subset)
  for i, batch in enumerate(myloader):
    # optionally track current iteration
    iteration = epoch * i
    # train as usual

Overfitting

Because DNNs have such a large number of parameters, it is very easy to inadvertently overfit when training (memorizing the training data without generalizing to unseen data).

Avoiding overfitting

There are several strategies we can adopt to reduce the risk of overfitting.

Fewer parameters

Reducing the overall number of parameters will help to address overfitting. Consider reducing the size of each layer (fewer hidden units) and the number of hidden layers (try a shallower network).

Weight decay

Functionally equivalent to L2 regularization, weight decay is a way of controlling the magnitude of the weights by downscaling according to some hyperparameter $\lambda$ . PyTorch's implementation of optimizers like SGD and Adam support weight decay as an optional hyperparameter. Increasing the value will increase the penalty on large weights:

$\frac{\delta{L}}{\delta w_{i}} + \lambda w_{i}$

from torch import optim
 
# model here
# ...
 
optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=0.5)

Dropout

At each forward pass during training with some probability $p$ , each neuron has a chance to have its activation set to 0. A good default value for $p$ commonly used in many NLP tasks is 0.5.

Early stopping

A lower loss on the training data doesn't always guarantee a more reliable model. To determine this, it is important to monitor your model's performance on some held-out data to better gauge the model's ability to generalize to unseen data. Often, this means setting aside a portion of your training data for development or validation (ex. a stratified selection of 10% of your training data). You may not want to perform this validation check after each iteration, as it may be noisy (especially early in training) and time-consuming. Instead, you may instead choose to do it every $k$ iterations or at the end of each training epoch.

Rather than training for a fixed number of epochs, it is better to specify a maximum number of epochs. By monitoring performance on training and validation partitions, you may find an earlier timestep to end training before your model begins to overfit or memorize the training data. This technique is known as early stopping. If performance on the validation partition begins to degrade while the loss on the training data continues to decrease, your model is likely overfitting.

If the degradation on validation is "small", though, you may want to wait to see if the degradation is a temporary flucuation (ex. a local instability) or representative of a trend. Once performance on validation has continued to show a degradation (a hyperparameter known as patience), you will want to stop training and use the best snapshot of your model. While PyTorch doesn't provide an out-of-the-box implementation for early stopping, many high-level wrappers like pytorch-lightning and pytorch-ignite have implementations. Here is an example of checkpointing and early stopping using pytorch-ignite:

from ignite.engine import Engine
from ignite.handlers import EarlyStopping, ModelCheckpoint
from ignite.metrics import Accuracy, Loss, Precision, Recall
 
# define model, loss, optimizer, 
# train_dataloader, validation_dataloader, etc.
# ...
 
# set device and configure model 
device = "cpu"
if torch.cuda.is_available():
  device = "cuda"
  model.cuda(device)
 
# see https://pytorch.org/ignite/metrics.html#metric-arithmetics
precision = Precision(average=False)
recall = Recall(average=False)
# we can compose metrics
F1 = (precision * recall * 2 / (precision + recall)).mean()
 
metrics = {
  "accuracy": Accuracy(),
  "precision": precision,
  "recall": recall,
  "f1": F1,
  "loss": Loss(criterion)
}
 
# just a function that defines 
# how we process a batch in training
def train_step(engine, batch):
  model.train()
  optimizer.zero_grad()
  x, y_true = batch
  x = x.to(device)
  y_true = y_true.to(device)
  y_hat = model(x)
  loss = criterion(y_hat, y_true.float())
  loss.backward()
  optimizer.step()
  return loss.item()
 
# a function that defines 
# how we process data for evaluation
# adjust as appropriate
def validation_step(engine, batch):
  model.eval()
  batch = {k: v.to(device) for k, v in batch.items()}
  x, y = batch
  with torch.no_grad():
    logits = model(x)
    predictions = torch.argmax(logits, dim=-1)
    return {'y_pred': predictions, 'y': y}
 
 
trainer = Engine(train_step)
 
evaluator = Engine(validation_step)
 
# attach metrics to our evaluator
for name, metric in metrics.items():
    metric.attach(evaluator, name)
 
 
n = 10
# after every 10 iterations (batches), 
# evaluate the model on both train and validation sets
@trainer.on(Events.ITERATION_COMPLETED(every=n))
def log_performance(trainer):
  # run against training
  evaluator.run(train_dataloader)
  metrics = evaluator.state.metrics
  print(f"""
  Training (Epoch {trainer.state.epoch})
  ........................................ 
  Precision: {metrics['precision']:.2f} 
      Recall: {metrics['recall']:.2f} 
          F1: {metrics['f1']:.2f}
  ........................................
  """
  )
  # run against validation
  evaluator.run(validation_dataloader)
  metrics = evaluator.state.metrics
  print(f"""
  Validation (Epoch {trainer.state.epoch})
  ........................................ 
  Precision: {metrics['precision']:.2f} 
      Recall: {metrics['recall']:.2f} 
          F1: {metrics['f1']:.2f}
  ........................................
  """
  )
 
checkpoint_handler = ModelCheckpoint(dirname="/data/experiment-x/models/snapshots", filename_prefix='my-model', n_saved=2, create_dir=True)
# as an alternative to decorators, 
# we can also register event handlers using .add_even_handler
trainer.add_event_handler(Events.ITERATION_COMPLETED(every=n), checkpoint_handler, {"model": model})
 
# in order to use early stopping,
# we need to periodically record performance
def evaluate(engine):
  evaluator.run(train_dataloader)
  evaluator.run(validation_dataloader)
 
trainer.add_event_handler(Events.EPOCH_STARTED, evaluate)
 
# now we can set up early stopping.
# we could use any recorded metric we want.
# here we'll use f1.
def early_stopping_using(engine):
  return engine.state.metrics['f1']
 
es_handler = EarlyStopping(
  patience=10, 
  score_function=early_stopping_using, 
  trainer=trainer
)
evaluator.add_event_handler(Events.EPOCH_COMPLETED, es_handler)
 
# train!
trainer.run(dataloader, max_epochs=50)

Failing to converge

If your loss decreases and then starts vacillating between increasing and decreasing, you may need to adjust your learning rate to continue learning.

Dynamic learning rates

When you first begin training, the larger updates may safely be taken. As learning progresses, smaller steps are often necessary to get a network to converge. One way of achieving this is through learning rate scheduling where for each update, the learning rate is calculated using the initial learning rate divided by the iteration (batch) number.

In PyTorch, we can do something like the following:

for i, data in enumerate(training_loader):
  # data and labels for batch
  x, y_true = data
  # zero gradients
  optimizer.zero_grad()
  # our forward pass
  y_hat = model(x)
  # calculate our loss
  loss = criterion(y_hat, y_true)
  # calculate our gradients
  loss.backward()
  optimizer.param_groups[0]["lr"] = starting_lr / i
  optimizer.step()

The torch.optim.lr_scheduler package provides implementations for several alternatives for adaptive learning rates.

Unstable learning

Is your loss going up and down? This suggests there is some instability in your network or in how you've structured your training loop.

Addressing unstable learning

There are several changes that can be made to improve learning stability.

Increase batch size

Larger batches may make for smoother learning/more consistent updates. Try doubling your batch size and training for a few epochs. What do you observe?

Gradient accumulation

Memory limitations may not make it feasible to increase the batch size. One solution is to accumulate gradients over several batches, average them, and then make a single update:

# how many batches should we wait
# before updating our parameters?
accumulate_for = 4
# a tally of how many batches we've processed 
# since we last updated our model's parameters
batches_processed = 0
loss = 0
for i, data in enumerate(training_loader):
  # data and labels for batch
  x, y_true = data
  # our forward pass
  y_hat = model(x)
  # calculate our loss
  current_loss = criterion(y_hat, y_true)
  # running average of our loss (for plotting, tracking, etc.)
  loss = (loss + current_loss) / batches_processed
  # calculate (and accumulate gradient)
  loss.backward()
  if (batches_processed) % accumulate_for == 0:
    optimizer.step()
    # why not optimizer.zero_grad()? 
    # sometimes they're the equivalent (sometimes not).
    # see https://stackoverflow.com/a/61901561
    model.zero_grad()
    # reset our loss and batch tally
    loss = 0
    batches_processed = 0

Layer normalization

Normalizing the parameters of each layer can help to improve learning speed and stability.

class MyAwesomeNetwork(nn.Module):
 
  def __init__(self, input_size: int, hidden_size: int):
    super(MyAwesomeNetwork, self).__init__()
 
    self.architecture = nn.Sequential(
      nn.Linear(input_size, hidden_size),
      # normalize weights after affine transformation
      nn.LayerNorm(hidden_size),
      # non-saturating activation function
      nn.ReLU(),
      # rinse and repeat for later layers
      nn.Linear(hidden_size, hidden_size),
      # normalize weights after affine transformation
      nn.LayerNorm(hidden_size),
      # classification layer 
      # (assuming a binary classification problem)
      nn.Sigmoid(),
    )
  def forward(self, x: torch.Tensor) -> torch.Tensor:
    return self.architecture(x)

L2 regularization

We can add a penalty term (also called a regularizer) to the loss to encourage the model to limit the magnitude of weights:

$L(f(x; \theta), y) + \lambda\Sigma_i w_{i}^{2}$

Vanishing gradients

When applying backpropagation to deep neural networks, the gradient generally decreases as we approach the input. In some cases, the gradient may approach zero for the parameters of early layers. Recall that when the gradient is zero, we can no longer update our parameters using gradient descent.

Signs of vanishing gradients

the model trains very slowly (the loss stops decreasing or slows to a crawl)
the gradient for early layers is zero while remaining large for late layers
weights for a layer are all zero

Causes of vanishing gradients

We've spent quite a bit of timing discussing the sigmoid activation function. The sigmoid takes some input and produces a value between zero and 1 no matter the magnitude of that input. When a neuron outputs a value close to either end of some bounded range, it is described as saturated. Activation functions that exhibit this property are sometimes described as "saturating non-linearities". Using saturating activation functions in early layers of a deep neural network can lead to vanishing gradients by dampening the signal we rely on for backpropagation. Luckily, there are many alternative activation functions that do not have this problem.

Avoiding vanishing gradients

Non-saturating activation functions

One of the mostly commonly used activation functions in deep learning architectures for NLP is the rectified linear unit (ReLU) which constrains only the lower bound of the input to be zero:

$\text{ReLU}(z) = \text{max}(0, z)$

Shallower networks

Reducing the number of layers may also help to avoid vanishing gradients.

Exploding gradients

In certain architectures and problems, one can encounter very large gradients where values approach positive infinity (ex. multiplying a series of gradients with values larger than 1). This results in large updates to parameters and model instability.

Signs of exploding gradients

parameters grow exponentially
parameters become NaN (approach positive or negative infinity)
loss becomes NaN (approach positive or negative infinity)
large swings in loss (instability)

Avoiding exploding gradients

The most effective means of addressing exploding gradients is gradient clipping.

Gradient clipping

"Clip" (or alternatively scale using gradient scaling) a gradients $g$ whenever they exceed some threshold ( $\gamma$ ). One typical version of this is to clip by the 2-norm of the gradient:

$\begin{cases} \gamma\frac{g}{\vert \vert g \vert \vert} &\text{if } g > \gamma \\ g &\text{otherwise } \end{cases}$

In PyTorch, we would use torch.nn.utils.clip_grad_norm_:

import torch
 
# perform backward pass/calculate gradients
loss.backward()
 
threshold = 1
torch.nn.utils.clip_grad_norm_(
  parameters=model.parameters(), 
  max_norm=threshold, 
  norm_type=2.
)
# update parameters
optimizer.step()

Not enough data

If labeled data is in short supply and expensive to supplement, you may struggle to train a deep neural network with many parameters. This can be mitigated through techniques like data augmentation (techniques for creating new datapoints from existing ones), multi-task learning (MTL), transfer learning, and fine tuning. We'll postpone in-depth discussion of most of these until later, but the key observation behind multi-task learning, transfer learning, and fine-tuning is that we can often repurpose a network trained for a different but related task by replacing the final layer (and potentially adding some layers between the classification). Why does this work? Deep neural networks are able to learn progressively more task-specific features from general ones. Early layers may contain information applicable to many different tasks.

Pre-trained embeddings

For neural network architectures using static word embeddings that are parameters of the model, we can initialize the value of those parameters using pre-trained word embeddings (ex. GloVe, skip-gram, etc.):

import torch
from torch import nn
 
# imagine these are loaded from numpy
pretrained_embeddings: torch.FloatTensor = torch.from_numpy(my_embeddings)
# see https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html
embeddings = nn.Embedding.from_pretrained(
  # imagine these are loaded from GloVe, etc.
  pretrained_embeddings, 
  # allow these to be adjusted through further training
  freeze=False
)
 
# get the embedding for the word (or char embedding, pos tag embedding, etc) with index 1
idx = 1
embedding_idx = torch.LongTensor([idx])
embeddings(embedding_idx)

This allows us to jump-start the learning process with a set of parameters that already encode useful information about language.

Many algorithms exist for learning embeddings in a self-supervised way (cf. skip-gram, CBOW).

Tokenization strategies

If we don't have enough data to learn good values for a deep neural network with a large number of parameters, we can try reducing the number of parameters. Aside from earlier discussions about removing hidden layers and reducing their size, we can also think about reducing the vocabulary size for our input. Rather than trying to learn a representation of every word in our vocabulary, we can learn representations for smaller units (sub-words) which will reduce our in turn reduce our vocabulary size (many words share sub-word sequences). On one extreme, we can tokenize into individual characters (or even bytes), but there are other tokenization stategies that afford a compromise.

Other tricks

This list of tips and tricks is by no means comprehensive. Here are a few other tricks to improve learning with deep neural networks.

Optimizers

With the right tuning of hyperparameters, many optimizers will yield equivalent performance. Adam has consistently shown to have excellent out-of-the-box performance for many NLP tasks.

Shuffle every epoch

Datapoints within a batch may have high variance/competing gradients. To avoid seeing the same batches (subgroups of datapoints) in each epoch, shuffle your datapoints for each epoch.

Residual connections

For deep neural networks, we can improve signal for backpropogation by adding residual connections that directly link earlier layers with later layers:

For input at layer $t$ ( $x_{t}$ ) we simply add the input for some earlier layer ( $x_{t-n}$ ) to the affine transformation:

$a(Wx_{t}+b) + x_{t-{n}}$

Fancier alternatives exist, such as dense connections where each layer is connected to all following layers in a weighted fashion.

References

Goldberg, Yoav (2017). Neural Network Methods for Deep Learning. Morgan & Claypool Publishers ebookcentral.proquest.com/lib/uaz/reader.action?docID=4843762
Ruder, Sebastian (2017). Deep Learning for NLP Best Practices. www.ruder.io/deep-learning-nlp-best-practices/