Overview

This tutorial provides an overview of recurrent neural networks (RNNs) and encoder-decoder architectures in the context of NLP.

Resources

Chapters 14-17 of Yoav Goldberg's Neural Network Methods for NLP [2017] covers many details about RNNs and encoder-decoder networks.
Understanding LSTM Networks (Olah, 2015)
The Unreasonable Effectiveness of Recurrent Neural Networks (Karpathy, 2015)

Outcomes

After completing this lesson, you'll be able to …

identify types of recurrent neural networks
describe uses of encoder-decoder architectures

Basic RNN

Many of the central tasks in natural language processing revolve around representing sequences.

We use these representations to do things like classify sequences, label elements in some sequences (POS tagging, NER, etc.), and map one sequence to another (translation, summarization, etc.). In earlier lessons, we saw that feedforward neural networks (MLPs, CNNs) are capable of learning local patterns in sequences (ex. $n$ -grams and $n$ -grams with holes), but they cannot generalize to sequences of arbitrary lengths. Handling longer sequences means increasing the input size. Feedforward neural networks cannot easily handle variable length sequences without resorting to tricks and simplifications (ex. padding, truncation, encoding positional information, etc.). Since it lacks a memory mechanism, a feedforward network is not well-suited to modeling a narrative or tracking changes to the state of some world over time.

Reccurrent neural networks (RNNs) overcome this limitation and possess a memory mechanism (a hidden state). More abstractly, memories of the past contextualize our interpretation of the present and influence the formation of new memories.

The basic formulation of an RNN is quite simple:

$h_{t} = \text{tanh}(Wx_{t} + Wh_{t-1})$

$h_{t}$ is the memory state at timestep $t$ (i.e., the memory associated with the $t^{th}$ element in the sequence). This design has many names including Elman network[1990], Jordan network, vanilla RNN, or a simple recurrent network (SRN).

Christopher Olah provides the following visual representation of an RNN:

![](images/RNN-rolled.png "A "rolled" representation of a simple recurrent network (SRN). Image credit to Chris Olah (https://colah.github.io/posts/2015-08-Understanding-LSTMs/)")

An unrolled representation of a simple recurrent network (SRN). Image credit to Chris Olah (https://colah.github.io/posts/2015-08-Understanding-LSTMs/)

The "unrolled" view of an RNN reveals it to be a feedforward neural network with parameter sharing that supports arbitrary sequences compactly. To better understand how memory is updated and passed along, let's look at a snippet of an RNN encoding three elements of some input $x$ :

In this way, a longer sequence can be encoded. Image credit to Chris Olah (https://colah.github.io/posts/2015-08-Understanding-LSTMs/)

$h$ represents a hidden state that encodes a memory of previously seen items. The $t$ subscript indicates the timestep for that memory. Each $t$ corresponds to an element in the input $x$ . Both the input $x$ and this hidden state $h$ share the same weight matrix $W$ . Rather than a sigmoid, here the activation function is a hyperbolic tangent (tanh) which ranges from -1 to 1. This activation provides better signal for longer sequences than the sigmoid and provides more control over how past information is applied to future states. Since our hidden state may take on negative values, the network can learn to discount the effect of the past.

Implementing an SRN encoder in PyTorch

from torch import nn
 
class ElmanRNN(nn.Module):
  """
  See https://pytorch.org/docs/stable/generated/torch.nn.RNNCell.html#torch.nn.RNNCell
  """
 
  def __init__(self, input_dim: int, hidden_dim: int):
    super().__init__()
    self.hidden_dim = hidden_dim
    # our initial hidden state.  We'll assume this is fixed.
    self.h0 = torch.rand(hidden_dim)
    self.rnn = nn.RNNCell(input_size=input_dim, hidden_size=hidden_dim, nonlinearity="tanh")
 
  def forward(self, x, h):
    # our new hidden state
    return self.rnn(x, h)

As a toy example, let's use this simple RNN to encode a character sequence:

import torch
EMB_DIM = 4
 
# we'll pretend this lookup table has pre-trained character embeddings
char_embeddings = {
  "<UNK>": torch.rand(EMB_DIM), # for unknown symbols (words, chars, etc.)
  "<BOS>": torch.rand(EMB_DIM),  # denotes "beginning of sequence"
  "<EOS>": torch.rand(EMB_DIM), # denotes "end of sequence"
  "<PAD>": torch.rand(EMB_DIM), # to create a fixed-length sequence for batching
  "c": torch.rand(EMB_DIM),
  "a": torch.rand(EMB_DIM),
  "t": torch.rand(EMB_DIM),
}
 
text = ["<BOS>"] + list("rat") + ["<EOS>"]
 
UNK = char_embeddings["<UNK>"]
 
simple_rnn = ElmanRNN(input_dim=EMB_DIM, hidden_dim=EMB_DIM)
 
# we'll start things off with a static start state
h = simple_rnn.h0
for char in text:
  embedding = char_embeddings.get(char, UNK)
  h = simple_rnn(embedding, h)
# our last h is a representation of our *entire* sequence

Implementing an SRN classifier in PyTorch

The encoder can be extended with a classification layer:

import torch.nn.functional as F
from torch import nn
from torch.autograd import Variable
 
class RNNClassifier(nn.Module):
  """
  See https://pytorch.org/docs/stable/generated/torch.nn.RNNCell.html#torch.nn.RNNCell
  """
 
  def __init__(
    self,
    input_dim: int, 
    hidden_dim: int, 
    num_classes: int, 
    # we can stack multiple layers of RNNs
    num_layers: int = 1,
    # If True, then the input and output tensors are provided as (batch, seq, feature) instead of (seq, batch, feature).
    batch_first: bool = False,
  ):
    super().__init__()
    self.hidden_dim = hidden_dim
    self.num_classes = num_classes
    self.batch_first = batch_first
    # our initial hidden state. 
    # Often initialized to zero, but this can be trained as part of the model.
    self.h0 = Variable(torch.rand(hidden_dim), requires_grad=True)
    self.rnn = nn.RNN(
      input_size=input_dim, 
      hidden_size=hidden_dim,
      nonlinearity="tanh", 
      num_layers=num_layers,
      batch_first=batch_first
    )
    # fully connected layer to further transform the hidden state
    self.fc = nn.Linear(self.hidden_dim, self.num_classes)
 
  def forward(self, x, pred: bool = True, h: Optional[torch.Tensor] = None):
    # our start state.
    # this might be passed in to the classifier (ex. imagine if you wanted a continuation of some "thought")
    h0 = h if h is not None else self.h0
    # our new hidden state
    if self.batch_first:
      batch_size, seq_len, elem_dim = x.size()
    else:
      seq_len, batch_size, elem_dim = x.size()
    # the dimensions of our hidden state must be compatible with that of our input
    # namely, we need to repeat our hidden state to match our batch size
    h0 = self.h0.unsqueeze(0).repeat(1, batch_size, 1)
    # output: the hidden states for each time step in the input sequence
    # hidden: the last hidden state
    output, hidden = self.rnn(x, h0)
    hn = self.fc(hidden)
    return F.log_softmax(hn) if pred else hn

Now let's use it to classify some random data points (let's imagine them as documents of some kind):

N_BATCHES = 6
EMB_DIM = 50
SEQ_LEN = 10
# let's create N_BATCHES batches of sequences of length SEQ_LEN where each element in each sequence has EMB_DIM dimensions
# in this case, the number of batches corresponds to the number of datapoints
x = torch.randn(N_BATCHES, SEQ_LEN, EMB_DIM)
 
clf = RNNClassifier(input_dim=EMB_DIM, hidden_dim=20, num_classes=3, batch_first=True)
clf(x)

Applications of RNNs

RNNs can be used to classify a sequence (words, documents, SQL queries, etc.) or produce a new sequence of any length (translation, summarization, etc.). The following image from Andrej Karpathy's The Unreasonable Effectiveness of Recurrent Neural Networks helps to illustrate this versatility:

![](images/diags.jpeg " From Andrej Karpathy's "The Unreasonable Effectiveness of Recurrent Neural Networks. We can produce a single label for a sequence, a label for each element of a sequence, or an entirely new sequence."")

Encoder-Decoders

Perhaps one of the most exciting applications of RNNs is in learning a mapping from one sequence to another (translation, summarization, words $\rightarrow$ syntactic structure, etc.).

We can train a pair of RNNs to map or translate from one sequence to another. We use one RNN to encode a source sequence (the encoder) and another RNN to "decode" the encoded source sequence to some target sequence (the decoder). This pairing of RNNs is known as an encoder-decoder architecture. The decoder's initial hidden state $D_{h_{0}}$ comes from the final hidden state of the encoder $E_{h_{n}}$ or a summation, averaging or tranformation of the hidden states of a stacked encoder (more sophisticated compositions of hidden states exist). The network is trained to produce the target sequence from the encoded source sequence.

Limitations of SRNs

While RNNs can theoretically learn to encode a sequence of any length, simple recursive networks suffer from vanishing gradients (ex. repeatedly multiplying smaller and smaller numbers) and memory limitations that can make it challenging to learn long (and even mid-) distance dependencies. For a real world example of such a dependency, consider the case of agreement in English:

The dolphins that traveled all the way to Miami from Havana each Thursday morning without fail are very friendly.

Various tricks exist for improving SRNs (sequence truncation, ReLU activation with gradient clipping, etc.), but none of these address limitations in the memory mechanism. SRNs have no explicit mechanism for selectively forgetting information. Gated RNNs address this shortcoming by introducing a series of gates for fine-grained control over the memory mechanism.

Improving RNNs with gates

One of the most popular and effective alternatives to SRNs is the Long short-term memory (LSTM) network[1997] which is composed of a memory state (cell) and 3 gates: input, output, and forget:

$i_{t} =\sigma(W_{ii} + x_{t} + b_{ii} + W_{hi}h_{t-1} + b_{hi})$

$f_{t}=\sigma(W_{if} + x_{t} + b_{if} + W_{hf}h_{t-1} + b_{hf})$

$g_{t}=\tanh(W_{ig} + x_{t} + b_{ig} + W_{hg}h_{t-1} + b_{hg})$

$o_{t}=\sigma(W_{io} + x_{t} + b_{io} + W_{ho}h_{t-1} + b_{ho})$

$c_{t}=f_{t} \odot c_{t-1} + i_{t} \odot g_{t}$

$h_{t}=o_{t} \odot \tanh(c_{t})$

where …

$i_{t}$ is the input gate
$f_{t}$ is the forget gate
$o_{t}$ is the output gate
$g_{t}$ is the cell gate
$c_{t}$ is the cell state at time $t$
$\tanh$ is the hyperbolic tangent function (an S-shaped function ranging from -1 to 1)
$\sigma$ is the sigmoid function
$\odot$ is the Hadamard (element-wise) product

The input gate controls the amount and type of new information allowed into the memory or cell of the LSTM. The output gate controls the amount and type of information (both past and new) that is retained in the memory of the LSTM. The forget gate controls amount and type of information that is erased from from the memory of the LSTM at each timestep.

These gates help provide stable gradients and selective memory. Later, bidirectional variants (BiLSTMs) [2013] further extended the capabilities of the LSTM by providing a way to encode sequences from both directions (forward and backward). The output of both the forward and backward encoders is concatenated, thus doubling the dimensionality of each hidden state. The code example below (see the GatedRnn's forward() implementation) shows one way of achieving this (note that you'll need to swap out the GRU in that example for an nn.LSTM).

GRUs

A popular alternative to the LSTM is the Gated Recurrent Unit (GRU) [2014] which removes the output gate and reduces the overall number of trainable parameters without sacrificing much performance. Just like LSTMs, several versions of GRUs exist, but the original is one seen most frequently in the wild.

Gated RNNs in PyTorch

Here is an example of using a stackable bidirectional GRU encoder in PyTorch:

from torch import nn
 
class GatedRNN(nn.Module):
    """
    Example of a bidirectional GRU-based RNN 
    for encoding some sequence.
    """
    def __init__(
      self,
      vocab_size: int,
      emb_dim: int,
      max_len: int,
      num_layers: int=1,
      dropout_prob: float=0.5,
    ):
        super().__init__()
 
        self.emb_dim = emb_dim
        self.max_len = max_len
        self.embed = nn.Embedding(vocab_size, emb_dim)
        # optionally stack layers of bidirectional GRUs
        self.rnn = nn.GRU(emb_dim, emb_dim, bidirectional = True, batch_first=True, num_layers=num_layers, dropout=dropout_prob)
        # we'll use the defaults for dropout
        self.dropout = nn.Dropout()
 
    def forward(
      self,
      src: torch.Tensor
    ) -> Tuple[torch.Tensor]:
 
        # add dropout to embedding
        embedded = self.dropout(self.embed(src))
 
        # output: the hidden states
        # hidden: the last hidden and cell state
        outputs, hidden = self.rnn(embedded)
        hidden = torch.tanh(
          # here we concatenate the last hidden state of the forward and backward gated RNN
          # you might choose to do something more exotic here 
          # (ex. concat the forward states for each layer and the backward states for each layer, etc.)
          torch.cat((
            # forward state of final layer (i.e., penultimate elem)
            hidden[-2,:,:], 
            # backward state of final layer (i.e., last elem)
            hidden[-1,:,:]
          ),
          # (i.e., h_f_n:h_b_n)
          dim = 1)
        )
 
        return outputs, hidden
 
model = GatedRNN(
  vocab_size = len(vocab),
  emb_dim = 50,
  max_len = MAX_LEN,
  num_layers = 2
)
 
# NOTE: model expects [seq_len, batch] by default
# since we used the batch_first=True flag, 
# we instead pass in [batch, seq_len]
 
# out: the state for each element of the sequence
# hidden: our final embedding for this sequence
out, hidden = model(x)

Preparing data

When encoding sequences, we must first build a vocabulary (a mapping of each unique symbol to some ID).

Padding

For efficient training, we pad groups of sequences to the same length. This can be done such that all elements in a batch have the same length or so that all elements in the dataset are padded to the same length. Padding performed by either repeatedly prepending or appending a special symbol to the sequence until the desired length is reached. The special symbol is represented with a tensor of zeros.

Preparing sequences with PyTorch for use with RNNs

import random
from typing import Text, Tuple, Sequence
from enum import Enum
 
#from torch.nn.utils.rnn import pad_sequence
from torchtext.vocab import build_vocab_from_iterator
from torchtext.vocab import Vocab
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
 
class SeqSymbols(Enum):
    """
    Some common special tokens.
    """
    BOS = "<BOS>"
    EOS = "<EOS>"
    PAD = "<PAD>"
    UNK = "<UNK>"
 
# imagine this is your dataset pre-tokenized.
examples = [
  # a single datapoint
  ["I", "like", "turtles"]
]
 
# lazily load the data one symbol at a time.
data_iterator = (symbol for datum in examples for symbol in examples)
# create a vocabulary from our set of symbols
vocab = build_vocab_from_iterator(
  data_iterator, 
  special_first=True, 
  specials=[symbol.value for symbol in SeqSymbols]
)
vocab.set_default_index(vocab[SeqSymbols.UNK.value])
 
def prepare_datum(
  datum: Sequence[Text], 
  max_len: int, 
  vocab: Vocab, 
  pad_initial: bool = True
) -> torch.Tensor:
  """
  Prepare a single datum for training by prepending and appending special tokens and padding to the specified length.
  """
  ids = [vocab[SeqSymbols.BOS.value]] + vocab.lookup_indices(datum) + [vocab[SeqSymbols.EOS.value]]
  padding = [vocab[SeqSymbols.PAD.value]] * (max_len - len(ids))
  # pad to max_len
  ids = padding + ids if pad_initial else ids + padding
  # convert to LongTensor for use with nn.Embedding layer
  return torch.tensor(ids, dtype=torch.long)
 
MAX_LEN = 10
 
# should be torch.tensor([0, 4, 5, 3, 1, 2, 2, 2, 2, 2], dtype=torch.long)
x = prepare_datum(["I", "like", "tortoises"], max_len=MAX_LEN, vocab=vocab, pad_initial=True)
# should be ['<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<BOS>', 'I', 'like', '<UNK>', '<EOS>']
x_inverse_transformed = vocab.lookup_tokens(x.tolist())

Embedding layers

RNNs use embedding layers to represent each element of the input sequence. These layers can be initialized with pre-trained embeddings (ex. skip-gram, GloVe, FastText, etc.) or randomly using method such Glorot or Xavier initialization. If pre-trained embeddings are used, one must decide whether to keep the parameters frozen or allow them to be updated as part of training (either from the beginning or gradually unfrozen).

Practice

Based on the samples above, create an encoder-decoder model using stacked bidirectional LSTMs. If you're feeling stuck, take a look at teh following resources:
https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html
https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

Next steps

See [2016] for a discussion of more exotic applications of RNNs
For some tips on improving the performance of LSTMs, see the LSTM Tricks section of Sebastian Ruder's Deep Learning for NLP Best Practices

References

Jeffrey L. Elman (1990). Finding Structure in Time. Cognitive Science. Wiley 179--211 10.1207/s15516709cog1402_1
Sepp Hochreiter and J\"urgen Schmidhuber (1997). Long Short-Term Memory. Neural Computation. MIT Press - Journals 1735--1780 10.1162/neco.1997.9.8.1735
Alex Graves (2013). Generating Sequences With Recurrent Neural Networks. ArXiv.
Cho, Kyunghyun and van Merrienboer, Bart and Bahdanau, Dzmitry and Bengio, Yoshua (2014). On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. arXiv 10.48550/ARXIV.1409.1259
Goldberg, Yoav (2017). Neural Network Methods for Deep Learning. Morgan & Claypool Publishers ebookcentral.proquest.com/lib/uaz/reader.action?docID=4843762
Ruder, Sebastian (2017). Deep Learning for NLP Best Practices. www.ruder.io/deep-learning-nlp-best-practices/
Chris Olah and Shan Carter (2016). Attention and Augmented Recurrent Neural Networks. Distill. Distill Working Group 10.23915/distill.00001