Overview

This tutorial provides an overview of transformers in the context of NLP.

Resources

Primary source

Code-first

The Annotated Transformer
a walkthrough and PyTorch implementation of the transformer architecture

Gentle introductions

The Illustrated Transformer
An animated walkthrough of the transformer from Jay Alammar

Textbook chapters

Outcomes

After completing this lesson, you'll be able to …

describe the transformer architecture
identify shortcoming of RNNs addressed by transformers

Motivation

As we saw previously, RNNs allow us to represent sequential information compactly and provide a representation of a memory. It may seem suprising to hear that RNNs are no longer the dominant architecture within neural methods for NLP.

Shortcomings of RNNs

Despite their strengths, RNNs have two major limitations.

Memory and focus limitations

In a typical encoder-decoder setup with RNN, we only use the final state of our encoder to seed our decoder. In other words, we cram all of that information (potentially for a very long sequence!) into a single fixed-length vector. This limits the amount of information that can be represented in the decoder. It is reasonable to think that accessing the memory and focus preferences associated with each token fed into the encoder would improve the decoder.

What should we be paying attention to when making each prediction? If you were to translate a story, you might revisit the source text at various times throughout the process, perhaps focusing on a different aspect of the text each time and ignoring other details (ex. agreement, characters and actions, etc.). While gated RNNs provide a way of generating selective memory and retaining and discarding information, they do not provide a way of encoding selective focus. In 2014, Bahdanau et al.[2014] introduced an attention mechanism to address this limitation.

Attention mechanisms are compatible with RNNs, and their inclusion markedly improves performance on most tasks (see ELMo for one example)[2018].

For a visual guide to attention mechanism, see the Let's Pay Attention Now section of Jay Alammar's Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention).

Slow to train

Since they process data sequentially (i.e., you need the prior hidden state to generate the next hidden state), RNNs are slow to train. ⏳

Transformer

The transformer architecture, introduced in Vaswani et al.'s Attention is All You Need[2017], addresses the shortcomings of RNNs. Transformers have three key components: position encodings, self-attention, and multi-head attention.

Encoding positions

The transformer is a deep feedforward architecture. How is the position of each token represented? How can we clue the model in that $t_{i}$ is closer (in terms of linear distance) to $t_{i+1}$ than $t_{i-2}$ ?

From Vaswani et al. (p. 5; Section 3.5):

Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence

Rather than learn embeddings for positions, the authors elected to use a fixed encoding based on sine and cosine waves where the frequency reflected a token's position (see Section 3.5).

Attention

Attention is an idea that preceded the advent of the transformer (Bahdanau et al., 2014)[2014]. Various forms of attention exist (hard, soft, etc). In Attention is All You Need, the authors elected to use a form of self-attention calculated using a scaled dot product:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}V\right)$

The self-attention mechanism from Figure 2 of Vaswani et al.'s Attention is All You Need (Section 3.2.1) and reproduced in Huang et al.'s The Annotated Transformer. In their work, self-attention was implemented as a scaled dot product.

The Query, Key, and Value matrices are parameters of the model that are learned during training. Conceptually, we can think of their interaction as running some query (query) against entries (keys) in a database to get back a set of top matches (values).

In the transformer, the role of the attention mechanism is to answer the following questions:

how much do the representations of earlier tokens contribute to predicting the current token $t_{i}$ in an encoder or decoder?
how much do the representations of the encoder at each timestep contribute to predicting each token in our decoder?

Jay Alammar's The Illustrated Transformer provides a visual guide to this attention mechanism.

Multi-head attention

Vaswani et al. define Multi-head attention as a concatenation of attention head outputs.

From Section 3.2.3, …

In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder.

The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder

Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.

Process for multi-head attention from Figure 2 of Vaswani et al.'s Attention is All You Need (Section 3.2.1) and reproduced in Huang et al.'s The Annotated Transformer. Several projections of attention are concatenated before being projected a final time.

TL;DR

learn to attend to different features and then combine these "attention heads" into attention-weighted representations of each token
for each token/timestep, linearly project the attention function $n$ times, concatenate these $n$ attention representations, and project the concatenated representation a final time to compress and compose.

Transformer-based models

A number of architectures known as transformer-based models have been proposed. Many of these use only an encoder (ex. BERT[2019] and its variants) or only a decoder (ex. the Generative Pre-trained Transformer[2018] (GPT- $n$ ) family).

Encoder-only transformers

BERT

Perhaps the most impactful variant of encoder-only transformers is BERT (Bidirectional Encoder Representations from Transformers)[2019] which uses a stack of encoder blocks. In each block, attention for each token is computed over the output of the previous block. Unlike autoregressive language models, BERT is bidirectional. For a token $t_{i}$ , the attention calculation involves tokens $t_{0:i-1}$ (the left context) and $t_{i+1:n}$ (the right context).

Features

12 layers of transformer blocks
each block contains a multi-head attention layer comprised of 12 attention heads
A subword vocabulary
Originally 30,000 tokens generated using the WordPiece algorithm (Schuster and Nakajima, 2012)[2012]
Gaussian Error Linear Unit (GELU) activation (Hendrycks and Gimpel, 2016)
Available in PyTorch
110 million parameters

Pretraining

BERT uses two self-supervised pretraining tasks to learn general features of language.

Masked Language Modeling (MLM)

Masked Language Modeling is akin to a Cloze test where a random percentage of tokens in each sentence is masked during training and the model is tasked with learning to "fill in the blanks". Unlike the original transformer, BERT makes use of bidirectional information to learn to make these predictions.

Next Sentence Prediction (NSP)

In the Next Sentence Prediction (NSP) task, BERT is trained to predict whether one sentence follows another in text. This binary classification problem helps BERT to learn long-distance dependencies and coherence.

Special tokens

BERT's pretraining task utilize a handful of special tokens. Like the subword tokens, embeddings are learned for each of these special tokens during training.

TOKEN	DESCRIPTION
`[CLS]`	Precedes the first token in all input. Often used in a feedforward network to represents class labels.
`[SEP]`	Separates two segments of text (sentences in the original implementation).
`[END]`	Final token in some input.
`[PAD]`	Padding token used to create batches of equal length. Repeatable.
`[MASK]`	Used for the MLM task for cloze-style blanks.

Variants of special tokens

Due to different tokenization algorithms or design decisions, variants of transformers (ex. BERT vs RoBERTa) may use different strings to represent these special tokens. If you're using the Hugging Face transformers and tokenizers libraries, you can easily determine this by inspecting a tokenizer's .all_special_tokens property:

from transformers import AutoTokenizer
 
bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
roberta_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
 
print(f"BERT special tokens: {bert_tokenizer.all_special_tokens}")
print(f"BERT MASK token: {bert_tokenizer.mask_token}")
 
print(f"RoBERTa special tokens: {roberta_tokenizer.all_special_tokens}")
print(f"RoBERTa MASK token: {roberta_tokenizer.mask_token}")

These differences are important to keep in mind when preparing data for fine-tuning or extended pre-training.

Decoder-only transformers

Some variations of the transformer architecture such as GPT are decoder-only. These models are autoregressive (i.e., single direction models where each prediction only factors in prior predictions). Decoder-only transformers are a form of generative models. Text-based decoder-only transformers generate output one word (fragment) at a time. To predict the next token, the previously generated token is concatenated to the input.

Decoder-only transformers (such as those based on the GPT architecture) have received widespread attention for their ability to generate plausible sounding text using zero or few-shot in-context learning (ICL) in the form of prompts. Essentially, the model is fed a) a task description, b) one or more examples of desired output, and c) a incomplete example. Given this input, the model generates a continuation (or completion) of the prompt. Because of the unidirectional generation process, decoder-only transformers have a greater computational cost compared encoder-only transformers.

Avoiding potential pitfalls of decoder-only transformers

When using decoder-only transformers, it is important to keep a few things in mind:

Non-deterministic output: Depending on the decoding strategy (ex. any form of temperature-based sampling), output can be non-deterministic.
Plausible, but not factual: Generative models are trained to produce plausible (probable) completions, but not necessarily factual completions.
Generative models may spew offensive responses: While products like OpenAI's GPT-4 have many safeguards on the output in place, many unfiltered web dumps used to train generative LLMs may produce models that echo the biases of the text on which it was trained to complete.
Tendency to overgenerate: When using a a decoder-only transformer in inference mode, you specify a number ( $n$ ) of tokens to generate. Your "answer" may take fewer tokens to be generated. Tricks for detecting boundaries include a) providing examples of the output structure, b) fine-tuning using special tokens denoting a sequence end, c) detecting repetitions, and d) tracking large shifts in the softmax value for each token in the generated sequence.
Lack of attribution for generation: It can be challenging to tease out an attribution for any generation.

Foundation Models

Transformer-based models typically consist of a minimum of hundreds of millions of parameters (sometimes billions or trillions) trained on billions of words. Training is typically parallelized across many units of specialty hardware (GPUs, TPUs, etc.). The cost to train such models is out of the reach of many. Fortunately, in most cases it isn't necessary to train these models from scratch. Instead, one takes a so-called "foundation model" (a large pre-trained transformer-based model) as starting point, adds one or more layers to it with a new classification head, and "fine-tunes" the parameters of the model on a smaller, task-specific dataset.

Fine-tuning LLMs

Given the large numbrer of parameters, it is inefficient (and inadvisable) to adjust all LLM parameters during fine-tuning. Moreover, in adjusting all (or even most) parameters during fine-tuning, one risks inadvertantly erasing or "forgetting" useful information learned during pre-training. One option is to freeze all parameters but the final few layers. The disadvantage of this is that the entire network becomes highly specialized to a specific task.

Another efficient alternative to fine-tuning a subset of network layers is to train an adapter (a modular network component that is task-specific). Adapters, however, can lead to slow inference 🐌.

The current dominant approach to parameter-efficient fine-tuning is Low-Rank Adaptation of Large Language Models (LoRA)¹ where two small, task-specific weight matrices are learned (see Figure 1) and then added to the original network when it used for the target task. Training these small matrices requires fewer resources (ex. reduced RAM consumption and disk storage) since fewer parameters need to be tuned. Since the original architecture is unaltered, the method has no impact on inference time. Many small, task-specific transformations can be learned and then effficiently applied to a shared common foundation at inference time.

Transformer variants

There are many variants of the transformer architecture. Most of the differences can be characterized in terms of a) different tokenization algorithms, b) encoder-only/decoder-only/encoder-decoder design, c) the number of model parameters, d) variations of pre-training tasks, or e) modality (text vs text+images). Below is a summary of several popular transformer-based architectures:

MODEL	YEAR	TYPE	PROPERTIES
BERT	2019	encoder	see prior section
RoBERTa	2019	encoder	BBPE tokenization for better representations of unknown words (see Section 4.4); Dynamic masking for MLM pretraining (see Section 4.1); Introduces harder alternative to NSP pre-training task (`DOC-SENTENCES` described in Section 4.2)
BART	2019	encoder-decoder	Noise-based transformations of input (deletion, permutation, rotation, etc.) for pre-training (see Section 2.2 and Figure 2).
T5	2019	encoder-decoder	Frame all tasks (classification, translation, etc.) as prompting or "text to text" generation (see Figure 1); presents evidence for scale-based improvements.
GPT	2018	decoder	Unidirectional ("causal") language modeling; 4 architectural versions to handle tasks (see sections 3.2. and 3.3 of the linked paper)

Popular libraries

The Hugging Face transformers library is a Python package that provides a unified interface to many popular transformer-based models, including support for fine-tuning these models for a variety of tasks (classification, translation, etc.). Hugging Face also maintains a Model Hub where users can share their models along with the datasets used to train them. The Hugging Face transformers API is compatible with PyTorch, allowing users to easily extend models that incorporate transformers in custom architectures.

Getting started with the Hugging Face `transformers` library

Chapters 1-8 of the official Hugging Face course
NOTE: Hugging Face estimates each chapter taking 6-8 hours to complete
Natural Language Processing with Transformers (book)

References

Goldberg, Yoav (2017). Neural Network Methods for Deep Learning. Morgan & Claypool Publishers ebookcentral.proquest.com/lib/uaz/reader.action?docID=4843762
Dzmitry Bahdanau and Kyunghyun Cho and Yoshua Bengio (2014). Neural Machine Translation by Jointly Learning to Align and Translate. CoRR.
Galassi, Andrea and Lippi, Marco and Torroni, Paolo (2020). Attention in natural language processing. IEEE Transactions on Neural Networks and Learning Systems. IEEE 4291-4308
Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics 4171-4186 10.18653/v1/N19-1423
Peters, Matthew E. and Neumann, Mark and Iyyer, Mohit and Gardner, Matt and Clark, Christopher and Lee, Kenton and Zettlemoyer, Luke (2018). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics 2227-2237 10.18653/v1/N18-1202
Alec Radford and Karthik Narasimhan (2018). Improving Language Understanding by Generative Pre-Training. OpenAI
Mike Schuster and Kaisuke Nakajima (2012). Japanese and Korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5149-5152
Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, \L ukasz and Polosukhin, Illia (2017). Attention is All You Need. Advances in Neural Information Processing Systems. Curran Associates, Inc.

Footnotes

QLoRA is a quantized form of LoRA that offers additional efficiency gains through reduced precision of weights. ↩