Overview

In this unit, we'll look at transformer architectures and attention mechanisms. 👀

Outcomes

By the end of this unit, you'll be able to …

• trace a small scaled dot product attention calculation
• explain which positions encoder, causal decoder, and encoder-decoder attention can use
• describe how BERT uses transformer encoder blocks [2019]
• describe the structure of BERT
• describe the pretraining tasks used for BERT

Resources

Readings	Self-attention mechanisms Transformers (first academic paper) The Illustrated Transformer BERT (first academic paper) Bidirectional Transformer Encoders	Section 9.7 of Jurafsky and Martin's Speech and Language Processing Attention is All you Need (Vaswani et al., 2017) Animated walkthrough of the transformer from Jay Alammar BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2019) Chapter 11 of Jurafsky and Martin's Speech and Language Processing
PyTorch	The Annotated Transformer	Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, and Stella Athena's walkthrough and PyTorch implementation of the transformer architecture (based on an earlier version by Alexander Rush, Vincent Nguyen, and Guillaume Klein)

Learn

Unit checklist

• Transformers for NLP
• Trace a small attention example
• Compare attention masks

Practice

Try to put these ideas into practice for your shared task and class competition submissions.

1. Ashish Vaswani and Noam M. Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin (2017). Attention is All you Need. NIPS.
2. Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics 4171-4186 10.18653/v1/N19-1423
3. Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
4. Goldberg, Yoav (2017). Neural Network Methods for Natural Language Processing. Morgan & Claypool Publishers https://ebookcentral.proquest.com/lib/uaz/reader.action?docID=4843762
5. Ruder, Sebastian (2017). Deep Learning for NLP Best Practices. https://www.ruder.io/deep-learning-nlp-best-practices