Language Understanding and Language Models¶

Table of Contents

Activations¶

Note

[SiLU] Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
[GELU] Gaussian Error Linear Units
[Swish] Searching for Activation Functions
[Swish v GELU] On the Disparity Between Swish and GELU
[GLU] GLU Variants Improve Transformer

Normalisation¶

[Internal Covariate Shift][BN] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
[LN] Layer Normalization
[RMSNorm] Root Mean Square Layer Normalization
[PreLN][Detailed Study with Mean-Field Theory] On Layer Normalization in the Transformer Architecture

Warning

For theoretical understanding of MFT and NTK, start from this MLSS video here.

Tokenizers¶

WordPiece¶

See also

Google’s Neural Machine Translation System

SentencePiece¶

See also

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Byte-Pair Encoding (BPE)¶

See also

Neural Machine Translation of Rare Words with Subword Units

Word Embeddings¶

Note

Word2Vec: Efficient Estimation of Word Representations in Vector Space
GloVe: Global Vectors forWord Representation
Evaluation methods for unsupervised word embeddings

Sequence Modeling¶

RNN¶

See also

Implementation examples live in Language Modeling Implementations.

Note

LSTM¶

See also

Implementation examples live in Language Modeling Implementations.

Note

StatQuest on LSTM

Transformer¶

General Resources¶

Warning

[github.com] LLM101n: Let’s build a Storyteller
[jmlr.org] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
[epoch.ai] How has DeepSeek improved the Transformer architecture?

Note

[harvard.edu] The Annotated Transformer
[jalammar.github.io] The Illustrated Transformer
[lilianweng.github.io] Attention? Attention!
[d2l.ai] The Transformer Architecture
[newsletter.languagemodels.co] The Illustrated DeepSeek-R1: A recipe for reasoning LLMs

Position Encoding¶

Note

[arxiv.org] Position Information in Transformers: An Overview
[arxiv.org] Rethinking Positional Encoding in Language Pre-training
[eleuther.ai] RoPE
[arxiv.org] LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
[arxiv.org] RoFormer: Enhanced Transformer with Rotary Position Embedding

Attention¶

Understanding Einsum¶

Warning

Implementation examples live in Language Modeling Implementations.

Note

Attention implementation examples live in Language Modeling Implementations.

UnitTest¶

See also

Unit tests live in Language Modeling Implementations.

Resources¶

[MHA] Attention Is All You Need
[MQA] Fast Transformer Decoding: One Write-Head is All You Need
[GQA] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
[tinkerd.net] Multi-Query & Grouped-Query Attention

Decoding¶

Beam Search, Top-K, Top-p/Nuclear, Temperature
[mlabonne.github.io] Decoding Strategies in Large Language Models
Speculative Deocding

Transformer Architecture¶

Encoder [BERT]¶

Note

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Additional Resources
- [tinkerd.net] BERT Tokenization
- [tinkerd.net] BERT Embeddings,
- [tinkerd.net] BERT Encoder Layer
A Primer in BERTology: What we know about how BERT works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
XLM: Cross-lingual Language Model Pretraining
TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval

Decoder [GPT]¶

Note

[jalammar.github.io] The Illustrated GPT-2
[github.com] karpathy/nanoGPT
[cameronrwolfe.substack.com] Decoder-Only Transformers: The Workhorse of Generative LLMs
[openai.com] GPT-2: Language Models are Unsupervised Multitask Learners
[openai.com] GPT-3: Language Models are Few-Shot Learners

Encoder-Decoder [T5]¶

Note

T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Autoencoder [BART]¶

Note

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Cross-Lingual¶

Note

[Encoder] XLM-R [Roberta]: Unsupervised Cross-lingual Representation Learning at Scale
[Decoder] XGLM [GPT-3]: Few-shot Learning with Multilingual Generative Language Models
[Encoder-Decoder] mT5 [T5]: A Massively Multilingual Pre-trained Text-to-Text Transformer
[Autoencoder] mBART [BART]: Multilingual Denoising Pre-training for Neural Machine Translation

See also

[ruder.io] The State of Multilingual AI