Reading List¶
Ilya’s Paper List¶
Fundamentals¶
The Annotated Transformer
The First Law of Complexodynamics
The Unreasonable Effectiveness of RNNs
Understanding LSTM Networks
Recurrent Neural Network Regularization
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Pointer Networks
ImageNet Classification with Deep CNNs
Order Matters: Sequence to Sequence for Sets
GPipe: Efficient Training of Giant Neural Networks
Deep Residual Learning for Image Recognition
Multi-Scale Context Aggregation by Dilated Convolutions
Neural Quantum Chemistry
Attention Is All You Need
Neural Machine Translation by Jointly Learning to Align and Translate
Identity Mappings in Deep Residual Networks
A Simple NN Module for Relational Reasoning
Variational Lossy Autoencoder
Relational RNNs
Quantifying the Rise and Fall of Complexity in Closed Systems
Neural Turing Machines
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Scaling Laws for Neural LMs (arxiv.org)
A Tutorial Introduction to the Minimum Description Length Principle (arxiv.org)
Machine Super Intelligence Dissertation (vetta.org)
PAGE 434 onwards: Komogrov Complexity (lirmm.fr)
CS231n Convolutional Neural Networks for Visual Recognition (cs231n.github.io)
Foundations and Trends® in Machine Learning¶
Additional Resources¶
[GPT-1](https://lnkd.in/gJ5Pe3HG)
[GPT-2](https://lnkd.in/gatQi8Ud)
[GPT-3](https://lnkd.in/g43GzYfZ)
[GPT-4](https://lnkd.in/ga_xEpEj)
[Llama-2](https://lnkd.in/gutaGW8h)
[Tools](https://lnkd.in/gqJ3aXpS)
[Gemini-Pro-1.5](https://lnkd.in/gbDcYp89)
[Agentic Patterns Series](https://lnkd.in/gphZ6Y5s)
University courses¶
[stanford.edu] cs224n Natural Language Processing
[stanford.edu] cs25 Transformers United
[stanford.edu] cs330 Deep Multi-task and Meta Learning
[stanford.edu] cs224w Machine Learning with Graphs
[stanford.edu] cs231n Convolutional Neural Networks for Visual Recognition
[stanford.edu] Deep generative models
[stanford.edu] cs229s Systems for Machine Learning
[stanford.edu] Stanford MLSys Seminar
[stanford.edu] cs246 Mining Massive Dataset
[stanford.edu] Math 104 Applied Matrix Theory
[cs.cmu.edu] Large Language Model Systems
[cs.cmu.edu] 15-496/15-859X: Computer Science Theory for the Information Age, Spring 2012
Search and Recommender Systems¶
Tag |
Title |
|---|---|
Survey; CTR |
Deep Learning for Click-Through Rate Estimation |
Survey; Embedding |
Embedding in Recommender Systems: A Survey |
Survey; Ranking |
A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation |
Survey; Retrieval |
A Comprehensive Survey on Retrieval Methods in Recommender Systems |
Survey; Objective |
Self-Supervised Learning for Recommender Systems: A Survey |
Method; CTR |
Factorization Machines |
Method; CTR |
Wide & Deep Learning for Recommender Systems |
Method; CTR |
DeepFM: A Factorization-Machine based Neural Network for CTR Predictio |
Method; CTR |
xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems |
Method; CTR |
Deep & Cross Network for Ad Click Predictions |
Method; CTR |
DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems |
Method; CTR |
Deep Interest Network for Click-Through Rate Prediction |
Method; Feedback Delay |
Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction |
LLMs¶
Note
[Practical] The Large Language Model Playbook
[Practice] masteringllm.com
[Sebastian Rachka] Understanding Large Language Models
[pair.withgoogle.com] Can Large Language Models Explain Their Internal Mechanisms?
[Github] LLM Course
Applied LLMs¶
Note
[Blog] eugeneyan.com
[Blog] sh-reya.com
[Blog] hamel.dev
[Github] LLM4Rec: Collection of papers
[Github] Large Language Models for Generative Information Extraction: Awesome-LLM4IE-Papers
[Github] Large Language Models Meet NLP: Awesome-LLM-for-NLP
[Github] Knowledge graphs (KGs) and large language models (LLMs): KG-LLM-Papers
[Harvard] CS50 Tech Talk: GPT-4 - How does it work, and how do I build apps with it?
Important
AI prompt-to-storyboard videos w/ GPT, Coqui voices, StabilityAI images
Web LLM runs the vicuna-7b Large Language Model entirely in your browser
[Paper] AI Agents That Matter
In Defense of RAG in the Era of Long-Context Language Models
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering
Must Read Papers¶
Tag |
Title |
|---|---|
Attention |
MHA: Attention Is All You Need |
Attention |
MQA: Fast Transformer Decoding: One Write-Head is All You Need |
Attention |
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints |
Attention |
Attention is Turing Complete |
Decoding |
Fast Inference from Transformers via Speculative Decoding |
Activation |
GLU Variants Improve Transformer |
Norm |
Layer Normalization |
Norm |
Root Mean Square Layer Normalization |
PE |
RoFormer: Enhanced Transformer with Rotary Position Embedding |
MLM |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding |
MLM |
RoBERTa: A Robustly Optimized BERT Pretraining Approach |
MLM |
TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval |
MLM |
Finetuned Language Models Are Zero-Shot Learners |
RTD |
Electra: Pre-training Text Encoders as Discriminators Rather Than Generators |
CLM |
GPT2: Language Models are Unsupervised Multitask Learners |
CLM |
GPT3: Language Models are Few-Shot Learners |
CLM |
LLaMA: Open and Efficient Foundation Language Models |
CLM |
LLaMA 2: Open Foundation and Fine-Tuned Chat Models |
CLM |
LLaMA 3: The Llama 3 Herd of Models |
MoE |
Mixtral: Mixtral of Experts |
MoE |
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning |
PLM |
XLNet: Generalized Autoregressive Pretraining for Language Understanding |
GLM |
GLM: General Language Model Pretraining with Autoregressive Blank Infilling |
MoE |
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity |
MoE |
OLMoE: Open Mixture-of-Experts Language Models |
Seq2Seq |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation Translation and Comprehension |
Seq2Seq |
T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |
Multilingual |
XLM: Cross-lingual Language Model Pretraining |
Multilingual |
XLM-R: Unsupervised Cross-lingual Representation Learning at Scale |
Multilingual |
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer |
Generalisation |
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets |
Scaling |
Scaling Laws for Neural Language Models |
Scaling |
Scaling Laws for Autoregressive Generative Modeling |
Scaling |
Scaling Laws for Data Filtering – Data Curation cannot be Compute Agnostic |
Contrastive |
E5: Text Embeddings by Weakly-Supervised Contrastive Pre-training |
Contrastive |
Unsupervised Dense Information Retrieval with Contrastive Learning |
IR |
Dense Passage Retrieval for Open-Domain Question Answering |
IE:NER |
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition |
LLM |
Aligning language models to follow instructions |
LLM |
Scaling Instruction-Finetuned Language Models |
LLM |
InstructGpt: Training language models to follow instructions with human feedback |
LLM |
Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning |
LLM |
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions |
LLM |
Self-instruct: Aligning language model with self generated instructions |
LLM |
PPO: Proximal Policy Optimization Algorithms |
LLM |
SFT+RLHF: Learning to summarize from human feedback |
LLM |
Reflexion: Language Agents with Verbal Reinforcement Learning |
LLM |
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment |
LLM |
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes |
LLM |
DPO: Direct Preference Optimization: Your Language Model is Secretly a Reward Model |
LLM |
Understanding Reference Policies in Direct Preference Optimization |
LLM |
D2PO: Discriminator-Guided DPO with Response Evaluation Models |
LLM |
Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators |
LLM |
RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks |
LLM |
Large Language Models Are Latent Variable Models |
LLM |
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models |
Quant |
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale |
Quant |
LoRA Low-Rank Adaptation of Large Language Models |
Quant |
QLORA: Efficient Finetuning of Quantized LLMs |
Quant |
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models |
DiT |
Scalable Diffusion Models with Transformers |
DiT |
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis |
ViT |
Patch n’ Pack: NaViT - a Vision Transformer for any Aspect Ratio and Resolution |
ViT |
Long Context Transfer from Language to Vision |
Multimodal |
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model |
Eval |
HELM: Holistic Evaluation of Language Models |
Eval |
MMLU: Measuring Massive Multitask Language Understanding |
Eval |
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI |
Hallucination |
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models |
Hallucination |
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment |
Hallucination |
To Believe or Not to Believe Your LLM |
Representation |
Scaling and evaluating sparse autoencoders |
Representation |
Probabilistic Topic Modelling with Transformer Representations |
Representation |
Matryoshka Representation Learning |
Representation |
Not All Language Model Features Are Linear |
Context: Full |
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness |
Context: Full |
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning |
Context: Full |
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision |
Context: Full |
Effective Long-Context Scaling of Foundation Models |
Context: Sparse |
Longformer: The Long-Document Transformer |
Context: Sparse |
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context |
Context |
Ring Attention with Blockwise Transformers for Near-Infinite Context |
Context |
Lost in the Middle: How Language Models Use Long Contexts |
Long Context |
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities |
Long Context |
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models |
Long Context |
YaRN: Efficient Context Window Extension of Large Language Models |
Long Context |
Data Engineering for Scaling Language Models to 128K Context |
Long Context |
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention |
Long Context |
LongAlign: A Recipe for Long Context Alignment of Large Language Models |
Long Context |
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks |
Memory |
MemoryBank: Enhancing Large Language Models with Long-Term Memory |
Memory |
Augmenting Language Models with Long-Term Memory |
Memory |
Recurrent Memory Transformer |
Memory |
Scaling Transformer to 1M tokens and beyond with RMT |
Memory |
Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory |
KG |
Language Models as Knowledge Bases? |
KG |
Language Models are Open Knowledge Graphs |
KG |
Unifying Large Language Models and Knowledge Graphs: A Roadmap |
KG |
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering |
KG |
SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models |
KG |
Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling |
KG |
Knowledge Graph Large Language Model (KG-LLM) for Link Prediction |
Math¶
[github.com] rangaeeeee/books-mir-mathematics
[3Blue1Brown] Essence of linear algebra
[3Blue1Brown] Essence of calculus
[cs.princeton.edu] High Dimensional Geometry, Curse of Dimensionality, Dimension Reduction
[Khan Academy] Multivariable calculus
[University of Victoria] MATH200: Calculus III: Multivariable Calculus
[statisticsmatt] Introduction to Mathematical Statistics with Illustrations using R
[phys.org] Mathematician uncovers methods to shrink sampling errors in large-dimensional data sets
[math.cornell.edu] Linear ODE
ML Theory¶
[Goodfellow] Deep Learning
[Dong] Deep Reinforcement Learning
[Roberts] The Principles of Deep Learning Theory
[Kevin Murphy] Probabilistic Machine Learning book1
[Kevin Murphy] Probabilistic Machine Learning book2
[Bronstein,Bruna,Cohen,Veickovic][2021] Geometric Deep Learning
[Shwartz David] Understanding Machine Learning - From Theory to Algorithms
[Mohri] Foundations of Machine Learning
[CMU] 11-785 Deep Learning
ML Practical¶
[Andrej Karpathy] Neural Networks: Zero to Hero
[Stevens] Deep Learning with PyTorch
[Geron] Hands-on Machine Learning
[Zheng Casari] Feature Engineering for Machine Learning
[CMU] Dive into Deep Learning
[Microsoft Research] LMOps
ML Design General Principle¶
[Andrew Ng] Machine Learning Yearning
[Chip Huyen] Designing Machine Learning Systems
[Burkov] Machine Learning Engineering
[Jeff Smith] Machine Learning Systems
[Lakshmanan] Machine Learning Design Patterns
[UCB] System Design for Large Scale Machine Learning
ML Math¶
[Gutmann] Pen and Paper Exercise in ML
ML Algorithms¶
[Naumann] The Art of Differentiating Computer Programs
Applied ML¶
[Liu] Learning to Rank for Information Retrieval
[MSR] A Short Introduction to Learning to Rank
[MSR] LambdaMART
[Ravichandiran] Getting Started with Google BERT
[101ai.net] BERT Explorer
[Rothman] Transformers for Natural Language Processing
[Tunstall] Natural Language Processing with Transformers
[lilianweng] The Transformer Family Version 2.0
[Lakshmanan] Practical Machine Learning for Computer Vision
Recent Advances and Trends in Multimodal Deep Learning
Recommender Systems
[Stanford] CS224n: Natural Language Processing with Deep Learning
[Stanford] CS224U - Natural Language Understanding
[Stanford] CS25 - Transformers United
[Stanford] CS330 - Deep Multi-Task and Meta-Learning
[CMU] Graham Neubig’s Teaching
[Princeton] Against Predictive Optimization
[Github] Must Read Papers on Pre-Training
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
ML Papers¶
[dair-ai] ML-Papers-Explained
Transformer models: an introduction and catalog — 2023 Edition
[Microsoft Research] Why Can GPT Learn In-Context?
[ICLR2023] Diffusion Models already have a Semantic Latent Space
MLE Papers¶
[ACM] DNN for YouTube Recommendations
[FB] Local Search
[FB] Photo Search
[FB] Recommeding items to more than a billion people
[ICML] ScaNN
[NeurIPS] DiskANN
[KDD] Predicting Clicks on Ads at Facebook
[RecSys] Recommending What Video to Watch Next
ML Interviews¶
[Kashan] Deep Learning Interviews
System Design General Principles¶
[Kleppmann] Designing Data-Intensive Applications
[Alex Xu] System Design Interview - An Insiders Guide
[Alex Xu] System Design Interview - An Insider’s Guide Volume 2
[Donne Matrin] System Design Primer
[Binh Nguyen] Awesome Scalability
[Educative] Grokking Modern System Design Interview for Engineers & Managers
System Design Algorithms¶
[Gakhov] Probabilistic Data Structures and Algorithms
[Tyler Neylon] Introduction to Locality-Sensitive Hashing
System Design Practical¶
Layoffs¶
Misc¶
Fully Dynamic k-Clustering with Fast Update Time and Small Recourse
[Developer-Y] CS Video Courses
[CMU] 15-751 CS Theory Toolkit
Introducing Austral: A Systems Language with Linear Types and Capabilities
The Embedding Archives: Millions of Wikipedia Article Embeddings in Many Languages
Why Oatmeal is Cheap: Kolmogorov Complexity and Procedural Generation
[readthedocs.io] Sphinx Admonitions