Models in Depth

Courses

• [CMU] 15-442/15-642: Machine Learning Systems

• [CMU] Deep Learning Systems - Algorithms and Implementation

• [Stanford] CS149 Parallel Computing

Books

• [Reddi] Machine Learning Systems

AutoGrad Engine

• [cs231n.stanford.edu] Lecture 6: Hardware and Software

• [github.com] karpathy/micrograd

• [pytorch.org] Overview of PyTorch Autograd Engine

• [pytorch.org] How Computational Graphs are Constructed in PyTorch

• [pytorch.org] How Computational Graphs are Executed in PyTorch

PyTorch Internals

• [blog.ezyang.com] PyTorch internals

• [medium.com] A Trip to Kernels: Understanding PyTorch’s Internal Architecture

Scaling

• [jax-ml.github.io] How to Scale Your Model

• [huggingface.co] The Ultra-Scale Playbook: Training LLMs on GPU Clusters