Distributed Trainingv1i

Why?

Cleanest intro to ring-allreduce data parallelism. Start here. Ring-allreduce is the gradient sync you'll keep meeting; this paper makes it concrete.

Why?

How DDP actually works: gradient bucketing, overlap with the backward pass, and the ring. Worth reading if your smolcluster runs on PyTorch.

Why?

The original parameter-server paper. Old, but it is the baseline ring-allreduce and ZeRO were built to beat.

Why?

Partitions optimizer states, gradients, and parameters across data-parallel workers (ZeRO stages 1 to 3), so you stop keeping full copies on every GPU. Same math helps a small cluster train models it could not otherwise hold.

Why?

Introduces tensor parallelism: split weight matrices column-wise and row-wise across GPUs. The attention and MLP splits here show up in almost every production LLM cluster.

Why?

How to compose tensor, pipeline, and data parallelism on one cluster. Also introduces the interleaved pipeline schedule that cuts bubble overhead by about 10%. Read after ZeRO and Megatron-LM.

Why?

Pipeline parallelism in its basic form: split layers across devices and feed microbatches so the bubble does not sit idle. Later pipeline papers build on this model.

Why?

Extends ZeRO by offloading to CPU and NVMe. Useful when Jetson or Pi nodes are tight on memory: weights do not have to live on the GPU.

Why?

ZeRO-3 as it ships in PyTorch: the sharding layer you would actually call from smolcluster. Covers the engineering trade-offs of parameter sharding in practice.