9 itemsv1
Distributed Training
- 88 papers
- Horovod: Fast and Easy Distributed Deep Learning in TensorFlow
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
- GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
- ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
- PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- 11 docs
- Large Scale Distributed Deep Networks
Data, model, and pipeline parallelism for multi-GPU training without a paper mountain.
