LLM Training

General

• The Llama 3 Herd of Models(https://arxiv.org/abs/2407.21783)
• TorchScale - A Library for Transformers at (Any) Scale(https://github.com/microsoft/torchscale)
• DLRover: An Automatic Distributed Deep Learning System(https://github.com/intelligent-machine-learning/dlrover)

2024

• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision(https://tridao.me/blog/2024/flash3/)
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs(https://arxiv.org/abs/2402.15627)
• ByteCheckpoint: A Unified Checkpointing System for LLM Development(https://arxiv.org/abs/2407.20143)