ZeRO, ZeRO-Offload, ZeRO-Infinity
Paper reading of ZeRO.
Distributed Training
DeepSpeedUlysses
Paper reading of Deepseed Ulysses.
Efficient Large-Scale Language Model Training on GPU
Paper reading about Efficient Large-Scale Language Model Training on GPU Clusters.
Megatron-LM
Paper reading about Megatron-LM
Ring Attention Principle
This is a brief introduction to the Ring Attention Principle.
Comparsion of Parallelsim Metods in ViT
Paper reading of