Communication Overlap

ASYRA: Automating Graph Scheduling for Communication-Computation Overlap in Efficient Model Parallelism

Scaling large models requires complex multi-dimensional (n-D) parallelism, yet this paradigm suffers from severe communication bubbles …

Lei Zhang, Zhisheng YE