Fault Tolerance

GPU 任务的暂停、恢复与迁移：调度器一直缺的那块拼图
一篇关于 GPU checkpoint/restore 的技术笔记：以 FlowGPU 为主线，介绍 cudaw prototype 如何探索透明的暂停、恢复与迁移。
Zhisheng YE
May 17, 2026
ResiHP: Surviving LLM Training Failures with Dynamic Hybrid Parallelism
A technical report on ResiHP, a resilient training system that detects fail-slow devices under noisy sequence-length variation and dynamically adapts 3D parallelism.
Zhisheng YE
May 17, 2026 3 min read
GPU Pause, Resume, and Migration: The Missing Primitive in Cluster Scheduling
A technical note on GPU checkpoint/restore for schedulers, using FlowGPU as the main reference and my cudaw prototype as the first version of the codebase.
Zhisheng YE
May 15, 2026 8 min read
FlowGPU: Transparent and Efficient GPU Checkpointing and Restore
GPU checkpointing and restore promises to enable emerging tasks, such as deep learning, to benefit from functionalities like task …
Zehua Yang, Xiao Zheng, Yonghao Zou, Junyang Zhang, Zhisheng YE, Feng Xie, Xiaolin Wang, Yingwei Luo, Zhenlin Wang, Diyu Zhou
PDF Cite DOI
ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual …
Tenghui Ma, Jihu Guo, Wei Gao, Sitian Lu, Zhisheng YE, Dahua Lin, Hanjing Wang
Preprint Cite DOI