Hugging Face's logo

Spaces:
sail
/
zero-bubble-pipeline-parallellism
Running

App Files Community
zero-bubble-pipeline-parallellism / description1.md
Wan Xinyi
Render schedule
4835f75
preview code
|
raw history blame
No virus
583 Bytes

Zero Bubble Pipeline Parallelism

Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.

Our paper is coming soon.

Try out our implementation based on Megatron on https://github.com/sail-sg/zero-bubble-pipeline-parallelism

Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.