Hugging Face's logo

Spaces:
sail
/
zero-bubble-pipeline-parallellism
Running

App Files Community
zero-bubble-pipeline-parallellism / README.md
Wan Xinyi
Add links to README
f8e95f6
preview code
|
raw history blame contribute delete
No virus
905 Bytes
metadata
title: Zero Bubble Pipeline Parallellism
emoji: 🏆
colorFrom: indigo
colorTo: red
sdk: gradio
sdk_version: 4.7.1
app_file: app.py
pinned: false
license: apache-2.0

Zero Bubble Pipeline Parallelism

Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.

Check out our paper at:

Try out our implementation based on Megatron on https://github.com/sail-sg/zero-bubble-pipeline-parallelism

Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.