semvec/pypi-packages huggingface dataset:
fastckpt 0.0.4
A fast gradient checkpointing strategy for training with memory-efficient attention (e.g., FlashAttention).
Requires Python: >=3.8

# FastCkpt: accelerate your LLM training in one line!

Fast gradient checkpoint is designed for accelerate the training with memory-efficient attention like FlashAttention and LightSeq. FastCkpt has monkey patch for both rematerialization-aware checkpointing and FlashAttention, so you can patch both in only one line!

Paper: https://arxiv.org/pdf/2310.03294.pdf

## News
- [2023/10] FastCkpt now supports LlamaModel in Huggingface!

## Install
```bash
pip install fastckpt
```

## Usage
FastCkpt now supports HF training pipeline. 

### Use FaskCkpt and FlashAttention
To use `fasckpt` with `flash_attn`, import and run `replace_hf_ckpt_with_fast_ckpt` *before* importing `transformers`
```python
# add monkey patch for fastckpt
from fastckpt.llama_flash_attn_ckpt_monkey_patch import replace_hf_ckpt_with_fast_ckpt
replace_llama_attn_with_flash_attn()

# import transformers and other packages
import transformers
...
```

### Use FlashAttention only
To only replace the `LlamaAttention` with `flash_attn` without chaning the checkpointing strategy, import and run `replace_llama_attn_with_flash_attn`

```python
# add monkey patch for fastckpt
from fastckpt.llama_flash_attn_monkey_patch import replace_llama_attn_with_flash_attn
replace_llama_attn_with_flash_attn()

# import transformers and other packages
import transformers
...
```