|
--- |
|
license: other |
|
language: |
|
- en |
|
tags: |
|
- causal-lm |
|
--- |
|
# `Stable LM 2 1.6B` (global_step420000) |
|
|
|
## Description |
|
|
|
`Stable LM 2 1.6B` is a 1.6 billion parameter decoder-only language model pre-trained on 2 trillion tokens of diverse multilingual and code datasets for two epochs. |
|
|
|
## Usage |
|
|
|
This branch contains the training checkpoint for `Stable LM 2 1.6B` at step 420,000. It is the final checkpoint taken before cooldown. |
|
We provide the following contents in the [`global_step420000`](https://huggingface.co/stabilityai/stablelm-2-1_6b/tree/global_step420000/global_step420000) directory: |
|
|
|
- `bf16_zero_pp_mp_rank_00_optim_states.pt`: The Adam states and FP32 weights for each parameter. You will need to port this to your optimizer format when importing into your training process. |
|
|
|
- `mp_rank_00_model_states.pt`: The model weights following the [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) convention. |
|
|
|
- `config.yml`: The pre-training configuration file for this checkpoint. Linear learning rate cooldown should be taken from `lr=0.0002529` to `lr=0.0`. |
|
|
|
The model weights are also converted to HuggingFace `transformers` format and can be loaded with the following code: |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-2-1_6b", trust_remote_code=True) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
"stabilityai/stablelm-2-1_6b", |
|
trust_remote_code=True, |
|
torch_dtype="auto", |
|
revision="global_step420000" |
|
) |
|
model.cuda() |
|
``` |
|
|
|
## License |
|
|
|
* **License**: [Stability AI Non-Commercial Research Community License](https://huggingface.co/stabilityai/stablelm-2-1_6b/blob/main/LICENSE). If you'd like to use this model for commercial products or purposes, please contact us [here](https://stability.ai/membership) to learn more. |
|
|
|
## Acknowledgements |
|
|
|
- Dakota Mahan for creating the ZeRO optimizer state merging script. |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@misc{StableLM-2-1.6B, |
|
url={[https://huggingface.co/stabilityai/stablelm-2-1_6b](https://huggingface.co/stabilityai/stablelm-2-1_6b)}, |
|
title={Stable LM 2 1.6B}, |
|
author={Stability AI Language Team} |
|
} |
|
``` |
|
|
