Update README.md

README.md

@@ -1,41 +1,82 @@
-Model Info (Interal):
-- Size: 7B
-- Dataset: The Pile v2
-  - `contaminated(P3) + lower_code(5%) + wiki(fixed) + books3(fixed & broken)`
-- Batch size (in tokens): 8M
-- Checkpoint Step: 69,000 (552B tokens)
-- Checkpoint path (AWS East): `/fsx/ckpts/7b_tok=neox_data=pilev2-recontam_lower-code_bs=8m_tp=4_pp=1_init=wang-small-init/global_step69000_hf`
+---
+language:
+- en
+tags:
+- causal-lm
+license: cc-by-nc-4.0
+---
 
-Notes:
-- Trained for 36k steps with incorrectly tokenized Books3 dataset (GPT-2 tokenizer instead of NeoX tokenizer)
-- tp=2 (not 4)
+# StableLM-Base-Alpha
 
-W&B Report: https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-7B-alpha---Vmlldzo2MjA
+## Model Description
 
-Usage:
+`StableLM-Base-Alpha` is a suite of 3B and 7B parameter decoder-only language models pre-trained on a diverse collection of English datasets with a sequence length of 4096 to push beyond the context window limitations of existing open-source language models.
+
+## Usage
+
+Get started generating text with `StableLM-Base-Alpha` by using the following code snippet:
 
 ```python
-import transformers
-
-model = transformers.AutoModelForCausalLM.from_pretrained("CarperAI/7b-alpha")
-tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
-tokenizer.pad_token = tokenizer.eos_token
-tokenizer.paddding_side = "left"
-
-prompts = [
-    "User1: The dog sat on a man's lap and barked 3 times.\nUser2: How many times did the dog bark?"
-    "Curious Person Question: A group of genetically identical individuals is called what?\nSmart Person Answer: a clone\n\nCurious Person Question: Who proposed the theory of evolution by natural selection?\nSmart Person Answer:"
-]
-batch_encoding = tokenizer(prompts, return_tensors="pt", padding=True)
-
-print(f"Generating {len(prompts)} prompts...")
-samples = model.generate(
-    **batch_encoding,
-    max_new_tokens=64,
-    temperature=0.0,
-    do_sample=False,
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
+model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
+model.half().cuda()
+
+inputs = tokenizer("What's your mood today?", return_tensors="pt")
+tokens = model.generate(
+  **inputs,
+  max_new_tokens=64,
+  temperature=0.7,
+  do_sample=True,
 )
-samples = tokenizer.batch_decode(samples, skip_special_tokens=True)
-for prompt, sample in zip(prompts, samples):
-    print(f"Prompt: {prompt}\nSample: {sample}\n")
-```
+print(tokenizer.decode(tokens[0], skip_special_tokens=True))
+```
+
+## Model Details
+
+* **Developed by**: [Stability AI](https://stability.ai/)
+* **Model type**: StableLM-Base-Alpha models are auto-regressive language models based on the NeoX transformer architecture.
+* **Language(s)**: English
+* **Library**: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
+* **License**: [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)
+* **Contact**: For questions and comments about the model, please email **{TODO: FILL IN CONTACT ADDRESS}**
+
+## Training
+
+| Parameters | Hidden Size | Layers | Heads | Sequence Length |
+|------------|-------------|--------|-------|-----------------|
+| 3B         | 4096        | 16     | 32    | 4096            |
+| 7B         | 6144        | 16     | 48    | 4096            |
+
+### Training Dataset
+
+`StableLM-Base-Alpha` is pre-trained on a new experimental dataset built atop [The Pile](https://huggingface.co/datasets/EleutherAI/the_pile) and is threes times larger at approximately 1.5T tokens.
+
+### Training Procedure
+
+Models are pre-trained on the aforementioned dataset in mixed-precision (FP16) and optimized with Adam. We outline the complete hyperparameters choices in the project's GitHub repository **{TODO: FILL IN LINK}**.
+
+## Use and Limitations
+
+### Intended Use
+
+These models are intended to be used by all individuals as foundational models for application-specific fine-tuning without strict limitations on commercial use.
+
+### Limitations and bias
+
+The pre-training dataset may contain offensive or inappropriate content even after applying data cleansing filters which can be reflected in generated text. We recommend users exercise reasonable caution when using these models in production systems. Do not use the models for any applications that may cause harm or distress to individuals or groups.
+
+## Citations
+
+```bibtext
+@software{gpt-neox-library,
+  title = {{GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch}},
+  author = {Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel},
+  url = {https://www.github.com/eleutherai/gpt-neox},
+  doi = {10.5281/zenodo.5879544},
+  month = {8},
+  year = {2021},
+  version = {0.0.1},
+}
+```