|
--- |
|
license: other |
|
language: |
|
- en |
|
tags: |
|
- causal-lm |
|
- code |
|
metrics: |
|
- code_eval |
|
library_name: transformers |
|
model-index: |
|
- name: stabilityai/stable-code-instruct-3b |
|
results: |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: nuprl/MultiPL-E |
|
name: MultiPL-HumanEval (Python) |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 32.4 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: nuprl/MultiPL-E |
|
name: MultiPL-HumanEval (C++) |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 30.9 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: nuprl/MultiPL-E |
|
name: MultiPL-HumanEval (Java) |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 32.1 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: nuprl/MultiPL-E |
|
name: MultiPL-HumanEval (JavaScript) |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 32.1 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: nuprl/MultiPL-E |
|
name: MultiPL-HumanEval (PHP) |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 24.2 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: nuprl/MultiPL-E |
|
name: MultiPL-HumanEval (Rust) |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 23.0 |
|
verified: false |
|
--- |
|
|
|
|
|
[Try it out here: https://huggingface.co/spaces/stabilityai/stable-code-instruct-3b](https://huggingface.co/spaces/stabilityai/stable-code-instruct-3b) |
|
|
|
 |
|
|
|
|
|
|
|
`stable-code-instruct-3b` is a 2.7B billion parameter decoder-only language model tuned from [`stable-code-3b`](https://huggingface.co/stabilityai/stable-code-3b/). This model was trained on a mix of publicly available datasets, synthetic datasets using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). |
|
|
|
This instruct tune demonstrates state-of-the-art performance (compared to models of similar size) on the MultiPL-E metrics across multiple programming languages tested using [BigCode's Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main), and on the code portions of |
|
[MT Bench](https://klu.ai/glossary/mt-bench-eval). |
|
The model is finetuned to make it useable in tasks like, |
|
- General purpose Code/Software Engineering like conversations. |
|
- SQL related generation and conversation. |
|
|
|
|
|
|
|
Here's how you can run the model use the model: |
|
|
|
```python |
|
|
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-instruct-3b", trust_remote_code=True) |
|
model = AutoModelForCausalLM.from_pretrained("stabilityai/stable-code-instruct-3b", torch_dtype=torch.bfloat16, trust_remote_code=True) |
|
model.eval() |
|
model = model.cuda() |
|
|
|
messages = [ |
|
{ |
|
"role": "system", |
|
"content": "You are a helpful and polite assistant", |
|
}, |
|
{ |
|
"role": "user", |
|
"content": "Write a simple website in HTML. When a user clicks the button, it shows a random joke from a list of 4 jokes." |
|
}, |
|
] |
|
|
|
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
|
|
|
inputs = tokenizer([prompt], return_tensors="pt").to(model.device) |
|
|
|
tokens = model.generate( |
|
**inputs, |
|
max_new_tokens=1024, |
|
temperature=0.5, |
|
top_p=0.95, |
|
top_k=100, |
|
do_sample=True, |
|
use_cache=True |
|
) |
|
|
|
output = tokenizer.batch_decode(tokens[:, inputs.input_ids.shape[-1]:], skip_special_tokens=False)[0] |
|
``` |
|
|
|
|
|
|
|
* **Developed by**: [Stability AI](https://stability.ai/) |
|
* **Model type**: `Stable Code Instruct 3B` model is an auto-regressive language model based on the transformer decoder architecture. |
|
* **Language(s)**: English |
|
* **Paper**: [Stable Code Technical Report](https://drive.google.com/file/d/16-DGsR5-qwoPztZ6HcM7KSRUxIXrjlSm/view) |
|
* **Library**: [Alignment Handbook](https://github.com/huggingface/alignment-handbook.git) |
|
* **Finetuned from model**: [https://huggingface.co/stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b) |
|
* **License**: [StabilityAI Non-Commercial Research Community License](https://huggingface.co/stabilityai/stable-code-instruct-3b/blob/main/LICENSE). If you want to use this model for your commercial products or purposes, please contact us [here](https://stability.ai/contact) to learn more. |
|
* **Contact**: For questions and comments about the model, please email `lm@stability.ai` |
|
|
|
|
|
|
|
|
|
| Model | Size | Avg | Python | C++ | JavaScript | Java | PHP | Rust | |
|
|------------------------------|------|------|--------|------|------------|------|------|------| |
|
| Codellama Instruct | 7B | 0.30 | 0.33 | 0.31 | 0.31 | 0.29 | 0.31 | 0.25 | |
|
| Deepseek Instruct | 1.3B | 0.44 | 0.52 | **0.52** | 0.41 | **0.46** | 0.45 | 0.28 | |
|
| Stable Code Instruct (SFT) | 3B | 0.44 | 0.55 | 0.45 | 0.42 | 0.42 | 0.44 | 0.32 | |
|
| Stable Code Instruct (DPO) | 3B | **0.47** | **0.59** | 0.49 | **0.49** | 0.44 | **0.45** | **0.37** | |
|
|
|
|
|
| Model | Size | Score | |
|
|-----------------------------|------|-----------------| |
|
| DeepSeek Coder | 1.3B | 4.6 | |
|
| Stable Code Instruct (DPO) | 3B | **5.8**(ours) | |
|
| Stable Code Instruct (SFT) | 3B | 5.5 | |
|
| DeepSeek Coder | 6.7B | **6.9** | |
|
| CodeLlama Instruct | 7B | 3.55 | |
|
| StarChat2 | 15B | 5.7 | |
|
|
|
|
|
| Model | Size | Date | Group By | Order By | Ratio | Join | Where | |
|
|-----------------------------|------|-------|----------|----------|-------|-------|-------| |
|
| Stable Code Instruct (DPO) | 3B | 24.0% | 54.2% | 68.5% | 40.0% | 54.2% | 42.8% | |
|
| DeepSeek-Coder Instruct | 1.3B | 24.0% | 37.1% | 51.4% | 34.3% | 45.7% | 45.7% | |
|
| SQLCoder | 7B | 64.0% | 82.9% | 74.3% | 54.3% | 74.3% | 74.3% | |
|
|
|
|
|
|
|
|
|
|
|
```bibtex |
|
@misc{stable-code-instruct-3b, |
|
url={[https://huggingface.co/stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-instruct-3b)}, |
|
title={Stable Code 3B}, |
|
author={Phung, Duy, and Pinnaparaju, Nikhil and Adithyan, Reshinth and Zhuravinskyi, Maksym and Tow, Jonathan and Cooper, Nathan} |
|
} |
|
``` |
