license: bsd-3-clause
starcoder-toolbench
starcoder-toolbench is a 15 billion parameter model used for api based action generation. It is instruction tuned from starcoder on api based action generation datasets.
Model Details
Model Description
- Developed by: SambaNova Systems
- Model type: Language Model
- Language(s): English
- License:
- Finetuned from model: starcoder
Basic Information
- Paper: [Link]
- Github: [Link]
Licensing
TBD
Uses
Click to expand
Direct Use
This model is intended for commercial and research use.
Out-of-Scope Use
starcoder-toolbench should NOT be used for purpose other than API based action generation.
Recommendations
Users should be made aware of the risks, biases, limitations, and restrictions of the model, which are listed down at the bottom of the page.
How to Get Started with the Model
Click to expand
Loading in model with Huggingface
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/starcoder-toolbench")
model = AutoModelForCausalLM.from_pretrained("sambanovasystems/starcoder-toolbench", device_map="auto", torch_dtype="auto")
Training Details
Click to expand
Training Data
The training data is curated for the 8 tasks in ToolBench. See Appendix A of the [paper](dummy link) for task details and Appendix C.1 for the training data curation details. In total, there are 9704 training samples, organized in all-shot format as described in Appendix C.2. Here is the download link to the training data.
Training Procedure
We trained starcoder-toolbench on 4 80GB A100 gpu's. We started from starcoder. We finetuned it on XXX dataset. All of the code used to prepare the datasets and the scripts to run training and inference are open-sourced and freely available at [githublink here](dummy link)
Prompting Style Used For Training
Hyperparameters
- Hardware: A100 GPU
- Optimizer: AdamW
- Grad accumulation: 1
- Epochs: 8
- Global Batch size: 16
- Batch tokens: 16 * 2048 = 32,768 tokens
- Learning Rate: 1e-5
- Learning Rate Scheduler: Fixed LR
- Weight decay: 0.1
Acknowledgment
Cite starcoder-toolbench
@software{bloomchat,
title = {{BLOOMChat: a New Open Multilingual Chat LLM}},
author = {SambaNova Systems, Together Computer},
url = {https://huggingface.co/sambanovasystems/BLOOMChat-176B-v1}
month = {5},
year = {2023},
version = {1.0},
}