|
--- |
|
library_name: peft |
|
tags: |
|
- tiiuae |
|
- code |
|
- instruct |
|
- databricks-dolly-15k |
|
- falcon-40b |
|
datasets: |
|
- databricks/databricks-dolly-15k |
|
base_model: tiiuae/falcon-40b |
|
--- |
|
|
|
For our finetuning process, we utilized the tiiuae/falcon-40b model and the Databricks-dolly-15k dataset. |
|
|
|
This dataset, a meticulous compilation of over 15,000 records, was a result of the dedicated work of thousands of Databricks professionals. It was specifically designed to further improve the interactive capabilities of ChatGPT-like systems. |
|
The dataset contributors crafted prompt / response pairs across eight distinct instruction categories. Besides the seven categories mentioned in the InstructGPT paper, they also ventured into an open-ended, free-form category. The contributors, emphasizing genuine and original content, refrained from sourcing information online, except in special cases where Wikipedia was the source for certain instruction categories. There was also a strict directive against the use of generative AI for crafting instructions or responses. |
|
The contributors could address questions from their peers. Rephrasing the original question was encouraged, and there was a clear preference to answer only those queries they were certain about. |
|
In some categories, the data comes with reference texts sourced from Wikipedia. Users might find bracketed Wikipedia citation numbers (like [42]) within the context field of the dataset. For smoother downstream applications, it's advisable to exclude these. |
|
|
|
Our finetuning was conducted using the [MonsterAPI](https://monsterapi.ai)'s intuitive, no-code [LLM finetuner](https://docs.monsterapi.ai/fine-tune-a-large-language-model-llm). |
|
|
|
Highlighting the cost-effectiveness and efficiency of the process, |
|
the entire session was finished in just 5 hours and 40 minutes, leveraging an A6000 48GB GPU. |
|
The total cost for this efficient run was a mere `$11.8`. |
|
|
|
#### Hyperparameters & Run details: |
|
- Epochs: 1 |
|
- Cost: $11.8 |
|
- Model Path: tiiuae/falcon-40b |
|
- Dataset: databricks/databricks-dolly-15k |
|
- Learning rate: 0.0002 |
|
- Data split: Training 90% / Validation 10% |
|
- Gradient accumulation steps: 4 |
|
|
|
license: apache-2.0 |
|
--- |
|
|
|
###### |
|
|
|
Prompt Used: |
|
|
|
### INSTRUCTION: |
|
[instruction] |
|
|
|
[context] |
|
|
|
### RESPONSE: |
|
[response] |
|
|
|
Loss metrics |
|
|
|
Training loss (Blue) Validation Loss (orange): |
|
![training loss](train-loss.png "Training loss") |
|
|