|
--- |
|
language: |
|
- en |
|
library_name: transformers |
|
datasets: |
|
- psmathur/lima_unchained_v1 |
|
license: llama2 |
|
--- |
|
|
|
# Lima_Unchained_70b |
|
|
|
A Llama2-70b model fine-tuned using QLora on all the linear layers with carefully selected ~900 conversations from the [Lima](https://arxiv.org/pdf/2305.11206.pdf) |
|
|
|
<br> |
|
|
|
|
|
**P.S. If you're interested to collaborate, please connect with me at www.linkedin.com/in/pankajam.** |
|
|
|
## Evaluation |
|
|
|
We evaluated Lima_Unchained_70b on a wide range of tasks using [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) from EleutherAI. |
|
|
|
Here are the results on metrics used by [HuggingFaceH4 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
|
|
||||| |
|
|:------:|:--------:|:-------:|:--------:| |
|
|**Task**|**Metric**|**Value**|**Stderr**| |
|
|*arc_challenge*|acc_norm|0.6826|0.0141| |
|
|*hellaswag*|acc_norm|0.8765|0.0038| |
|
|*mmlu*|acc_norm|0.70|0.0351| |
|
|*truthfulqa_mc*|mc2|0.4876|0.0157| |
|
|**Total Average**|-|**0.6867**|| |
|
|
|
|
|
<br> |
|
|
|
## Example Usage |
|
|
|
Here is the prompt format |
|
|
|
``` |
|
|
|
### User: |
|
Write a stand-up skit in the style of George Carlin that ridicules Pacific Gas and Electric. |
|
|
|
### Assistant: |
|
|
|
``` |
|
|
|
Below shows a code example on how to use this model |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline |
|
|
|
model_path="pankajmathur/Lima_Unchained_70b" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_path, |
|
torch_dtype=torch.float16, |
|
load_in_8bit=True, |
|
low_cpu_mem_usage=True, |
|
device_map="auto" |
|
) |
|
|
|
#generate text steps |
|
instruction = "Write a stand-up skit in the style of George Carlin that ridicules Pacific Gas and Electric." |
|
prompt = f"### User: {instruction}\n\n### Assistant:\n" |
|
inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
|
output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=4096) |
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True)) |
|
|
|
``` |
|
|
|
<br> |
|
|
|
#### Limitations & Biases: |
|
|
|
While this model aims for accuracy, it can occasionally produce inaccurate or misleading results. |
|
|
|
Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content. |
|
|
|
Exercise caution and cross-check information when necessary. |
|
|
|
|
|
<br> |
|
|
|
### Citiation: |
|
|
|
Please kindly cite using the following BibTeX: |
|
|
|
``` |
|
@misc{Lima_Unchained_70b, |
|
author = {Pankaj Mathur}, |
|
title = {Lima_Unchained_70b: A LIMA style Llama2-70b model}, |
|
year = {2023}, |
|
publisher = {HuggingFace}, |
|
journal = {HuggingFace repository}, |
|
howpublished = {\url{https://https://huggingface.co/psmathur/model_42_70b}, |
|
} |
|
``` |
|
|
|
``` |
|
@misc{ChuntingZhou, |
|
title={LIMA: Less Is More for Alignment}, |
|
author={Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, |
|
Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy}, |
|
year={2023}, |
|
eprint={2305.11206}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
|
``` |
|
@software{touvron2023llama2, |
|
title={Llama 2: Open Foundation and Fine-Tuned Chat Models}, |
|
author={Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, |
|
Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, |
|
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann, |
|
Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, |
|
Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, |
|
Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu , Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, |
|
Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom}, |
|
year={2023} |
|
} |
|
``` |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_psmathur__test_42_70b) |
|
|
|
| Metric | Value | |
|
|-----------------------|---------------------------| |
|
| Avg. | 59.77 | |
|
| ARC (25-shot) | 68.26 | |
|
| HellaSwag (10-shot) | 87.65 | |
|
| MMLU (5-shot) | 70.0 | |
|
| TruthfulQA (0-shot) | 48.76 | |
|
| Winogrande (5-shot) | 83.66 | |
|
| GSM8K (5-shot) | 45.94 | |
|
| DROP (3-shot) | 14.09 | |
|
|