metadata
license: apache-2.0
library_name: transformers
tags:
- juanako
- UNA
datasets:
- fblgit/tree-of-knowledge
- Open-Orca/SlimOrca-Dedup
- HuggingFaceH4/ultrafeedback_binarized
model-index:
- name: una-cybertron-7b-v1-fp16
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 68.43
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 85.42
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 63.34
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 63.28
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 81.37
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 55.12
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/una-cybertron-7b-v1-fp16
name: Open LLM Leaderboard
Model Card for una-cybertron-7b-v1 (UNA: Uniform Neural Alignment)
We strike back, introducing Cybertron 7B v1 a 7B MistralAI based model, best on it's series. Trained on SFT, DPO and UNA (Unified Neural Alignment) on multiple datasets. He scores 64.60+ on HF LeaderTests (without DROP for now).
Scoring #1 at 2 December 2023:
| Model | Average | ARC (25-s) | HellaSwag (10-s) | MMLU (5-s) | TruthfulQA (MC) (0-s) | Winogrande (5-s) | GSM8K (5-s) |
|---|---|---|---|---|---|---|---|
| mistralai/Mistral-7B-v0.1 | 60.97 | 59.98 | 83.31 | 64.16 | 42.15 | 78.37 | 37.83 |
| perlthoughts/Chupacabra-7B-v2 | 63.54 | 66.47 | 85.17 | 64.49 | 57.6 | 79.16 | 28.35 |
| fblgit/una-cybertron-7b-v1 | 64.60 | 68.17 | 85.14 | 62.07 | 63.98 | 80.9 | 27.34 |
The model excels in mathematics, logic, reasoning, overall very smart.
Model Details
Adiestrated with UNA: Uniform Neural Alignment technique (paper going out soon).
Model Description
- Developed by: juanako.ai
- Author: Xavier M.
- Model type: MistralAI 7B
- Funded by Cybertron's H100's
Prompt
The model is very good, works well on almost any prompt but ChatML format and Alpaca System gets the best
<|im_start|>system
- You are a helpful assistant chatbot trained by MosaicML.
- You answer questions.
- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.<|im_end|>
<|im_start|>user
Explain QKV<|im_end|>
<|im_start|>assistant
### Assistant: I am StableVicuna, a large language model created by CarperAI. I am here to chat!
### Human: Explain QKV
### Assistant:
[Round <|round|>]
问：Explain QKV
答：
[Round <|round|>]
Question：Explain QKV
Answer：
Question：Explain QKV
Answer：
Evaluation
| Tasks |Version|Shots | Metric |Value | |Stderr|
|--------------|-------|------|--------|-----:|---|-----:|
|arc_challenge | | 25 |acc_norm|0.6817|± |0.0136|
|truthfulqa_mc2| | 0 |acc |0.6398|± |0.0151|
|hellaswag | | 10 |acc_norm|0.8492|± |0.0036|
|winogrande | | 0 |acc |0.809 |± |0.011 |
|gsm8k | | 5 |acc |0.2733|± |0.0137|
|mmlu | | 5 |acc |0.6207|± |0.1230|
| |average| |acc |0.6456| | |
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.6207|_ |0.1230|
| - humanities |N/A |none | 5|acc |0.5675|_ |0.1125|
| - other |N/A |none | 5|acc |0.6933|_ |0.1108|
| - social_sciences|N/A |none | 5|acc |0.7270|_ |0.0666|
| - stem |N/A |none | 5|acc |0.5249|_ |0.1311|
Framework versions
- Transformers 4.35.0-UNA
- Pytorch 2.1.0
- Datasets 2.14.6
- Tokenizers 0.14.1
Citations
If you find Cybertron, Juanako or any of our models useful, specially if you use it for your big brand.. cite please:
@misc{unacybertron7a,
title={Cybertron: Uniform Neural Alignment},
author={Xavier Murias},
year={2023},
publisher = {HuggingFace},
journal = {HuggingFace repository},
howpublished = {\url{https://huggingface.co/fblgit/una-cybertron-7b-v1}},
}
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 69.49 |
| AI2 Reasoning Challenge (25-Shot) | 68.43 |
| HellaSwag (10-Shot) | 85.42 |
| MMLU (5-Shot) | 63.34 |
| TruthfulQA (0-shot) | 63.28 |
| Winogrande (5-shot) | 81.37 |
| GSM8k (5-shot) | 55.12 |