|
--- |
|
language: |
|
- ja |
|
- en |
|
- zh |
|
license: apache-2.0 |
|
model-index: |
|
- name: laser-polyglot-4x7b |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: AI2 Reasoning Challenge (25-Shot) |
|
type: ai2_arc |
|
config: ARC-Challenge |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: acc_norm |
|
value: 64.16 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: HellaSwag (10-Shot) |
|
type: hellaswag |
|
split: validation |
|
args: |
|
num_few_shot: 10 |
|
metrics: |
|
- type: acc_norm |
|
value: 84.98 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU (5-Shot) |
|
type: cais/mmlu |
|
config: all |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 63.88 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: TruthfulQA (0-shot) |
|
type: truthful_qa |
|
config: multiple_choice |
|
split: validation |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: mc2 |
|
value: 55.47 |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: Winogrande (5-shot) |
|
type: winogrande |
|
config: winogrande_xl |
|
split: validation |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 77.82 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GSM8k (5-shot) |
|
type: gsm8k |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 48.45 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b |
|
name: Open LLM Leaderboard |
|
--- |
|
# Polyglot-4x7b-24b |
|
|
|
 |
|
|
|
Polyglot-4x7b is a Mixture of Experts approach to a multilingual model. |
|
|
|
This project is an experiment to see if each expert can be of a different language. The answer is yes. |
|
|
|
The model is a merge of models that are capable of Chinese and Japanese output. |
|
|
|
+ teknium/OpenHermes-2.5-Mistral-7B |
|
+ oshizo/japanese-e5-mistral-7b_slerp |
|
+ cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser |
|
+ s3nh/Mistral-7B-Evol-Instruct-Chinese |
|
|
|
TODO: |
|
1. [] polyglot tokenizer |
|
|
|
## Other polyglot models |
|
|
|
+ [macadeliccc/Polyglot-8x7b-v0.1](https://huggingface.co/macadeliccc/Polyglot-8x7b-v0.1) (adds 3 more languages) |
|
# Code Example |
|
|
|
Inference [Colab](https://colab.research.google.com/drive/1tYSb63IKZDsiQ5BIJU8Oc92phxugAmB3?usp=sharing) |
|
Live demo available on [Spaces](https://huggingface.co/spaces/macadeliccc/polyglot-4x7b-chat?logs=build) |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
def generate_response(prompt): |
|
""" |
|
Generate a response from the model based on the input prompt. |
|
|
|
Args: |
|
prompt (str): Prompt for the model. |
|
|
|
Returns: |
|
str: The generated response from the model. |
|
""" |
|
# Tokenize the input prompt |
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
|
# Generate output tokens |
|
outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id) |
|
|
|
# Decode the generated tokens to a string |
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
|
return response |
|
|
|
# Load the model and tokenizer |
|
model_id = "macadeliccc/laser-polyglot-4x7b" |
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True) |
|
|
|
# Example prompts in different languages |
|
english_prompt = "Write a quicksort algorithm in python" |
|
chinese_prompt = "用Python写一个快速排序算法" |
|
japanese_prompt = "Pythonでクイックソートアルゴリズムを書いてください" |
|
|
|
# Generate and print responses for each language |
|
print("English Response:") |
|
print(generate_response(english_prompt), "\n") |
|
|
|
print("Chinese Response:") |
|
print(generate_response(chinese_prompt), "\n") |
|
|
|
print("Japanese Response:") |
|
print(generate_response(japanese_prompt), "\n") |
|
|
|
``` |
|
|
|
## Example Output |
|
English: |
|
|
|
Write a quicksort algorithm in python. |
|
|
|
```python |
|
def quicksort(arr): |
|
if len(arr) <= 1: |
|
return arr |
|
else: |
|
pivot = arr[0] |
|
less = [i for i in arr[1:] if i <= pivot] |
|
greater = [i for i in arr[1:] if i > pivot] |
|
return quicksort(less) + [pivot] + quicksort(greater) |
|
|
|
arr = [5, 2, 9, 1, 5, 7, 4, 8, 6, 3] |
|
print(quicksort(arr)) |
|
``` |
|
|
|
This is a simple implementation of the quicksort algorithm in python. The function `quicksort` takes an array as input and returns a sorted array. The algorithm works by selecting a pivot element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The process is then repeated recursively on the sub-arrays until the entire array is sorted. |
|
|
|
Chinese Response: |
|
|
|
用Python写一个快速排序算法 |
|
|
|
```python |
|
def quick_sort(arr): |
|
if len(arr) <= 1: |
|
return arr |
|
else: |
|
pivot = arr[0] |
|
less = [i for i in arr[1:] if i <= pivot] |
|
greater = [i for i in arr[1:] if i > pivot] |
|
return quick_sort(less) + [pivot] + quick_sort(greater) |
|
|
|
arr = [3, 5, 2, 1, 4, 6, 8, 7] |
|
print(quick_sort(arr)) |
|
``` |
|
这个程序的时间复杂度为O(nlogn)，空间复杂度为O(n)。 |
|
|
|
Japanese Response: |
|
|
|
Pythonでクイックソートアルゴリズムを書いてください。 |
|
|
|
```python |
|
def quicksort(arr): |
|
if len(arr) <= 1: |
|
return arr |
|
pivot = arr[0] |
|
left = [x for x in arr[1:] if x < pivot] |
|
right = [x for x in arr[1:] if x >= pivot] |
|
return quicksort(left) + [pivot] + quicksort(right) |
|
|
|
print(quicksort([3,6,8,10,1,5,9,2,4,7])) |
|
``` |
|
|
|
このコードはクイックソートアルゴリズムを実装しています。クイックソートは一種の分割と conquers アルゴリズムで、配列を分割し、それぞれの部分配列を再帰的にソートします。 |
|
|
|
この実装では、配列の最初の要素をピボットとして使用します。そして、配列を2つの |
|
|
|
|
|
|
|
# Evaluations |
|
|
|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |
|
|-------------|-------|------|-----:|--------|-----:|---|-----:| |
|
|arc_challenge|Yaml |none | 0|acc |0.5495|± |0.0145| |
|
| | |none | 0|acc_norm|0.5794|± |0.0144| |
|
|arc_easy |Yaml |none | 0|acc |0.8304|± |0.0077| |
|
| | |none | 0|acc_norm|0.8068|± |0.0081| |
|
|boolq |Yaml |none | 0|acc |0.8749|± |0.0058| |
|
|hellaswag |Yaml |none | 0|acc |0.6276|± |0.0048| |
|
| | |none | 0|acc_norm|0.8157|± |0.0039| |
|
|openbookqa |Yaml |none | 0|acc |0.3180|± |0.0208| |
|
| | |none | 0|acc_norm|0.4460|± |0.0223| |
|
|piqa |Yaml |none | 0|acc |0.8139|± |0.0091| |
|
| | |none | 0|acc_norm|0.8237|± |0.0089| |
|
|winogrande |Yaml |none | 0|acc |0.7419|± |0.0123| |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_macadeliccc__laser-polyglot-4x7b) |
|
|
|
| Metric |Value| |
|
|---------------------------------|----:| |
|
|Avg. |65.79| |
|
|AI2 Reasoning Challenge (25-Shot)|64.16| |
|
|HellaSwag (10-Shot) |84.98| |
|
|MMLU (5-Shot) |63.88| |
|
|TruthfulQA (0-shot) |55.47| |
|
|Winogrande (5-shot) |77.82| |
|
|GSM8k (5-shot) |48.45| |
|
|
|
|
