--- language: - ja - en - zh license: apache-2.0 model-index: - name: laser-polyglot-4x7b results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 64.16 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 84.98 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 63.88 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 55.47 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 77.82 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 48.45 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b name: Open LLM Leaderboard --- # Polyglot-4x7b-24b ![polyglot](polyglot.png) Polyglot-4x7b is a Mixture of Experts approach to a multilingual model. This project is an experiment to see if each expert can be of a different language. The answer is yes. The model is a merge of models that are capable of Chinese and Japanese output. + teknium/OpenHermes-2.5-Mistral-7B + oshizo/japanese-e5-mistral-7b_slerp + cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser + s3nh/Mistral-7B-Evol-Instruct-Chinese TODO: 1. [] polyglot tokenizer ## Other polyglot models + [macadeliccc/Polyglot-8x7b-v0.1](https://huggingface.co/macadeliccc/Polyglot-8x7b-v0.1) (adds 3 more languages) # Code Example Inference [Colab](https://colab.research.google.com/drive/1tYSb63IKZDsiQ5BIJU8Oc92phxugAmB3?usp=sharing) Live demo available on [Spaces](https://huggingface.co/spaces/macadeliccc/polyglot-4x7b-chat?logs=build) ```python from transformers import AutoModelForCausalLM, AutoTokenizer def generate_response(prompt): """ Generate a response from the model based on the input prompt. Args: prompt (str): Prompt for the model. Returns: str: The generated response from the model. """ # Tokenize the input prompt inputs = tokenizer(prompt, return_tensors="pt") # Generate output tokens outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id) # Decode the generated tokens to a string response = tokenizer.decode(outputs[0], skip_special_tokens=True) return response # Load the model and tokenizer model_id = "macadeliccc/laser-polyglot-4x7b" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True) # Example prompts in different languages english_prompt = "Write a quicksort algorithm in python" chinese_prompt = "用Python写一个快速排序算法" japanese_prompt = "Pythonでクイックソートアルゴリズムを書いてください" # Generate and print responses for each language print("English Response:") print(generate_response(english_prompt), "\n") print("Chinese Response:") print(generate_response(chinese_prompt), "\n") print("Japanese Response:") print(generate_response(japanese_prompt), "\n") ``` ## Example Output English: Write a quicksort algorithm in python. ```python def quicksort(arr): if len(arr) <= 1: return arr else: pivot = arr[0] less = [i for i in arr[1:] if i <= pivot] greater = [i for i in arr[1:] if i > pivot] return quicksort(less) + [pivot] + quicksort(greater) arr = [5, 2, 9, 1, 5, 7, 4, 8, 6, 3] print(quicksort(arr)) ``` This is a simple implementation of the quicksort algorithm in python. The function `quicksort` takes an array as input and returns a sorted array. The algorithm works by selecting a pivot element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The process is then repeated recursively on the sub-arrays until the entire array is sorted. Chinese Response: 用Python写一个快速排序算法 ```python def quick_sort(arr): if len(arr) <= 1: return arr else: pivot = arr[0] less = [i for i in arr[1:] if i <= pivot] greater = [i for i in arr[1:] if i > pivot] return quick_sort(less) + [pivot] + quick_sort(greater) arr = [3, 5, 2, 1, 4, 6, 8, 7] print(quick_sort(arr)) ``` 这个程序的时间复杂度为O(nlogn)，空间复杂度为O(n)。 Japanese Response: Pythonでクイックソートアルゴリズムを書いてください。 ```python def quicksort(arr): if len(arr) <= 1: return arr pivot = arr[0] left = [x for x in arr[1:] if x < pivot] right = [x for x in arr[1:] if x >= pivot] return quicksort(left) + [pivot] + quicksort(right) print(quicksort([3,6,8,10,1,5,9,2,4,7])) ``` このコードはクイックソートアルゴリズムを実装しています。クイックソートは一種の分割と conquers アルゴリズムで、配列を分割し、それぞれの部分配列を再帰的にソートします。 この実装では、配列の最初の要素をピボットとして使用します。そして、配列を2つの # Evaluations | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |-------------|-------|------|-----:|--------|-----:|---|-----:| |arc_challenge|Yaml |none | 0|acc |0.5495|± |0.0145| | | |none | 0|acc_norm|0.5794|± |0.0144| |arc_easy |Yaml |none | 0|acc |0.8304|± |0.0077| | | |none | 0|acc_norm|0.8068|± |0.0081| |boolq |Yaml |none | 0|acc |0.8749|± |0.0058| |hellaswag |Yaml |none | 0|acc |0.6276|± |0.0048| | | |none | 0|acc_norm|0.8157|± |0.0039| |openbookqa |Yaml |none | 0|acc |0.3180|± |0.0208| | | |none | 0|acc_norm|0.4460|± |0.0223| |piqa |Yaml |none | 0|acc |0.8139|± |0.0091| | | |none | 0|acc_norm|0.8237|± |0.0089| |winogrande |Yaml |none | 0|acc |0.7419|± |0.0123| # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_macadeliccc__laser-polyglot-4x7b) | Metric |Value| |---------------------------------|----:| |Avg. |65.79| |AI2 Reasoning Challenge (25-Shot)|64.16| |HellaSwag (10-Shot) |84.98| |MMLU (5-Shot) |63.88| |TruthfulQA (0-shot) |55.47| |Winogrande (5-shot) |77.82| |GSM8k (5-shot) |48.45|