--- language: - fi license: apache-2.0 tags: - text-generation-inference - transformers - unsloth - llama - trl - sft base_model: LumiOpen/Viking-7B datasets: - mpasila/Finnish-Alpaca-Tiny --- This is a merge of [mpasila/Finnish-Alpaca-Tiny-V2-LoRA-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Tiny-V2-LoRA-7B). LoRA trained in 4-bit with 2k context using [LumiOpen/Viking-7B](https://huggingface.co/LumiOpen/Viking-7B/) as the base model for 1 epoch. Dataset used is [mpasila/Finnish-Alpaca-Tiny](https://huggingface.co/datasets/mpasila/Finnish-Alpaca-Tiny). It works relatively well for question and answering. I will make a bigger dataset for the next fine-tune. ### Prompt format: Alpaca It uses Alpaca format but with a translated instruction at the start: ``` { "instruction,output": "Alla on ohje, jossa kuvataan tehtävä. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%", "instruction,input,output": "Alla on ohje, jossa kuvataan tehtävä ja joka on yhdistetty kontekstia lisäävään syötteeseen. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%" } ``` ## Evaluation | Model | Size | Type | FIN-bench (score) | |-------|------|------|-------| | **mpasila/Finnish-Alpaca-Tiny-V2-7B** | 7B | Instruct | **0.4654** | | [mpasila/Finnish-Alpaca-Small-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Small-7B) | 7B | Instruct | 0.3586 | | [mpasila/Alpacazord-Viking-7B](https://huggingface.co/mpasila/Alpacazord-Viking-7B) | 7B | Instruct | 0.4123 | | [mpasila/NordicAlpaca-Finnish-V1-7B](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B) | 7B | Instruct | 0.3891 | | [mpasila/Finnish-Viking-Alpaca-V1-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-7B) | 7B | Instruct | 0.3943 | | [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) | 7B | Instruct | 0.4365 | | [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) | 7B | Instruct | 0.3993 | | [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) | 7B | Base | 0.2350 | | [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) | 7B | Base | 0.3721 | | [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) | 7B | Base | 0.3169 | [Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing) #### FIN-bench scores: | Task |Version| Metric |Value | |Stderr| |------------------------------------------------|------:|---------------------|-----:|---|-----:| |bigbench_analogies | 0|multiple_choice_grade|0.6385|± |0.0423| |bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.7200|± |0.0451| |bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.7391|± |0.0936| |bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.4800|± |0.0502| |bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.6300|± |0.0485| |bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.4000|± |0.0492| |bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.5000|± |0.0503| |bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.2800|± |0.0451| |bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.4300|± |0.0498| |bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.5800|± |0.0496| |bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.3100|± |0.0465| |bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.2900|± |0.0456| |bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.5100|± |0.0502| |bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.5300|± |0.0502| |bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.3900|± |0.0490| |bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.3100|± |0.0465| |bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.6200|± |0.0488| |bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.6500|± |0.0479| |bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.3200|± |0.0469| |bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.3000|± |0.0461| |bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.6400|± |0.0482| |bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.5686|± |0.0700| |bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.6471|± |0.0676| |bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.4314|± |0.0700| |bigbench_emotions | 0|multiple_choice_grade|0.2250|± |0.0331| |bigbench_empirical_judgments | 0|multiple_choice_grade|0.2525|± |0.0439| |bigbench_general_knowledge | 0|multiple_choice_grade|0.3429|± |0.0571| |bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3793|± |0.0643| |bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3390|± |0.0622| |bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.3729|± |0.0635| |bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5349|± |0.0770| |bigbench_intent_recognition | 0|multiple_choice_grade|0.2153|± |0.0156| |bigbench_misconceptions | 0|multiple_choice_grade|0.5224|± |0.0433| |bigbench_paraphrase | 0|multiple_choice_grade|0.4750|± |0.0354| |bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.4833|± |0.0651| |bigbench_similarities_abstraction | 0|multiple_choice_grade|0.6974|± |0.0530| # Uploaded model - **Developed by:** mpasila - **License:** apache-2.0 - **Finetuned from model :** LumiOpen/Viking-7B This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. [](https://github.com/unslothai/unsloth)