--- language: - en license: mit library_name: transformers tags: - peft datasets: - HuggingFaceH4/ultrachat_200k model-index: - name: Maixtchup-4x7b-QLoRA-SFT-UltraChat results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 60.92 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 83.23 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 60.78 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 53.33 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 77.19 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 43.21 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat name: Open LLM Leaderboard --- LoRA adapter for kaitchup/Maixtchup-4x7b briefly fine-tuned on UltraChat. To load and use this adapter: ``` model_name = "kaitchup/Maixtchup-4x7b" #Tokenizer tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True) compute_dtype = getattr(torch, "float16") bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=compute_dtype, bnb_4bit_use_double_quant=True, ) model = AutoModelForCausalLM.from_pretrained( model_name, quantization_config=bnb_config, device_map="auto", attn_implementation="flash_attention_2", ) model.config.use_cache = True model = PeftModel.from_pretrained(model, "kaitchup/Maixtchup-4x7b-QLoRA-SFT-UltraChat") ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_kaitchup__Maixtchup-4x7b-QLoRA-SFT-UltraChat) | Metric |Value| |---------------------------------|----:| |Avg. |63.11| |AI2 Reasoning Challenge (25-Shot)|60.92| |HellaSwag (10-Shot) |83.23| |MMLU (5-Shot) |60.78| |TruthfulQA (0-shot) |53.33| |Winogrande (5-shot) |77.19| |GSM8k (5-shot) |43.21|