pinzhenchen/sft-lora-en-baichuan-2-7b

Model Description

This HF repository contains base LLMs instruction tuned (SFT) with LoRA and then used to study whether monolingual or multilingual instruction tuning is more favourable.

• GitHub
• Paper

Instruction tuning details

• Base model: baichuan-inc/Baichuan2-7B-Base
• Instruction tuning language: English
• Training method: LoRA.
• LoRA details: rank=8, alpha=16, target modules={key, query, value}.
• Best checkpoint: best cross-entropy on a validation set, trained for 5 epochs.
• Dataset: machine-translated from yahma/alpaca-cleaned. You can download our data HERE.

Usage

The model checkpoint should be loaded with the base model together using transformers and peft libraries.

Please refer to our Github repository HERE for inference and training instructions.

Citation

@inproceedings{chen-etal-2024-monolingual,
  title="Monolingual or multilingual instruction tuning: Which makes a better {Alpaca}",
  author="Pinzhen Chen and Shaoxiong Ji and Nikolay Bogoychev and Andrey Kutuzov and Barry Haddow and Kenneth Heafield",
  year="2024",
  booktitle = "Findings of the Association for Computational Linguistics: EACL 2024",
}