|
|
|
--- |
|
language: |
|
- ru |
|
tags: |
|
- generation |
|
- question answering |
|
- instruction tuning |
|
license: cc-by-nc-4.0 |
|
--- |
|
|
|
### Model Description |
|
|
|
This HF repository contains base LLMs instruction tuned (SFT) with LoRA and then used to study whether monolingual or multilingual instruction tuning is more favourable. |
|
* [GitHub](https://github.com/hplt-project/monolingual-multilingual-instruction-tuning/tree/main) |
|
* [Paper](https://arxiv.org/abs/2309.08958) |
|
|
|
#### Instruction tuning details |
|
* Base model: [EleutherAI/pythia-2.8b-deduped](https://huggingface.co/EleutherAI/pythia-2.8b-deduped) |
|
* Instruction tuning language: Russian |
|
* Training method: LoRA. |
|
* LoRA details: rank=8, alpha=16, target modules={key, query, value}. |
|
* Best checkpoint: best cross-entropy on a validation set, trained for 5 epochs. |
|
* Dataset: machine-translated from [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned). You can download our data [HERE](https://github.com/hplt-project/monolingual-multilingual-instruction-tuning/tree/main/training-data). |
|
|
|
#### Usage |
|
The model checkpoint should be loaded with the base model together using `transformers` and `peft` libraries. |
|
|
|
Please refer to our Github repository [HERE](https://github.com/hplt-project/monolingual-multilingual-instruction-tuning/tree/main/loraft) for inference and training instructions. |
|
|
|
#### Citation |
|
``` |
|
@inproceedings{chen-etal-2024-monolingual, |
|
title="Monolingual or multilingual instruction tuning: Which makes a better {Alpaca}", |
|
author="Pinzhen Chen and Shaoxiong Ji and Nikolay Bogoychev and Andrey Kutuzov and Barry Haddow and Kenneth Heafield", |
|
year="2024", |
|
booktitle = "Findings of the Association for Computational Linguistics: EACL 2024", |
|
} |
|
``` |
|
|
|
|
