haonan-li commited on
Commit
4ac4ca9
1 Parent(s): 9db6b6d

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +65 -0
README.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ ---
4
+
5
+ This repo contains a low-rank adapter (LoRA) for BLOOM-7b1
6
+ fit on the [Stanford-Alpaca-52k](https://github.com/tatsu-lab/stanford_alpaca)
7
+ and [databricks-dolly-15k](https://github.com/databrickslabs/dolly/tree/master/data) data in Turkish.
8
+
9
+ ### Dataset Creation
10
+
11
+ 1. English Instructions: The English instuctions are obtained from [alpaca-52k](https://github.com/tatsu-lab/stanford_alpaca), and [dolly-15k](https://github.com/databrickslabs/dolly/tree/master/data).
12
+ 2. Instruction Translation: The instructions (and inputs) are translated into the target languages using Google Translation API (conducted on April 2023).
13
+ 3. Output Generation: We generate output from `gpt-3.5-turbo` for each language (conducted on April 2023).
14
+
15
+ <h3 align="center">
16
+ <img src="https://raw.githubusercontent.com/fajri91/eval_picts/master/BactrianX_dataset.jpg" width="950" align="center">
17
+ </h3>
18
+
19
+ ### Training Parameters
20
+
21
+ The code for training the model is provided in our [github](https://github.com/mbzuai-nlp/Bactrian-X), which is adapted from [Alpaca-LoRA](https://github.com/tloen/alpaca-lora).
22
+ This version of the weights was trained with the following hyperparameters:
23
+
24
+ - Epochs: 8
25
+ - Batch size: 128
26
+ - Cutoff length: 1024
27
+ - Learning rate: 3e-4
28
+ - Lora _r_: 16
29
+ - Lora target modules: query_key_value
30
+
31
+
32
+ That is:
33
+
34
+ ```
35
+ python finetune.py \
36
+ --base_model='bigscience/bloom-7b1' \
37
+ --num_epochs=5 \
38
+ --cutoff_len=1024 \
39
+ --group_by_length \
40
+ --output_dir='./bactrian-tr-bloom-7b1-lora' \
41
+ --lora_target_modules='query_key_value' \
42
+ --lora_r=16 \
43
+ --micro_batch_size=32
44
+ ```
45
+
46
+ Instructions for running it can be found at https://github.com/MBZUAI-nlp/Bactrian-X.
47
+
48
+ ### Discussion of Biases
49
+
50
+ (1) Translation bias; (2) Potential English-culture bias in the translated dataset.
51
+
52
+
53
+ ### Citation Information
54
+
55
+ ```
56
+ @misc{li2023bactrianx,
57
+ title={Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation},
58
+ author={Haonan Li and Fajri Koto and Minghao Wu and Alham Fikri Aji and Timothy Baldwin},
59
+ year={2023},
60
+ eprint={2305.15011},
61
+ archivePrefix={arXiv},
62
+ primaryClass={cs.CL}
63
+ }
64
+
65
+ ```