Commit
•
2d78108
1
Parent(s):
1f663e4
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
# IndicTrans2 HF Compatible Models
|
2 |
|
3 |
In this section, we provide details on how to use our [IndicTrans2](https://github.com/AI4Bharat/IndicTrans2) models which were originally trained with the [fairseq](https://github.com/facebookresearch/fairseq) to [HuggingFace transformers](https://huggingface.co/docs/transformers/index) for inference purpose. Our scripts for HuggingFace compatible models are adapted from [M2M100 repository](https://github.com/huggingface/transformers/tree/main/src/transformers/models/m2m_100).
|
@@ -19,17 +32,6 @@ source install.sh
|
|
19 |
> Note: The `install.sh` script in this directory is specifically for running HF compatible models for inference.
|
20 |
|
21 |
|
22 |
-
### Converting
|
23 |
-
|
24 |
-
In order to convert the fairseq checkpoint to a PyTorch checkpoint that is compatible with HuggingFace Transformers, use the following command:
|
25 |
-
|
26 |
-
```bash
|
27 |
-
python3 convert_indictrans_checkpoint_to_pytorch.py --fairseq_path <fairseq_checkpoint_best.pt> --pytorch_dump_folder_path <hf_output_dir>
|
28 |
-
```
|
29 |
-
- `<fairseq_checkpoint_best.pt>`: path to the fairseq `checkpoint_best.pt` that needs to be converted to HF compatible models
|
30 |
-
- `<hf_output_dir>`: path to the output directory where the HF compatible models will be saved
|
31 |
-
|
32 |
-
|
33 |
### Models
|
34 |
|
35 |
| Model | 🤗 HuggingFace Checkpoints |
|
@@ -59,4 +61,4 @@ Feel free to modify the `example.py` script to suit your translation needs.
|
|
59 |
year = {2023},
|
60 |
journal = {arXiv preprint arXiv: 2305.16307}
|
61 |
}
|
62 |
-
```
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- en
|
4 |
+
- hi
|
5 |
+
- te
|
6 |
+
- ta
|
7 |
+
- mr
|
8 |
+
- kn
|
9 |
+
- ml
|
10 |
+
pipeline_tag: translation
|
11 |
+
tags:
|
12 |
+
- indic_trans_v2
|
13 |
+
---
|
14 |
# IndicTrans2 HF Compatible Models
|
15 |
|
16 |
In this section, we provide details on how to use our [IndicTrans2](https://github.com/AI4Bharat/IndicTrans2) models which were originally trained with the [fairseq](https://github.com/facebookresearch/fairseq) to [HuggingFace transformers](https://huggingface.co/docs/transformers/index) for inference purpose. Our scripts for HuggingFace compatible models are adapted from [M2M100 repository](https://github.com/huggingface/transformers/tree/main/src/transformers/models/m2m_100).
|
|
|
32 |
> Note: The `install.sh` script in this directory is specifically for running HF compatible models for inference.
|
33 |
|
34 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
35 |
### Models
|
36 |
|
37 |
| Model | 🤗 HuggingFace Checkpoints |
|
|
|
61 |
year = {2023},
|
62 |
journal = {arXiv preprint arXiv: 2305.16307}
|
63 |
}
|
64 |
+
```
|