codemix 0.1.5.1
Codemix translation package
Keywords: NLP deep learning transformer pytorch translation NMT codemix Hindi Bengali

# Library Under Development

## Robust Machine Translation for Codemix Languages

This project aims to provide multiple pretrained models for translating text from codemix languages such as Hinglish, Bengalish to English. The models are capable of translating all-inclusive text i.e. an input with a mix of Devanagari / Romanized Hindi. The robustness capabilities of the model enables it to effectively handle spelling mistakes.

### Requirements

1. Python<3.7
2. torch==1.4.0
3. tqdm
4. numpy
5. sentencepiece
6. fairseq (commit=```cb84694c195afced474d17318b5e746d1a9d20a3```)

### Installation

```pip install codemix```

```pip install git+https://github.com/Kartikaggarwal98/fairseq.git```

## Usage
3 different models are trained on 2 codemix datasets: HINMIX_CALIGN and HINMIX_CTRANS. The output translations will vary according to the type of data and training methodology.

Create directories ```data-bin```, ```checkpoints``` in the working folder. Download preprocess files and checkpoints for a model and unzip it respective folders:

| Dataset | Model | preprocess files | checkpoint | 
| :-----: | :---: | :--------------: | :--------: |
| align   | rcmt1 | [link](https://drive.google.com/file/d/18xG5K7ufAlNL0y57GEyf5ARLCMclfzip/view?usp=sharing)             | [link](https://drive.google.com/file/d/1Sd5r9ABIbifJ9We0WlJN-sm73-kZIe9k/view?usp=sharing)       |
| align   | rcmt2 | [link](https://drive.google.com/file/d/15KBDY6ITOBHaalyPRg2vV8yTMKY86FN8/view?usp=sharing)             | [link](https://drive.google.com/file/d/1Uhe-3WpeXFihgZTjipPu7XymsUvwiejy/view?usp=sharing)       |
| align   | zcmt  | [link](https://drive.google.com/file/d/1zvptqomBUQHIe4WwKoQH5FO69VOeu9gw/view?usp=sharing)             | [link](https://drive.google.com/file/d/1O5Kc5FSuejBuzUKczJShDSGAW1Nr777m/view?usp=sharing)       |
| trans   | rcmt1 | [link](https://drive.google.com/file/d/12yqPom2NsOmpXO-VV3AmyOVLQwv43cJP/view?usp=sharing)             | [link](https://drive.google.com/file/d/1cfQR161wUNHaFkjrGRxmTRLMjIy10USJ/view?usp=sharing)       |
| trans   | rcmt2 | [link](https://drive.google.com/file/d/1PCy1_JCOYvSIgdH0X-fItzoU_JpVC_co/view?usp=sharing)             | [link](https://drive.google.com/file/d/1LU9b8O3mugTn6NP5wrP4mlsa4Ljwiv9C/view?usp=sharing)       |
| trans   | zcmt  | [link](https://drive.google.com/file/d/17wUq7egSapT5ELt5Tjt73R8FKBB8wkfP/view?usp=sharing)             | [link](https://drive.google.com/file/d/1IozgrL93amIMycWwMco_sDjwVBmrduvE/view?usp=sharing)       |

To run default values of text, dataset and model, run:

```cm_translate```

For customized translations, run:

```cm_translate --model model_name --dataset dataset_name --source input_language --target output_language --input input_sentence```

The following datasets are available:

1. trans
2. align

The following models are available:

1. rcmt1
2. rcmt2
3. zcmt

The following languages are available:

1. Hindi (```hi```)
2. Bengali (```bn```)
3. English (```en```)

The file structure would look like this:

```
├──    current_directory
|   ├──    checkpoints
|   |   ├──    hinmix_calign_rcmt1
|   |   ├──    checkpoint.pt
|   ├──    data-bin
|   |   ├──    hinmix_calign_rcmt1
|   |   |   ├──    dict.lang.txt
|   |   |   ├──    fairseq.vocab
|   |   |   ├──    sentencepiece.uni.model
```