README.md

---
license: apache-2.0
language:
- vi
- zh
metrics:
- bleu
library_name: transformers
pipeline_tag: translation
---
# viT5 for Sino-Vietnamese transliteration

<!-- Provide a quick summary of what the model is/does. -->

Finetuned model from viT5 for Chinese MMORPG translation.

## Model Description

<!-- Provide a longer summary of what this model is. -->

Enhanced version from version 1.0 with larger dataset.

## Uses
### Default
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
Step 1: Map all Chinese word from original text to Sino-Vietnamese with [map.json](https://huggingface.co/haruyuu/viT5_han-vie_v1.1/blob/main/map.json) file
```python
with open('map.json', encoding = 'utf-8') as f:
    map = json.load(f)
global map

def mapping(text):
    for i in text:
        try:
            x = ' ' + map[i] + ' '
            text = text.replace(i, x)
        except:
            continue
    return text.strip()

input_text = mapping('“ 早就知道叶微情是卧底了，于是将计就计，想要趁机嫁祸。 ” 的正确证物是：')
```
Step 2: Load model and generate
```python
from transformers import T5ForConditionalGeneration, T5Tokenizer

model = T5ForConditionalGeneration.from_pretrained('haruyuu/viT5_han-vie_v1.1')
tokenizer = T5Tokenizer.from_pretrained('haruyuu/viT5_han-vie_v1.1')

input_ids = tokenizer.encode(input_text, return_tensors="pt")
translated_ids = model.generate(input_ids)
translated_text = tokenizer.decode(translated_ids[0], skip_special_tokens=True)

print("Chinese Input:", input_text)
print("\nVietnamese Translation:", translated_text)
```
## Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

450k rows of system notifications, names and conversations translated from Chinese MMORPG games.