Hugging Face's logo

sappho192
/
jesc-ja-en-translator

Translation
Transformers
PyTorch
ONNX
Safetensors
Japanese
English
encoder-decoder
text2text-generation
Model card Files Community
2
jesc-ja-en-translator
File size: 2,376 Bytes 
f14af46
 
21e09ac
 
 
 
f14af46
21e09ac
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
---
license: mit
language:
- ja
- en
pipeline_tag: translation
---

# Japanese to Korean translator

Japanese to Korean translator model based on [EncoderDecoderModel](https://huggingface.co/docs/transformers/model_doc/encoder-decoder)([bert-japanese](https://huggingface.co/cl-tohoku/bert-base-japanese)+[GPT2](https://huggingface.co/openai-community/gpt2))

# Usage
## Demo
Please visit https://huggingface.co/spaces/sappho192/jesc-ja-en-translator-demo

## Dependencies (PyPI)

- torch
- transformers
- fugashi
- unidic-lite

## Inference

```Python
import transformers
import torch

encoder_model_name = "cl-tohoku/bert-base-japanese-v2"
decoder_model_name = "openai-community/gpt2"
src_tokenizer = transformers.BertJapaneseTokenizer.from_pretrained(encoder_model_name)
trg_tokenizer = transformers.PreTrainedTokenizerFast.from_pretrained(decoder_model_name)
model = transformers.EncoderDecoderModel.from_pretrained("sappho192/jesc-ja-en-translator")


def translate(text_src):
    embeddings = src_tokenizer(text_src, return_attention_mask=False, return_token_type_ids=False, return_tensors='pt')
    embeddings = {k: v for k, v in embeddings.items()}
    output = model.generate(**embeddings, max_length=512)[0, 1:-1]
    text_trg = trg_tokenizer.decode(output.cpu())
    return text_trg

texts = [
    "逃げろ!",  # Should be "run!"
    "εˆγ‚γΎγ—γ¦.",  # "nice to meet you."
    "γ‚ˆγ‚γ—γγŠι‘˜γ„γ—γΎγ™.",  # "thank you."
    "倜にγͺγ‚ŠγΎγ—γŸ",  # "and then it got dark."
    "γ”ι£―γ‚’ι£ŸγΉγΎγ—γ‚‡γ†."  # "let's eat."
 ]

for text in texts:
    print(translate(text))
    print()

```

# Dataset

The dataset used to train the model is JESC(Japanese-English Subtitle Corpus).  
Its license is [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). 
All data information can be accessed through following links:

- Dataset link: https://nlp.stanford.edu/projects/jesc/
- Paper link: https://arxiv.org/abs/1710.10639
- Github link: https://github.com/rpryzant/JESC
- Bibtex:

```bibtex
@ARTICLE{pryzant_jesc_2017,
   author = {{Pryzant}, R. and {Chung}, Y. and {Jurafsky}, D. and {Britz}, D.},
    title = "{JESC: Japanese-English Subtitle Corpus}",
  journal = {ArXiv e-prints},
archivePrefix = "arXiv",
   eprint = {1710.10639},
 keywords = {Computer Science - Computation and Language},
     year = 2017,
    month = oct,
}
```