|
--- |
|
language: |
|
- ja |
|
thumbnail: |
|
tags: |
|
- xlnet |
|
- lm-head |
|
- causal-lm |
|
license: |
|
- apache-2.0 |
|
datasets: |
|
- Japanese_Business_News |
|
metrics: |
|
--- |
|
|
|
# XLNet-japanese |
|
|
|
## Model description |
|
This model require Mecab and senetencepiece with XLNetTokenizer. |
|
See details https://qiita.com/mkt3/items/4d0ae36f3f212aee8002 |
|
|
|
This model uses NFKD as the normalization method for character encoding. |
|
Japanese muddle marks and semi-muddle marks will be lost. |
|
|
|
*日本語の濁点・半濁点がないモデルです* |
|
|
|
#### How to use |
|
|
|
```python |
|
from fugashi import Tagger |
|
|
|
from transformers import ( |
|
pipeline, |
|
XLNetLMHeadModel, |
|
XLNetTokenizer |
|
) |
|
|
|
class XLNet(): |
|
def __init__(self): |
|
self.m = Tagger('-Owakati') |
|
self.gen_model = XLNetLMHeadModel.from_pretrained("hajime9652/xlnet-japanese") |
|
self.gen_tokenizer = XLNetTokenizer.from_pretrained("hajime9652/xlnet-japanese") |
|
|
|
def generate(self, prompt="福岡のご飯は美味しい。コンパクトで暮らしやすい街。"): |
|
prompt = self.m.parse(prompt) |
|
inputs = self.gen_tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt") |
|
prompt_length = len(self.gen_tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)) |
|
outputs = self.gen_model.generate(inputs, max_length=200, do_sample=True, top_p=0.95, top_k=60) |
|
generated = prompt + self.gen_tokenizer.decode(outputs[0])[prompt_length:] |
|
return generated |
|
``` |
|
|
|
#### Limitations and bias |
|
This model's training use the Japanese Business News. |
|
|
|
|
|
# Important matter |
|
The company that created and published this model is called Stockmark. |
|
This repository is for use by HuggingFace and not for infringement. |
|
See this documents https://qiita.com/mkt3/items/4d0ae36f3f212aee8002 |
|
published by https://github.com/mkt3 |
|
|
