metadata
language: ja
tags:
- t5
- text2text-generation
- seq2seq
license: apache-2.0
datasets:
- mc4
- wiki40b
t5-base-japanese-web (with Byte-fallback)
Description
megagonlabs/t5-base-japanese-web is a T5 (Text-to-Text Transfer Transformer) model pre-trained on Japanese web texts.
Training codes are available on GitHub.
Corpus
- Japanese in mC4/3.0.1 (We used Tensorflow native format)
- Japanese in wiki40b/1.3.0
Tokenizer
SentencePiece trained on Japanese Wikipedia
- Vocabulary size: 32,000
- Byte-fallback: Enabled
Parameters
- T5 model: models/t5.1.1.base.gin
- Training steps: 1,000,000
It took about 126 hours with TPU v3-8
Related models
License
Apache License 2.0
Citations
- mC4
Contains information from mC4 which is made available under the ODC Attribution License.
@article{2019t5,
author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
journal = {arXiv e-prints},
year = {2019},
archivePrefix = {arXiv},
eprint = {1910.10683},
}