t5-base-japanese-web (with Byte-fallback, 32K)

Description

megagonlabs/t5-base-japanese-web is a T5 (Text-to-Text Transfer Transformer) model pre-trained on Japanese web texts.
Training codes are available on GitHub.

The vocabulary size of this model is 32K. 8K version is also available.

Corpora

We used following corpora for pre-training.

• Japanese in mC4/3.0.1 (We used Tensorflow native format)
  • 87,425,304 pages
  • 782 GB in TFRecord format
• Japanese in wiki40b/1.3.0
  • 828,236 articles (2,073,584 examples)
  • 2 GB in TFRecord format

Tokenizer

We used Japanese Wikipedia to train SentencePiece.

• Vocabulary size: 32,000
• Byte-fallback: Enabled

Parameters

• T5 model: models/t5.1.1.base.gin
• Training steps: 1,000,000

It took about 126 hours with TPU v3-8

Related models

• 日本語T5事前学習済みモデル (sonoisa/t5-base-japanese)
• 日本語T5事前学習済みモデル (sonoisa/t5-base-japanese-mC4-Wikipedia)

License

Apache License 2.0

Citations

• mC4

Contains information from mC4 which is made available under the ODC Attribution License.

@article{2019t5,
    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
    journal = {arXiv e-prints},
    year = {2019},
    archivePrefix = {arXiv},
    eprint = {1910.10683},
}

• wiki40b

@inproceedings{49029,
title = {Wiki-40B: Multilingual Language Model Dataset},
author = {Mandy Guo and Zihang Dai and Denny Vrandecic and Rami Al-Rfou},
year = {2020},
booktitle   = {LREC 2020}
}