initial commit

README.md

@@ -1,3 +1,70 @@
 ---
+language: ja
 license: mit
+datasets:
+- mC4-ja
 ---
+
+# electra-base-japanese-discriminator (sudachitra-wordpiece, mC4 Japanese) - [SHINOBU](https://dl.ndl.go.jp/info:ndljp/pid/1302683/3)
+
+This is an [ELECTRA](https://github.com/google-research/electra) model pretrained on approximately 200M Japanese sentences.
+
+The input text is tokenized by [SudachiTra](https://github.com/WorksApplications/SudachiTra) with the WordPiece subword tokenizer.
+See `tokenizer_config.json` for the setting details.
+
+## How to use
+
+Please install `SudachiTra` in advance.
+
+```console
+$ pip install -U torch transformers sudachitra
+```
+
+You can load the model and the tokenizer via AutoModel and AutoTokenizer, respectively.
+
+```python
+from transformers import AutoModel, AutoTokenizer
+model = AutoModel.from_pretrained("megagonlabs/electra-base-japanese-discriminator")
+tokenizer = AutoTokenizer.from_pretrained("megagonlabs/electra-base-japanese-discriminator", trust_remote_code=True)
+model(**tokenizer("まさにオールマイティーな商品だ。", return_tensors="pt")).last_hidden_state
+tensor([[[-0.0498, -0.0285,  0.1042,  ...,  0.0062, -0.1253,  0.0338],
+         [-0.0686,  0.0071,  0.0087,  ..., -0.0210, -0.1042, -0.0320],
+         [-0.0636,  0.1465,  0.0263,  ...,  0.0309, -0.1841,  0.0182],
+         ...,
+         [-0.1500, -0.0368, -0.0816,  ..., -0.0303, -0.1653,  0.0650],
+         [-0.0457,  0.0770, -0.0183,  ..., -0.0108, -0.1903,  0.0694],
+         [-0.0981, -0.0387,  0.1009,  ..., -0.0150, -0.0702,  0.0455]]],
+       grad_fn=<NativeLayerNormBackward>)
+```
+
+## Model architecture
+
+The model architecture is the same as the original ELECTRA base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads.
+
+## Training data and libraries
+
+This model is trained on the Japanese texts extracted from the [mC4](https://huggingface.co/datasets/mc4) Common Crawl's multilingual web crawl corpus.
+We used the [Sudachi](https://github.com/WorksApplications/Sudachi) to split texts into sentences, and also applied a simple rule-based filter to remove nonlinguistic segments of mC4 multilingual corpus.
+The extracted texts contains over 600M sentences in total, and we used approximately 200M sentences for pretraining.
+
+We used [NVIDIA's TensorFlow2-based ELECTRA implementation](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/LanguageModeling/ELECTRA) for pretraining. The time required for the pretrainig was about 110 hours using GCP DGX A100 8gpu instance with enabling Automatic Mixed Precision.
+
+## Licenses
+
+The pretrained models are distributed under the terms of the [MIT License](https://opensource.org/licenses/mit-license.php).
+
+## Citations
+
+- mC4
+
+Contains information from `mC4` which is made available under the [ODC Attribution License](https://opendatacommons.org/licenses/by/1-0/).
+```
+@article{2019t5,
+    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
+    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
+    journal = {arXiv e-prints},
+    year = {2019},
+    archivePrefix = {arXiv},
+    eprint = {1910.10683},
+}
+```

config.json

@@ -0,0 +1,26 @@
+{
+ "architectures": [
+  "ElectraForPreTraining"
+ ],
+ "model_type": "electra",
+ "model_name": "base",
+ "vocab_size": 30112,
+ "embedding_size": 768,
+ "hidden_size": 768,
+ "num_hidden_layers": 12,
+ "num_attention_heads": 12,
+ "intermediate_size": 3072,
+ "hidden_act": "gelu",
+ "hidden_dropout_prob": 0.1,
+ "attention_probs_dropout_prob": 0.1,
+ "max_position_embeddings": 512,
+ "type_vocab_size": 2,
+ "initializer_range": 0.02,
+ "layer_norm_eps": 1e-12,
+ "summary_type": "first",
+ "summary_use_proj": true,
+ "summary_activation": "gelu",
+ "summary_last_dropout": 0.1,
+ "pad_token_id": 0,
+ "position_embedding_type": "absolute"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4412201f4146092b562f92ec0ea5be3ad697cd0c83c4817a5ea411375900cd9
+size 436755117

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+ "tokenizer_class": "BertJapaneseTokenizer",
+ "do_lower_case": false,
+ "do_word_tokenize": true,
+ "do_subword_tokenize": true,
+ "word_tokenizer_type": "sudachi",
+ "subword_tokenizer_type": "wordpiece",
+ "model_max_length": 512,
+ "sudachi_kwargs": {"sudachi_split_mode":"A","sudachi_dict_type":"core"}
+}
+