tugstugi commited on
Commit
b3e33f8
1 Parent(s): bd52f57

initial commit

Browse files

README.md ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: "mn"
3
+ tags:
4
+ - mongolian
5
+ - cased
6
+ ---
7
+
8
+ # BERT-BASE-MONGOLIAN-CASED
9
+ [Link to Official Mongolian-BERT repo](https://github.com/tugstugi/mongolian-bert)
10
+
11
+ ## Model description
12
+ This repository contains pre-trained Mongolian [BERT](https://arxiv.org/abs/1810.04805) models trained by [tugstugi](https://github.com/tugstugi), [enod](https://github.com/enod) and [sharavsambuu](https://github.com/sharavsambuu).
13
+ Special thanks to [nabar](https://github.com/nabar) who provided 5x TPUs.
14
+
15
+ This repository is based on the following open source projects: [google-research/bert](https://github.com/google-research/bert/),
16
+ [huggingface/pytorch-pretrained-BERT](https://github.com/huggingface/pytorch-pretrained-BERT) and [yoheikikuta/bert-japanese](https://github.com/yoheikikuta/bert-japanese).
17
+
18
+ #### How to use
19
+
20
+ ```python
21
+ from transformers import pipeline, AutoTokenizer, BertForMaskedLM
22
+
23
+ tokenizer = AutoTokenizer.from_pretrained('tugstugi/bert-large-mongolian-cased')
24
+ model = BertForMaskedLM.from_pretrained('tugstugi/bert-large-mongolian-cased')
25
+
26
+ ## declare task ##
27
+ pipe = pipeline(task="fill-mask", model=model, tokenizer=tokenizer)
28
+
29
+ ## example ##
30
+ input_ = 'Миний [MASK] хоол идэх нь тун чухал.'
31
+
32
+ output_ = pipe(input_)
33
+ for i in range(len(output_)):
34
+ print(output_[i])
35
+
36
+ ```
37
+
38
+
39
+ ## Training data
40
+ Mongolian Wikipedia and the 700 million word Mongolian news data set [[Pretraining Procedure](https://github.com/tugstugi/mongolian-bert#pre-training)]
41
+
42
+ ### BibTeX entry and citation info
43
+
44
+ ```bibtex
45
+ @misc{mongolian-bert,
46
+ author = {Tuguldur, Erdene-Ochir and Gunchinish, Sharavsambuu and Bataa, Enkhbold},
47
+ title = {BERT Pretrained Models on Mongolian Datasets},
48
+ year = {2019},
49
+ publisher = {GitHub},
50
+ journal = {GitHub repository},
51
+ howpublished = {\url{https://github.com/tugstugi/mongolian-bert/}}
52
+ }
53
+ ```
added_tokens.json ADDED
@@ -0,0 +1 @@
 
1
+ {"<pad>": 32000}
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/content/model-large-32k-512-4000000",
3
+ "architectures": [
4
+ "BertForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "directionality": "bidi",
8
+ "gradient_checkpointing": false,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 1024,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 4096,
14
+ "layer_norm_eps": 1e-12,
15
+ "max_position_embeddings": 512,
16
+ "model_type": "bert",
17
+ "num_attention_heads": 16,
18
+ "num_hidden_layers": 24,
19
+ "pad_token_id": 0,
20
+ "pooler_fc_size": 768,
21
+ "pooler_num_attention_heads": 12,
22
+ "pooler_num_fc_layers": 3,
23
+ "pooler_size_per_head": 128,
24
+ "pooler_type": "first_token_transform",
25
+ "type_vocab_size": 2,
26
+ "vocab_size": 32000,
27
+ "tokenizer_class": "AlbertTokenizer"
28
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b785d92c3da4579f60d3335ce83d6f7f50c756dad9411f14c06bb44e2a8a2044
3
+ size 1346919669
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
1
+ {"bos_token": "[CLS]", "eos_token": "[SEP]", "unk_token": "<unk>", "sep_token": "[SEP]", "pad_token": "<pad>", "cls_token": "[CLS]", "mask_token": "[MASK]"}
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7316d915a97e6f879ab59e7bd8a2e27abe098ca96c4eee7a55fbb3196222b224
3
+ size 1000174
tf_model.h5 ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a85b4d6029133f6332c567fcba9b36b83ed96a8b4663e5d3ab800196f2a4600c
3
+ size 1484666328
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
1
+ {"keep_accents": true, "do_lower_case": false}