update auto tokenizer support

README.md

@@ -1,5 +1,8 @@
 ---
 language: zh
+pipeline_tag: fill-mask
+widget:
+- text: "今天[MASK]情很好"
 ---
 
 # albert_chinese_base
@@ -7,6 +10,9 @@ language: zh
 This a albert_chinese_base model from [Google's github](https://github.com/google-research/ALBERT)  
 converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)
 
+## Update
+Support AutoTokenizer
+
 ## Attention (注意)
 
 Since sentencepiece is not used in albert_chinese_base model   
@@ -20,12 +26,12 @@ we can eval it using an example on MaskedLM
 ## Justify (驗證有效性)
 [colab trial](https://colab.research.google.com/drive/1Wjz48Uws6-VuSHv_-DcWLilv77-AaYgj)   
 ```python
-from transformers import *
+from transformers import AutoTokenizer, AlbertForMaskedLM
 import torch
 from torch.nn.functional import softmax
 
-pretrained = 'voidful/albert_chinese_base'
-tokenizer = BertTokenizer.from_pretrained(pretrained)
+pretrained = './albert_chinese_base'
+tokenizer = AutoTokenizer.from_pretrained(pretrained)
 model = AlbertForMaskedLM.from_pretrained(pretrained)
 
 inputtext = "今天[MASK]情很好"
@@ -33,11 +39,11 @@ inputtext = "今天[MASK]情很好"
 maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)
 
 input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
-outputs = model(input_ids, masked_lm_labels=input_ids)
+outputs = model(input_ids, labels=input_ids)
 loss, prediction_scores = outputs[:2]
-logit_prob = softmax(prediction_scores[0, maskpos]).data.tolist()
+logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
 predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
 predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
-print(predicted_token,logit_prob[predicted_index])
+print(predicted_token, logit_prob[predicted_index])
 ```
 Result: `感 0.36333346366882324`   

config.json

@@ -1,4 +1,7 @@
 {
+  "architectures": [
+    "AlbertForMaskedLM"
+  ],
   "attention_probs_dropout_prob": 0,
   "bos_token_id": 2,
   "classifier_dropout_prob": 0.1,

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "special_tokens_map_file": null, "tokenizer_file": null, "name_or_path": "voidful/albert_chinese_base", "tokenizer_class": "BertTokenizer"}