update auto tokenizer support

README.md

@@ -1,31 +1,35 @@
 ---
 language: zh
+pipeline_tag: fill-mask
+widget:
+- text: "今天[MASK]情很好"
 ---
 
+
 # albert_chinese_xlarge
 
 This a albert_chinese_xlarge model from [Google's github](https://github.com/google-research/ALBERT)  
 converted by huggingface's [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py)
 
-## Attention (注意)
+## Notice
+*Support AutoTokenizer*
 
-Since sentencepiece is not used in albert_chinese_xlarge model   
+Since sentencepiece is not used in albert_chinese_base model   
 you have to call BertTokenizer instead of AlbertTokenizer !!!    
 we can eval it using an example on MaskedLM   
    
-由於 albert_chinese_xlarge 模型沒有用 sentencepiece   
+由於 albert_chinese_base 模型沒有用 sentencepiece   
 用AlbertTokenizer會載不進詞表，因此需要改用BertTokenizer !!!   
 我們可以跑MaskedLM預測來驗證這個做法是否正確   
    
 ## Justify (驗證有效性)
-[colab trial](https://colab.research.google.com/drive/1Wjz48Uws6-VuSHv_-DcWLilv77-AaYgj)   
 ```python
-from transformers import *
+from transformers import AutoTokenizer, AlbertForMaskedLM
 import torch
 from torch.nn.functional import softmax
 
 pretrained = 'voidful/albert_chinese_xlarge'
-tokenizer = BertTokenizer.from_pretrained(pretrained)
+tokenizer = AutoTokenizer.from_pretrained(pretrained)
 model = AlbertForMaskedLM.from_pretrained(pretrained)
 
 inputtext = "今天[MASK]情很好"
@@ -33,11 +37,11 @@ inputtext = "今天[MASK]情很好"
 maskpos = tokenizer.encode(inputtext, add_special_tokens=True).index(103)
 
 input_ids = torch.tensor(tokenizer.encode(inputtext, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
-outputs = model(input_ids, masked_lm_labels=input_ids)
+outputs = model(input_ids, labels=input_ids)
 loss, prediction_scores = outputs[:2]
-logit_prob = softmax(prediction_scores[0, maskpos]).data.tolist()
+logit_prob = softmax(prediction_scores[0, maskpos],dim=-1).data.tolist()
 predicted_index = torch.argmax(prediction_scores[0, maskpos]).item()
 predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
-print(predicted_token,logit_prob[predicted_index])
+print(predicted_token, logit_prob[predicted_index])
 ```
 Result: `心 0.9942440390586853`   

config.json

@@ -1,4 +1,7 @@
 {
+  "architectures": [
+    "AlbertForMaskedLM"
+  ],
   "attention_probs_dropout_prob": 0,
   "bos_token_id": 2,
   "classifier_dropout_prob": 0.1,

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "special_tokens_map_file": null, "tokenizer_file": null, "name_or_path": "voidful/albert_chinese_xlarge", "tokenizer_class": "BertTokenizer"}