Use FastTokenizer

README.md

@@ -11,14 +11,12 @@ This repository provides Japanese language models trained by [SB Intuitions](htt
 
 ## How to use
 
-Please set **use_fast=False** to use our tokenizer properly.
-
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
  
 model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina1-65b", torch_dtype=torch.float16, device_map="auto")
-tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina1-65b", use_fast=False)
+tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina1-65b")
 generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
 set_seed(123)
  

tokenizer_config.json

@@ -9,7 +9,8 @@
     "mask_token": "<mask>",
     "cls_token": "<cls>",
     "sep_token": "<sep>",
+    "padding_side": "left",
     "sp_model_kwargs": {},
     "special_tokens_map_file": null,
-    "tokenizer_class": "T5Tokenizer"
+    "tokenizer_class": "PreTrainedTokenizerFast"
 }