add tokenizer

README.md

@@ -1,3 +1,8 @@
 ---
-license: mit
+license: bigscience-bloom-rail-1.0
 ---
+# Tokenizer used for all BLOOM models
+
+Tokenizer information are provided at [https://huggingface.co/bigscience/bloom#preprocessing](https://huggingface.co/bigscience/bloom#preprocessing)
+
+TODO: point to paper once it comes out with extra details on the tokenizer

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": "<unk>", "eos_token": "</s>", "bos_token": "<s>", "pad_token": "<pad>", "name_or_path": "bigscience/tokenizer", "special_tokens_map_file": null, "tokenizer_class": "PreTrainedTokenizerFast"}