pszemraj
/

claude-tokenizer-mlm

Inference Endpoints

Model card Files Files and versions Community

pszemraj commited on Mar 14

Commit

986f197

•

1 Parent(s): e28096f

Update README.md

Files changed (1) hide show

README.md +2 -0

README.md CHANGED Viewed

@@ -18,4 +18,6 @@ text = "Hello, this is a test input."
 ids = tokenizer(text)
 print(tokenizer.decode(ids['input_ids'], skip_special_tokens=False))
 # <bos>Hello, this is a test input.<EOT>
 ```

 ids = tokenizer(text)
 print(tokenizer.decode(ids['input_ids'], skip_special_tokens=False))
 # <bos>Hello, this is a test input.<EOT>
+len(tokenizer)
+# 65004
 ```