Post
some mandatory hello world as first post 🤗
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
tokenizer.encode("Hello world~ Hello 2024").tokens
- 2 replies