Update README.md
Browse files
README.md
CHANGED
@@ -15,4 +15,5 @@ You can load the tokenizer from huggingface hub as follows.
|
|
15 |
from transformers import AutoTokenizer
|
16 |
|
17 |
tokenizer = AutoTokenizer.from_pretrained("rasyosef/gpt2-oscar-amharic-tokenizer")
|
|
|
18 |
```
|
|
|
15 |
from transformers import AutoTokenizer
|
16 |
|
17 |
tokenizer = AutoTokenizer.from_pretrained("rasyosef/gpt2-oscar-amharic-tokenizer")
|
18 |
+
tokenizer("α α£αα α«αα¨ α¨ααα α²α¬α΅ α₯α½ααααα’")
|
19 |
```
|