peyash
/

distilbert-base-multilingual-cased

Model card Files Files and versions Community

peyash commited on Feb 2

Commit

41d04e0

•

1 Parent(s): 0443f66

update

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -2,5 +2,5 @@
 license: apache-2.0
 ---
 This is a tokenizer only, with the following modification:
-- Replaced [unused0], [unused1], [unused2] with [ES], [DE], [FR] respectively in the vocabulary
-- Added [ES], [DE], [FR] as special tokens and therefore they won't lowercased or splitted

 license: apache-2.0
 ---
 This is a tokenizer only, with the following modification:
+- Replaced `[unused0]`, `[unused1]`, `[unused2]` with `[ES]`, `[DE]`, `[FR]` respectively in the vocabulary
+- Added `[ES]`, `[DE]`, `[FR]` as special tokens and therefore they won't lowercased or splitted