Hugging Face's logo

Spaces:
shreeyad
/
tokenizers-multilingual
Sleeping

App Files Community
shreeyad commited on
Commit
502e939
•
1 Parent(s): 256cb1a

added vocab

Browse files
Files changed (6) hide show
  1. .gitignore +0 -1
  2. vocab/gpt-4-vocab-err.txt +0 -0
  3. vocab/gpt-4-vocab.txt +0 -0
  4. vocab/nllb-vocab.txt +0 -0
  5. vocab/roberta-vocab.txt +0 -0
  6. vocab/tiktoken_cache/9b5ad71b2ce5302211f9c61530b329a4922fc6a4 +0 -0
.gitignore CHANGED
@@ -3,6 +3,5 @@ data/aya_dataset_segments.csv
3
  data/aya_dataset_features_large.csv
4
 
5
  scripts/
6
- vocab/
7
  data/.DS_Store
8
  .DS_Store
 
3
  data/aya_dataset_features_large.csv
4
 
5
  scripts/
 
6
  data/.DS_Store
7
  .DS_Store
vocab/gpt-4-vocab-err.txt ADDED
Binary file (745 kB). View file
 
vocab/gpt-4-vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
vocab/nllb-vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
vocab/roberta-vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
vocab/tiktoken_cache/9b5ad71b2ce5302211f9c61530b329a4922fc6a4 ADDED
The diff for this file is too large to render. See raw diff