Spaces:

Annorita
/

tokenizer_comparison

Sleeping

Annorita commited on Jan 24

Commit

e58cb95

•

1 Parent(s): 1c512f5

Update app.py

Files changed (1) hide show

app.py CHANGED Viewed

@@ -12,6 +12,7 @@ model_option = ['deepseek-ai/deepseek-coder-1.3b-instruct',
              'microsoft/phi-2',
              'mistralai/Mistral-7B-Instruct-v0.2',
              'codellama/CodeLlama-7b-hf',
              'gpt2'
              'enter by myself']

              'microsoft/phi-2',
              'mistralai/Mistral-7B-Instruct-v0.2',
              'codellama/CodeLlama-7b-hf',
+             'hf-internal-testing/llama-tokenizer',
              'gpt2'
              'enter by myself']