Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

Is there any documentation within this leaderboard?

by zhiminy - opened Jun 29

Discussion

zhiminy

Jun 29

•

edited Jun 29

I cannot locate any specification in this space...What is this leaderboard used for?

xu-song

Owner Jul 13

Thanks for your attention. A brief document has been added to the demo.
The leaderboard aims to evaluate tokenizer performance on different languages.

Lower oov_ratio refers to less out-of-vocabulary tokens.
Higher char/token means less words be segmented into subwords.

xu-song changed discussion status to closed Jul 22

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment