Spaces:

yenniejun
/

tokenizers-languages

Running

App Files Files Community

tokenizers-languages

Commit History

Add barchart showing shortest/longest median tokenized text

40b05d1
verified

yenniejun commited on 8 days ago

Update app.py

111053f
verified

yenniejun commited on 8 days ago

Update app.py

94a7df7
verified

yenniejun commited on 8 days ago

Update requirements.txt

6dcf310
verified

yenniejun commited on 8 days ago

update huggingface package

803e0b6
verified

yenniejun commited on 8 days ago

update streamlit

1f834cd
verified

yenniejun commited on 8 days ago

fix tabs

cb5b76e
verified

yenniejun commited on 8 days ago

fix tab

ba38c3f
verified

yenniejun commited on 8 days ago

Upload 2 files

8c475b6
verified

yenniejun commited on 8 days ago

Updating link to blog in app

c4e64d6

yenniejun commited on Sep 29, 2023

Updating link to blog post

e7c169f

yenniejun commited on Sep 29, 2023

Updating the About the Project section

96bd252

Yennie Jun commited on Sep 20, 2023

Adding yaml to top of README

21a3b4f

yenniejun commited on Jun 28, 2023

Merge branch 'main' of https://github.com/yenniejun/tokenizers-languages into main

7b528cd

yenniejun commited on May 25, 2023

Adding images for README

cd2f205

yenniejun commited on May 25, 2023

Modifying README

ac19006

yenniejun commited on May 25, 2023

Adding license

18991e3

yenniejun commited on May 25, 2023

Merge pull request #1 from yenniejun/yenniejun-patch-1

dc53228
unverified

yenniejun commited on May 25, 2023

Create sync_hf_hub.yaml for syncing repo to HF

701efbc
unverified

yenniejun commited on May 25, 2023

Rearranging some of the figures

5943f5c

yenniejun commited on Apr 30, 2023

Rename text in button for refreshing

26e280e

yenniejun commited on Apr 28, 2023

Adding example texts to show

62551b2

yenniejun commited on Apr 28, 2023

Update explanation message and link to amazon

8641796

yenniejun commited on Apr 27, 2023

Adding explanation for project

616477f

yenniejun commited on Apr 27, 2023

Remove divider

fe5c81b

yenniejun commited on Apr 27, 2023

Update small formatting things:

c1ad373

yenniejun commited on Apr 27, 2023

Make label visibility collapsed for some options

aaff21f

yenniejun commited on Apr 27, 2023

clean up the figure, add data caching, add headers

535e678

yenniejun commited on Apr 27, 2023

update code to match updated data with pre-calculated token lens

29daff8

yenniejun commited on Apr 27, 2023

Updating data file with tokenizer token numbers

0a9ef68

yenniejun commited on Apr 27, 2023

Adding requirements

fdbf317

yenniejun commited on Apr 27, 2023

rename to app.py

87987fe

yenniejun commited on Apr 27, 2023

Adding main file

d0ab3b0

yenniejun commited on Apr 27, 2023

Adding dataset

8a3ab6c

yenniejun commited on Apr 27, 2023

initial commit

3cccf5e

yenniejun commited on Apr 26, 2023