Commits · eson/tokenizer-arena

remove lru_cache: has some bugs

f652c69

eson commited on 1 day ago

update

97354e0

eson commited on 2 days ago

update

d3c1316

eson commited on 2 days ago

update

5a684a6

eson commited on 2 days ago

update

42127b1

eson commited on 2 days ago

remove vocabs; update compression_app; add character_app;

2bd606a

eson commited on 2 days ago

update

f331792

eson commited on 22 days ago

update

9558ae0

eson commited on 22 days ago

add compression leaderboard

1b7fc74

eson commited on 22 days ago

update compress rate

367a536

eson commited on 30 days ago

update compress rate

988921c

eson commited on 30 days ago

update

7d2062e

eson commited on 30 days ago

add grok mixtral

480ae5d

eson commited on 30 days ago

add compress rate

814ee6b

eson commited on about 1 month ago

add zephyr

a6aee1d

eson commited on Mar 15

requi

6b70021

eson commited on Mar 13

requi

510279b

eson commited on Mar 13

requi

7b522e7

eson commited on Mar 13

config python version

11379e2

eson commited on Mar 13

add xlm-roberta

057bc67

eson commited on Mar 13

add amber and crystal_coder

5db13e0

eson commited on Mar 8

add character glm

f0f84b2

eson commited on Mar 5

update

f02dd94

eson commited on Mar 4

fix PyO3PanicException

2461705

eson commited on Mar 4

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

bce41d0

eson commited on Mar 4

fix fastchat_t5_3b

c766a08

eson commited on Mar 4

fix tiktoken special tokens

adcfb97

eson commited on Mar 1

add aya

44c3329

eson commited on Feb 28

fix olmo

2442c83

eson commited on Feb 28

add olmo tokenizer

bbefe94

eson commited on Feb 28

update

24b4aa5

eson commited on Feb 26

update

1f833af

eson commited on Feb 26

fix tiktoken

a6c67ec

eson commited on Feb 25

fix gemma_7b

7011963

eson commited on Feb 22

add gemma_7b

9c8ace5

eson commited on Feb 22

add more tokenizer

5425d5d

eson commited on Feb 22

fix tokenize

e6543ac

eson commited on Feb 21

add more tokenizer

c75633b

eson commited on Jan 31

update

6bdf6c6

eson commited on Jan 31

update

9820e00

eson commited on Jan 31

fix chatglm; new feature about add_special_tokens;

d27a756

eson commited on Jan 31

update

a37f943

eson commited on Jan 26

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena

0415b36

eson commited on Jan 26

add more tokenizer

d2417c7

eson commited on Jan 26

Update README.md

fab95c3

eson commited on Dec 20, 2023

update

ae282a4

eson commited on Dec 12, 2023

add more tokenizers

a1b0cd0

eson commited on Dec 6, 2023

add more tokenizer

3030d21

eson commited on Dec 5, 2023

add more tokenizer

293bad6

eson commited on Dec 5, 2023

fix moss

aa0c637

eson commited on Dec 4, 2023

Commit History

remove lru_cache: has some bugs f652c69

update 97354e0

update d3c1316

update 5a684a6

update 42127b1

remove vocabs; update compression_app; add character_app; 2bd606a

update f331792

update 9558ae0

add compression leaderboard 1b7fc74

update compress rate 367a536

update compress rate 988921c

update 7d2062e

add grok mixtral 480ae5d

add compress rate 814ee6b

add zephyr a6aee1d

requi 6b70021

requi 510279b

requi 7b522e7

config python version 11379e2

add xlm-roberta 057bc67

add amber and crystal_coder 5db13e0

add character glm f0f84b2

update f02dd94

fix PyO3PanicException 2461705

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name bce41d0

fix fastchat_t5_3b c766a08

fix tiktoken special tokens adcfb97

add aya 44c3329

fix olmo 2442c83

add olmo tokenizer bbefe94

update 24b4aa5

update 1f833af

fix tiktoken a6c67ec

fix gemma_7b 7011963

add gemma_7b 9c8ace5

add more tokenizer 5425d5d

fix tokenize e6543ac

add more tokenizer c75633b

update 6bdf6c6

update 9820e00

fix chatglm; new feature about add_special_tokens; d27a756

update a37f943

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena 0415b36

add more tokenizer d2417c7

Update README.md fab95c3

update ae282a4

add more tokenizers a1b0cd0

add more tokenizer 3030d21

add more tokenizer 293bad6

fix moss aa0c637

remove lru_cache: has some bugs

f652c69

update

97354e0

update

d3c1316

update

5a684a6

update

42127b1

remove vocabs; update compression_app; add character_app;

2bd606a

update

f331792

update

9558ae0

add compression leaderboard

1b7fc74

update compress rate

367a536

update compress rate

988921c

update

7d2062e

add grok mixtral

480ae5d

add compress rate

814ee6b

add zephyr

a6aee1d

requi

6b70021

requi

510279b

requi

7b522e7

config python version

11379e2

add xlm-roberta

057bc67

add amber and crystal_coder

5db13e0

add character glm

f0f84b2

update

f02dd94

fix PyO3PanicException

2461705

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

bce41d0

fix fastchat_t5_3b

c766a08

fix tiktoken special tokens

adcfb97

add aya

44c3329

fix olmo

2442c83

add olmo tokenizer

bbefe94

update

24b4aa5

update

1f833af

fix tiktoken

a6c67ec

fix gemma_7b

7011963

add gemma_7b

9c8ace5

add more tokenizer

5425d5d

fix tokenize

e6543ac

add more tokenizer

c75633b

update

6bdf6c6

update

9820e00

fix chatglm; new feature about add_special_tokens;

d27a756

update

a37f943

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena

0415b36

add more tokenizer

d2417c7

Update README.md

fab95c3

update

ae282a4

add more tokenizers

a1b0cd0

add more tokenizer

3030d21

add more tokenizer

293bad6

fix moss

aa0c637