Commits · yhavinga/dutch-tokenizer-arena

Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.

c78da21

yhavinga commited on 23 days ago

Pin protobuf dep

7d83c88

yhavinga commited on Mar 10

Hack some Dutch tokenizers into it

55df72d

yhavinga commited on Jan 2

update

f331792

eson commited on 27 days ago

update

9558ae0

eson commited on 27 days ago

add compression leaderboard

1b7fc74

eson commited on 27 days ago

update compress rate

367a536

eson commited on Apr 23

update compress rate

988921c

eson commited on Apr 23

update

7d2062e

eson commited on Apr 23

add grok mixtral

480ae5d

eson commited on Apr 23

add compress rate

814ee6b

eson commited on Apr 22

add zephyr

a6aee1d

eson commited on Mar 15

requi

6b70021

eson commited on Mar 13

requi

510279b

eson commited on Mar 13

requi

7b522e7

eson commited on Mar 13

config python version

11379e2

eson commited on Mar 13

add xlm-roberta

057bc67

eson commited on Mar 13

add amber and crystal_coder

5db13e0

eson commited on Mar 8

add character glm

f0f84b2

eson commited on Mar 5

update

f02dd94

eson commited on Mar 4

fix PyO3PanicException

2461705

eson commited on Mar 4

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

bce41d0

eson commited on Mar 4

fix fastchat_t5_3b

c766a08

eson commited on Mar 4

fix tiktoken special tokens

adcfb97

eson commited on Mar 1

add aya

44c3329

eson commited on Feb 28

fix olmo

2442c83

eson commited on Feb 28

add olmo tokenizer

bbefe94

eson commited on Feb 28

update

24b4aa5

eson commited on Feb 26

update

1f833af

eson commited on Feb 26

fix tiktoken

a6c67ec

eson commited on Feb 25

fix gemma_7b

7011963

eson commited on Feb 22

add gemma_7b

9c8ace5

eson commited on Feb 22

add more tokenizer

5425d5d

eson commited on Feb 22

fix tokenize

e6543ac

eson commited on Feb 21

add more tokenizer

c75633b

eson commited on Jan 31

update

6bdf6c6

eson commited on Jan 31

update

9820e00

eson commited on Jan 31

fix chatglm; new feature about add_special_tokens;

d27a756

eson commited on Jan 31

update

a37f943

eson commited on Jan 26

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena

0415b36

eson commited on Jan 26

add more tokenizer

d2417c7

eson commited on Jan 26

Update README.md

fab95c3

eson commited on Dec 20, 2023

update

ae282a4

eson commited on Dec 12, 2023

add more tokenizers

a1b0cd0

eson commited on Dec 6, 2023

add more tokenizer

3030d21

eson commited on Dec 5, 2023

add more tokenizer

293bad6

eson commited on Dec 5, 2023

fix moss

aa0c637

eson commited on Dec 4, 2023

update

da93e39

eson commited on Dec 4, 2023

update

2d550af

eson commited on Dec 1, 2023

add skywork

c7ed4a2

eson commited on Dec 1, 2023

Commit History

Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX. c78da21

Pin protobuf dep 7d83c88

Hack some Dutch tokenizers into it 55df72d

update f331792

update 9558ae0

add compression leaderboard 1b7fc74

update compress rate 367a536

update compress rate 988921c

update 7d2062e

add grok mixtral 480ae5d

add compress rate 814ee6b

add zephyr a6aee1d

requi 6b70021

requi 510279b

requi 7b522e7

config python version 11379e2

add xlm-roberta 057bc67

add amber and crystal_coder 5db13e0

add character glm f0f84b2

update f02dd94

fix PyO3PanicException 2461705

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name bce41d0

fix fastchat_t5_3b c766a08

fix tiktoken special tokens adcfb97

add aya 44c3329

fix olmo 2442c83

add olmo tokenizer bbefe94

update 24b4aa5

update 1f833af

fix tiktoken a6c67ec

fix gemma_7b 7011963

add gemma_7b 9c8ace5

add more tokenizer 5425d5d

fix tokenize e6543ac

add more tokenizer c75633b

update 6bdf6c6

update 9820e00

fix chatglm; new feature about add_special_tokens; d27a756

update a37f943

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena 0415b36

add more tokenizer d2417c7

Update README.md fab95c3

update ae282a4

add more tokenizers a1b0cd0

add more tokenizer 3030d21

add more tokenizer 293bad6

fix moss aa0c637

update da93e39

update 2d550af

add skywork c7ed4a2

Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.

c78da21

Pin protobuf dep

7d83c88

Hack some Dutch tokenizers into it

55df72d

update

f331792

update

9558ae0

add compression leaderboard

1b7fc74

update compress rate

367a536

update compress rate

988921c

update

7d2062e

add grok mixtral

480ae5d

add compress rate

814ee6b

add zephyr

a6aee1d

requi

6b70021

requi

510279b

requi

7b522e7

config python version

11379e2

add xlm-roberta

057bc67

add amber and crystal_coder

5db13e0

add character glm

f0f84b2

update

f02dd94

fix PyO3PanicException

2461705

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

bce41d0

fix fastchat_t5_3b

c766a08

fix tiktoken special tokens

adcfb97

add aya

44c3329

fix olmo

2442c83

add olmo tokenizer

bbefe94

update

24b4aa5

update

1f833af

fix tiktoken

a6c67ec

fix gemma_7b

7011963

add gemma_7b

9c8ace5

add more tokenizer

5425d5d

fix tokenize

e6543ac

add more tokenizer

c75633b

update

6bdf6c6

update

9820e00

fix chatglm; new feature about add_special_tokens;

d27a756

update

a37f943

Merge branch 'main' of hf.co:spaces/eson/tokenizer-arena

0415b36

add more tokenizer

d2417c7

Update README.md

fab95c3

update

ae282a4

add more tokenizers

a1b0cd0

add more tokenizer

3030d21

add more tokenizer

293bad6

fix moss

aa0c637

update

da93e39

update

2d550af

add skywork

c7ed4a2