Spaces:

eson
/

tokenizer-arena

Running

tokenizer-arena / utils /digit_util.py

add compress rate

814ee6b 2 months ago

No virus

50 Bytes

	"""

	qwen segments numbers by single digits.


	"""