How this model count the token size?

#10

by WeiZhenKun - opened Aug 18, 2023

Aug 18, 2023

•

How this model count the token size?
Is there a certain proportional relationship between the token size and the length of characters?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment