How are sequence lengths measured?

#47
by ynnoj - opened

What is the unit for the sequence lengths displayed? Are these tokens or characters?

Massive Text Embedding Benchmark org

tokens

Thank you!

ynnoj changed discussion status to closed

Sign up or log in to comment