unfair comparison

#20
by raulcarlomagno - opened

i think that some of the models on the board are multilingual or maybe they tackle a few languages (english and spanish for example)
these kind of models will perform worse than just plain english models, so maybe a new column specifying how many languages besides language english it supports, will lead to a more fair comparison in a quick look to the leaderboard

Massive Text Embedding Benchmark org

Good point. The problem is what language is model supports is ambiguous, see https://huggingface.co/spaces/mteb/leaderboard/discussions/15#64ad01bc6864362a7e037688

I think the better approach is to have separate leaderboards for as many languages as possible where this can be directly assessed. Many of the task tabs already have languages beyond English!

Sign up or log in to comment