Быстрая модель BERT для расчетов эмбеддингов предложений на русском языке. Модель основана на cointegrated/rubert-tiny2 - имеет аналогичные размеры контекста (2048), ембеддинга (312) и быстродействие.
Использование
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('sergeyzh/rubert-tiny-turbo')
sentences = ["привет мир", "hello world", "здравствуй вселенная"]
embeddings = model.encode(sentences)
print(util.dot_score(embeddings, embeddings))
Метрики
Оценки модели на бенчмарке encodechka:
model |
CPU |
GPU |
size |
Mean S |
Mean S+W |
dim |
sergeyzh/LaBSE-ru-turbo |
120.40 |
8.05 |
490 |
0.789 |
0.702 |
768 |
BAAI/bge-m3 |
523.40 |
22.50 |
2166 |
0.787 |
0.696 |
1024 |
intfloat/multilingual-e5-large |
506.80 |
30.80 |
2136 |
0.780 |
0.686 |
1024 |
intfloat/multilingual-e5-base |
130.61 |
14.39 |
1061 |
0.761 |
0.669 |
768 |
sergeyzh/rubert-tiny-turbo |
5.51 |
3.25 |
111 |
0.749 |
0.667 |
312 |
intfloat/multilingual-e5-small |
40.86 |
12.09 |
449 |
0.742 |
0.645 |
384 |
cointegrated/rubert-tiny2 |
5.51 |
3.25 |
111 |
0.704 |
0.638 |
312 |
model |
STS |
PI |
NLI |
SA |
TI |
IA |
IC |
ICX |
NE1 |
NE2 |
sergeyzh/LaBSE-ru-turbo |
0.864 |
0.748 |
0.490 |
0.814 |
0.974 |
0.806 |
0.815 |
0.801 |
0.305 |
0.404 |
BAAI/bge-m3 |
0.864 |
0.749 |
0.510 |
0.819 |
0.973 |
0.792 |
0.809 |
0.783 |
0.240 |
0.422 |
intfloat/multilingual-e5-large |
0.862 |
0.727 |
0.473 |
0.810 |
0.979 |
0.798 |
0.819 |
0.773 |
0.224 |
0.374 |
intfloat/multilingual-e5-base |
0.835 |
0.704 |
0.459 |
0.796 |
0.964 |
0.783 |
0.802 |
0.738 |
0.235 |
0.376 |
sergeyzh/rubert-tiny-turbo |
0.828 |
0.722 |
0.476 |
0.787 |
0.955 |
0.757 |
0.780 |
0.685 |
0.305 |
0.373 |
intfloat/multilingual-e5-small |
0.822 |
0.714 |
0.457 |
0.758 |
0.957 |
0.761 |
0.779 |
0.691 |
0.234 |
0.275 |
cointegrated/rubert-tiny2 |
0.750 |
0.651 |
0.417 |
0.737 |
0.937 |
0.746 |
0.757 |
0.638 |
0.360 |
0.386 |
Оценки модели на бенчмарке ruMTEB:
Model Name |
Metric |
sbert_large_ mt_nlu_ru |
sbert_large_ nlu_ru |
rubert-tiny2 |
rubert-tiny-turbo |
multilingual-e5-small |
multilingual-e5-base |
multilingual-e5-large |
CEDRClassification |
Accuracy |
0.368 |
0.358 |
0.369 |
0.390 |
0.401 |
0.423 |
0.448 |
GeoreviewClassification |
Accuracy |
0.397 |
0.400 |
0.396 |
0.414 |
0.447 |
0.461 |
0.497 |
GeoreviewClusteringP2P |
V-measure |
0.584 |
0.590 |
0.442 |
0.597 |
0.586 |
0.545 |
0.605 |
HeadlineClassification |
Accuracy |
0.772 |
0.793 |
0.742 |
0.686 |
0.732 |
0.757 |
0.758 |
InappropriatenessClassification |
Accuracy |
0.646 |
0.625 |
0.586 |
0.591 |
0.592 |
0.588 |
0.616 |
KinopoiskClassification |
Accuracy |
0.503 |
0.495 |
0.491 |
0.505 |
0.500 |
0.509 |
0.566 |
RiaNewsRetrieval |
NDCG@10 |
0.214 |
0.111 |
0.140 |
0.513 |
0.700 |
0.702 |
0.807 |
RuBQReranking |
MAP@10 |
0.561 |
0.468 |
0.461 |
0.622 |
0.715 |
0.720 |
0.756 |
RuBQRetrieval |
NDCG@10 |
0.298 |
0.124 |
0.109 |
0.517 |
0.685 |
0.696 |
0.741 |
RuReviewsClassification |
Accuracy |
0.589 |
0.583 |
0.570 |
0.607 |
0.612 |
0.630 |
0.653 |
RuSTSBenchmarkSTS |
Pearson correlation |
0.712 |
0.588 |
0.694 |
0.787 |
0.781 |
0.796 |
0.831 |
RuSciBenchGRNTIClassification |
Accuracy |
0.542 |
0.539 |
0.456 |
0.529 |
0.550 |
0.563 |
0.582 |
RuSciBenchGRNTIClusteringP2P |
V-measure |
0.522 |
0.504 |
0.414 |
0.481 |
0.511 |
0.516 |
0.520 |
RuSciBenchOECDClassification |
Accuracy |
0.438 |
0.430 |
0.355 |
0.415 |
0.427 |
0.423 |
0.445 |
RuSciBenchOECDClusteringP2P |
V-measure |
0.473 |
0.464 |
0.381 |
0.411 |
0.443 |
0.448 |
0.450 |
SensitiveTopicsClassification |
Accuracy |
0.285 |
0.280 |
0.220 |
0.244 |
0.228 |
0.234 |
0.257 |
TERRaClassification |
Average Precision |
0.520 |
0.502 |
0.519 |
0.563 |
0.551 |
0.550 |
0.584 |
Model Name |
Metric |
sbert_large_ mt_nlu_ru |
sbert_large_ nlu_ru |
rubert-tiny2 |
rubert-tiny-turbo |
multilingual-e5-small |
multilingual-e5-base |
multilingual-e5-large |
Classification |
Accuracy |
0.554 |
0.552 |
0.514 |
0.535 |
0.551 |
0.561 |
0.588 |
Clustering |
V-measure |
0.526 |
0.519 |
0.412 |
0.496 |
0.513 |
0.503 |
0.525 |
MultiLabelClassification |
Accuracy |
0.326 |
0.319 |
0.294 |
0.317 |
0.314 |
0.329 |
0.353 |
PairClassification |
Average Precision |
0.520 |
0.502 |
0.519 |
0.563 |
0.551 |
0.550 |
0.584 |
Reranking |
MAP@10 |
0.561 |
0.468 |
0.461 |
0.622 |
0.715 |
0.720 |
0.756 |
Retrieval |
NDCG@10 |
0.256 |
0.118 |
0.124 |
0.515 |
0.697 |
0.699 |
0.774 |
STS |
Pearson correlation |
0.712 |
0.588 |
0.694 |
0.787 |
0.781 |
0.796 |
0.831 |
Average |
Average |
0.494 |
0.438 |
0.431 |
0.548 |
0.588 |
0.594 |
0.630 |