--- language: - ru library_name: sentence-transformers tags: - sentence-transformers - sentence-similarity - feature-extraction widget: [] pipeline_tag: sentence-similarity license: apache-2.0 datasets: - deepvk/ru-HNP - deepvk/ru-WANLI - Shitao/bge-m3-data - RussianNLP/russian_super_glue - reciTAL/mlsum - Milana/russian_keywords - IlyaGusev/gazeta - d0rj/gsm8k-ru - bragovo/dsum_ru - CarlBrendt/Summ_Dialog_News --- # USER-bge-m3 **U**niversal **S**entence **E**ncoder for **R**ussian (USER) is a [sentence-transformer](https://www.SBERT.net) model for extracting embeddings exclusively for Russian language. It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is initialized from [`TatonkaHF/bge-m3_en_ru`](https://huggingface.co/TatonkaHF/bge-m3_en_ru) which is shrinked version of [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) model and trained to work mainly with the Russian language. Its quality on other languages was not evaluated. ## Usage Using this model becomes easy when you have [`sentence-transformers`](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer input_texts = [ "Когда был спущен на воду первый миноносец «Спокойный»?", "Есть ли нефть в Удмуртии?", "Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года.", "Нефтепоисковые работы в Удмуртии были начаты сразу после Второй мировой войны в 1945 году и продолжаются по сей день. Добыча нефти началась в 1967 году." ] model = SentenceTransformer("deepvk/USER-bge-m3") embeddings = model.encode(input_texts, normalize_embeddings=True) ``` However, you can use model directly with [`transformers`](https://huggingface.co/docs/transformers/en/index) ```python import torch.nn.functional as F from torch import Tensor, inference_mode from transformers import AutoTokenizer, AutoModel input_texts = [ "Когда был спущен на воду первый миноносец «Спокойный»?", "Есть ли нефть в Удмуртии?", "Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года.", "Нефтепоисковые работы в Удмуртии были начаты сразу после Второй мировой войны в 1945 году и продолжаются по сей день. Добыча нефти началась в 1967 году." ] tokenizer = AutoTokenizer.from_pretrained("deepvk/USER-bge-m3") model = AutoModel.from_pretrained("deepvk/USER-bge-m3") model.eval() encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') with torch.no_grad(): model_output = model(**encoded_input) # Perform pooling. In this case, cls pooling. sentence_embeddings = model_output[0][:, 0] # normalize embeddings sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1) # [[0.5567, 0.3014], # [0.1701, 0.7122]] scores = (sentence_embeddings[:2] @ sentence_embeddings[2:].T) ``` Also, you can use native [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) library for evaluation. Usage is described in [`bge-m3` model card](https://huggingface.co/BAAI/bge-m3). # Training Details We follow the [`USER-base`](https://huggingface.co/deepvk/USER-base) model training algorithm, with several changes as we use different backbone. **Initialization:** [`TatonkaHF/bge-m3_en_ru`](https://huggingface.co/TatonkaHF/bge-m3_en_ru) – shrinked version of [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) to support only Russian and English tokens. **Fine-tuning:** Supervised fine-tuning two different models based on data symmetry and then merging via [`LM-Cocktail`](https://arxiv.org/abs/2311.13534): 1. Since we split the data, we could additionally apply the [AnglE loss](https://arxiv.org/abs/2309.12871) to the symmetric model, which enhances performance on symmetric tasks. 2. Finally, we added the original `bge-m3` model to the two obtained models to prevent catastrophic forgetting, tuning the weights for the merger using `LM-Cocktail` to produce the final model, **USER-bge-m3**. ### Dataset During model development, we additional collect 2 datasets: [`deepvk/ru-HNP`](https://huggingface.co/datasets/deepvk/ru-HNP) and [`deepvk/ru-WANLI`](https://huggingface.co/datasets/deepvk/ru-WANLI). | Symmetric Dataset | Size | Asymmetric Dataset | Size | |-------------------|-------|--------------------|------| | **AllNLI** | 282 644 | [**MIRACL**](https://huggingface.co/datasets/Shitao/bge-m3-data/tree/main) | 10 000 | | [MedNLI](https://github.com/jgc128/mednli) | 3 699 | [MLDR](https://huggingface.co/datasets/Shitao/bge-m3-data/tree/main) | 1 864 | | [RCB](https://huggingface.co/datasets/RussianNLP/russian_super_glue) | 392 | [Lenta](https://github.com/yutkin/Lenta.Ru-News-Dataset) | 185 972 | | [Terra](https://huggingface.co/datasets/RussianNLP/russian_super_glue) | 1 359 | [Mlsum](https://huggingface.co/datasets/reciTAL/mlsum) | 51 112 | | [Tapaco](https://huggingface.co/datasets/tapaco) | 91 240 | [Mr-TyDi](https://huggingface.co/datasets/Shitao/bge-m3-data/tree/main) | 536 600 | | [**deepvk/ru-WANLI**](https://huggingface.co/datasets/deepvk/ru-WANLI) | 35 455 | [Panorama](https://huggingface.co/datasets/its5Q/panorama) | 11 024 | | [**deepvk/ru-HNP**](https://huggingface.co/datasets/deepvk/ru-HNP) | 500 000 | [PravoIsrael](https://huggingface.co/datasets/TarasHu/pravoIsrael) | 26 364 | | | | [Xlsum](https://huggingface.co/datasets/csebuetnlp/xlsum) | 124 486 | | | | [Fialka-v1](https://huggingface.co/datasets/0x7o/fialka-v1) | 130 000 | | | | [RussianKeywords](https://huggingface.co/datasets/Milana/russian_keywords) | 16 461 | | | | [Gazeta](https://huggingface.co/datasets/IlyaGusev/gazeta) | 121 928 | | | | [Gsm8k-ru](https://huggingface.co/datasets/d0rj/gsm8k-ru) | 7 470 | | | | [DSumRu](https://huggingface.co/datasets/bragovo/dsum_ru) | 27 191 | | | | [SummDialogNews](https://huggingface.co/datasets/CarlBrendt/Summ_Dialog_News) | 75 700 | **Total positive pairs:** 2,240,961 **Total negative pairs:** 792,644 (negative pairs from AIINLI, MIRACL, deepvk/ru-WANLI, deepvk/ru-HNP) For all labeled datasets, we only use its training set for fine-tuning. For datasets Gazeta, Mlsum, Xlsum: pairs (title/text) and (title/summary) are combined and used as asymmetric data. `AllNLI` is an translated to Russian combination of SNLI, MNLI and ANLI. ## Experiments We compare our mode with the basic [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) on the [`encodechka`](https://github.com/avidale/encodechka) benchmark. In addition, we evaluate model on the russian subset of [`MTEB`](https://github.com/embeddings-benchmark/mteb) on Classification, Reranking, Multilabel Classification, STS, Retrieval, and PairClassification tasks. We use validation scripts from the official repositories for each of the tasks. Results on encodechka: | Model | Mean S | Mean S+W | STS | PI | NLI | SA | TI | IA | IC | ICX | NE1 | NE2 | |-------------|--------|----------|------|------|------|------|------|------|------|------|------|------| | [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) | 0.787 | 0.696 | 0.86 | 0.75 | 0.51 | 0.82 | 0.97 | 0.79 | 0.81 | 0.78 | 0.24 | 0.42 | | `USER-bge-m3` | **0.799** | **0.709** | **0.87** | **0.76** | **0.58** | 0.82 | 0.97 | 0.79 | 0.81 | 0.78 | **0.28** | **0.43** | Results on MTEB: | Type | [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) | `USER-bge-m3` | |---------------------------|--------|-------------| | Average (30 datasets) | 0.689 | **0.706** | | Classification Average (12 datasets) | 0.571 | **0.594** | | Reranking Average (2 datasets) | **0.698** | 0.688 | | MultilabelClassification (2 datasets) | 0.343 | **0.359** | | STS Average (4 datasets) | 0.735 | **0.753** | | Retrieval Average (6 datasets) | **0.945** | 0.934 | | PairClassification Average (4 datasets) | 0.784 | **0.833** | ## Limitations We did not thoroughly evaluate the model's ability for sparse and multi-vec encoding. ## Citations ``` @misc{deepvk2024user, title={USER: Universal Sentence Encoder for Russian}, author={Malashenko, Boris and Zemerov, Anton and Spirin, Egor}, url={https://huggingface.co/datasets/deepvk/USER-base}, publisher={Hugging Face} year={2024}, } ```