--- pipeline_tag: sentence-similarity language: - pl tags: - sentence-transformers - feature-extraction - sentence-similarity - transformers datasets: - ipipan/polqa - ipipan/maupqa license: cc-by-sa-4.0 widget: - source_sentence: "Pytanie: W jakim mieście urodził się Zbigniew Herbert?" sentences: - "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg." - "Zbigniew HerbertLato 1968 Herbert spędził w USA (na zaproszenie Poetry Center)." - "Herbert George WellsHerbert George Wells (ur. 21 września 1866 w Bromley, zm. 13 sierpnia 1946 w Londynie) – brytyjski pisarz i biolog." example_title: "Zbigniew Herbert" --- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/5eb2c5ef4e876668a0c3779e/j2JE7_VnbRifCmV7_4BP9.png) # Silver Retriever Base (v1) Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dimensional dense vector space and can be used for tasks like document retrieval or semantic search. It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/datasets/ipipan/polqa) and [MAUPQA](https://huggingface.co/datasets/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024. Please refer to the [SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://arxiv.org/abs/2309.08469) for more details. ## Evaluation | **Model** | **Average [Acc]** | **Average [NDCG]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[Acc]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[NDCG]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[Acc]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[NDCG]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[Acc]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[NDCG]** | |--------------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:| | BM25 | 74.87 | 51.81 | 61.35 | 24.51 | 66.89 | 48.71 | 96.38 | **82.21** | | BM25 (lemma) | 80.46 | 55.44 | 71.49 | 31.97 | 75.33 | 55.70 | 94.57 | 78.65 | | [MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) | 62.62 | 39.21 | 37.24 | 11.93 | 71.67 | 51.25 | 78.97 | 54.44 | | [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) | 64.89 | 39.47 | 46.23 | 15.53 | 67.11 | 46.71 | 81.34 | 56.16 | | [mContriever-Base](https://huggingface.co/nthakur/mcontriever-base-msmarco) | 86.31 | 60.37 | 78.66 | 36.30 | 84.44 | 67.38 | 95.82 | 77.42 | | [E5-Base](https://huggingface.co/intfloat/multilingual-e5-base) | 91.58 | 66.56 | 86.61 | **46.08** | 91.89 | 75.90 | 96.24 | 77.69 | | [ST-DistilRoBERTa](https://huggingface.co/sdadas/st-polish-paraphrase-from-distilroberta) | 73.78 | 48.29 | 48.43 | 16.73 | 84.89 | 64.39 | 88.02 | 63.76 | | [ST-MPNet](https://huggingface.co/sdadas/st-polish-paraphrase-from-mpnet) | 76.66 | 49.99 | 56.80 | 21.55 | 86.00 | 65.44 | 87.19 | 62.99 | | [HerBERT-QA](https://huggingface.co/ipipan/herbert-base-qa-v1) | 84.23 | 54.36 | 75.84 | 32.52 | 85.78 | 63.58 | 91.09 | 66.99 | | [Silver Retriever v1](https://huggingface.co/ipipan/silver-retriever-base-v1) | 92.45 | 66.72 | 87.24 | 43.40 | **94.56** | 79.66 | 95.54 | 77.10 | | [Silver Retriever v1.1](https://huggingface.co/ipipan/silver-retriever-base-v1.1) | **93.18** | **67.55** | **88.60** | 44.88 | 94.00 | **79.83** | **96.94** | 77.95 | Legend: - **Acc** is the Accuracy at 10 - **NDCG** is the Normalized Discounted Cumulative Gain at 10 ## Usage ### Preparing inputs The model was trained on question-passage pairs and works best when the input is the same format as that used during training: - We added the phrase `Pytanie:` to the beginning of the question. - The training passages consisted of `title` and `text` concatenated with the special token ``. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `` token. - Although we used the dot product during training, the model usually works better with the cosine distance. ### Inference with Sentence-Transformers Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = [ "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", ] model = SentenceTransformer('ipipan/silver-retriever-base-v1') embeddings = model.encode(sentences) print(embeddings) ``` ### Inference with HuggingFace Transformers Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. ```python from transformers import AutoTokenizer, AutoModel import torch def cls_pooling(model_output, attention_mask): return model_output[0][:,0] # Sentences we want sentence embeddings for sentences = [ "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", ] # Load model from HuggingFace Hub tokenizer = AutoTokenizer.from_pretrained('ipipan/silver-retriever-base-v1') model = AutoModel.from_pretrained('ipipan/silver-retriever-base-v1') # Tokenize sentences encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') # Compute token embeddings with torch.no_grad(): model_output = model(**encoded_input) # Perform pooling. In this case, cls pooling. sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask']) print("Sentence embeddings:") print(sentence_embeddings) ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) ) ``` ## Additional Information ### Model Creators The model was created by Piotr Rybak from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/). This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19. ### Licensing Information CC BY-SA 4.0 ### Citation Information ``` @inproceedings{rybak-ogrodniczuk-2024-silver-retriever, title = "Silver Retriever: Advancing Neural Passage Retrieval for {P}olish Question Answering", author = "Rybak, Piotr and Ogrodniczuk, Maciej", editor = "Calzolari, Nicoletta and Kan, Min-Yen and Hoste, Veronique and Lenci, Alessandro and Sakti, Sakriani and Xue, Nianwen", booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)", month = may, year = "2024", address = "Torino, Italia", publisher = "ELRA and ICCL", url = "https://aclanthology.org/2024.lrec-main.1291", pages = "14826--14831", abstract = "Modern open-domain question answering systems often rely on accurate and efficient retrieval components to find passages containing the facts necessary to answer the question. Recently, neural retrievers have gained popularity over lexical alternatives due to their superior performance. However, most of the work concerns popular languages such as English or Chinese. For others, such as Polish, few models are available. In this work, we present Silver Retriever, a neural retriever for Polish trained on a diverse collection of manually or weakly labeled datasets. Silver Retriever achieves much better results than other Polish models and is competitive with larger multilingual models. Together with the model, we open-source five new passage retrieval datasets.", } ```