michaelfeil commited on
Commit
9af28c3
1 Parent(s): 0b81b1e

Upload intfloat/e5-small ctranslate2 weights

Browse files
Files changed (2) hide show
  1. README.md +61 -6
  2. model.bin +2 -2
README.md CHANGED
@@ -4,6 +4,9 @@ tags:
4
  - int8
5
  - float16
6
  - mteb
 
 
 
7
  model-index:
8
  - name: e5-small
9
  results:
@@ -2608,7 +2611,7 @@ Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on
2608
 
2609
  quantized version of [intfloat/e5-small](https://huggingface.co/intfloat/e5-small)
2610
  ```bash
2611
- pip install hf-hub-ctranslate2>=2.12.0 ctranslate2>=3.16.0
2612
  ```
2613
 
2614
  ```python
@@ -2648,16 +2651,20 @@ embeddings = model.encode(
2648
  print(embeddings.shape, embeddings)
2649
  scores = (embeddings @ embeddings.T) * 100
2650
 
 
 
 
 
2651
  ```
2652
 
2653
- Checkpoint compatible to [ctranslate2>=3.16.0](https://github.com/OpenNMT/CTranslate2)
2654
  and [hf-hub-ctranslate2>=2.12.0](https://github.com/michaelfeil/hf-hub-ctranslate2)
2655
  - `compute_type=int8_float16` for `device="cuda"`
2656
  - `compute_type=int8` for `device="cpu"`
2657
 
2658
- Converted on 2023-06-19 using
2659
  ```
2660
- ct2-transformers-converter --model intfloat/e5-small --output_dir ~/tmp-ct2fast-e5-small --force --copy_files tokenizer.json modules.json README.md tokenizer_config.json sentence_bert_config.json vocab.txt special_tokens_map.json .gitattributes --trust_remote_code
2661
  ```
2662
 
2663
  # Licence and other remarks:
@@ -2668,6 +2675,8 @@ This is just a quantized version. Licence conditions are intended to be idential
2668
 
2669
  # E5-small
2670
 
 
 
2671
  [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf).
2672
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022
2673
 
@@ -2706,7 +2715,7 @@ batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=Tru
2706
  outputs = model(**batch_dict)
2707
  embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
2708
 
2709
- # (Optionally) normalize embeddings
2710
  embeddings = F.normalize(embeddings, p=2, dim=1)
2711
  scores = (embeddings[:2] @ embeddings[2:].T) * 100
2712
  print(scores.tolist())
@@ -2721,6 +2730,52 @@ Please refer to our paper at [https://arxiv.org/pdf/2212.03533.pdf](https://arxi
2721
  Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results
2722
  on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316).
2723
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2724
  ## Citation
2725
 
2726
  If you find our paper or models helpful, please consider cite as follows:
@@ -2736,4 +2791,4 @@ If you find our paper or models helpful, please consider cite as follows:
2736
 
2737
  ## Limitations
2738
 
2739
- This model only works for English texts. Long texts will be truncated to at most 512 tokens.
 
4
  - int8
5
  - float16
6
  - mteb
7
+ - Sentence Transformers
8
+ - sentence-similarity
9
+ - sentence-transformers
10
  model-index:
11
  - name: e5-small
12
  results:
 
2611
 
2612
  quantized version of [intfloat/e5-small](https://huggingface.co/intfloat/e5-small)
2613
  ```bash
2614
+ pip install hf-hub-ctranslate2>=2.12.0 ctranslate2>=3.17.1
2615
  ```
2616
 
2617
  ```python
 
2651
  print(embeddings.shape, embeddings)
2652
  scores = (embeddings @ embeddings.T) * 100
2653
 
2654
+ # Hint: you can also host this code via REST API and
2655
+ # via github.com/michaelfeil/infinity
2656
+
2657
+
2658
  ```
2659
 
2660
+ Checkpoint compatible to [ctranslate2>=3.17.1](https://github.com/OpenNMT/CTranslate2)
2661
  and [hf-hub-ctranslate2>=2.12.0](https://github.com/michaelfeil/hf-hub-ctranslate2)
2662
  - `compute_type=int8_float16` for `device="cuda"`
2663
  - `compute_type=int8` for `device="cpu"`
2664
 
2665
+ Converted on 2023-10-13 using
2666
  ```
2667
+ LLama-2 -> removed <pad> token.
2668
  ```
2669
 
2670
  # Licence and other remarks:
 
2675
 
2676
  # E5-small
2677
 
2678
+ **News (May 2023): please switch to [e5-small-v2](https://huggingface.co/intfloat/e5-small-v2), which has better performance and same method of usage.**
2679
+
2680
  [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf).
2681
  Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022
2682
 
 
2715
  outputs = model(**batch_dict)
2716
  embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
2717
 
2718
+ # normalize embeddings
2719
  embeddings = F.normalize(embeddings, p=2, dim=1)
2720
  scores = (embeddings[:2] @ embeddings[2:].T) * 100
2721
  print(scores.tolist())
 
2730
  Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results
2731
  on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316).
2732
 
2733
+ ## Support for Sentence Transformers
2734
+
2735
+ Below is an example for usage with sentence_transformers.
2736
+ ```python
2737
+ from sentence_transformers import SentenceTransformer
2738
+ model = SentenceTransformer('intfloat/e5-small')
2739
+ input_texts = [
2740
+ 'query: how much protein should a female eat',
2741
+ 'query: summit define',
2742
+ "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
2743
+ "passage: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments."
2744
+ ]
2745
+ embeddings = model.encode(input_texts, normalize_embeddings=True)
2746
+ ```
2747
+
2748
+ Package requirements
2749
+
2750
+ `pip install sentence_transformers~=2.2.2`
2751
+
2752
+ Contributors: [michaelfeil](https://huggingface.co/michaelfeil)
2753
+
2754
+ ## FAQ
2755
+
2756
+ **1. Do I need to add the prefix "query: " and "passage: " to input texts?**
2757
+
2758
+ Yes, this is how the model is trained, otherwise you will see a performance degradation.
2759
+
2760
+ Here are some rules of thumb:
2761
+ - Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.
2762
+
2763
+ - Use "query: " prefix for symmetric tasks such as semantic similarity, paraphrase retrieval.
2764
+
2765
+ - Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.
2766
+
2767
+ **2. Why are my reproduced results slightly different from reported in the model card?**
2768
+
2769
+ Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences.
2770
+
2771
+ **3. Why does the cosine similarity scores distribute around 0.7 to 1.0?**
2772
+
2773
+ This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.
2774
+
2775
+ For text embedding tasks like text retrieval or semantic similarity,
2776
+ what matters is the relative order of the scores instead of the absolute values,
2777
+ so this should not be an issue.
2778
+
2779
  ## Citation
2780
 
2781
  If you find our paper or models helpful, please consider cite as follows:
 
2791
 
2792
  ## Limitations
2793
 
2794
+ This model only works for English texts. Long texts will be truncated to at most 512 tokens.
model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:738ee67946d5969622bf1777a455b1b6eef99d62ce859cd781e1633bcc0b4260
3
- size 133448364
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:129a95bd0bfa92362778e4e9329003d15b6407344aa987d858b95809af7e0d49
3
+ size 66728364