Update README.md

README.md

@@ -20,10 +20,17 @@ license: apache-2.0
   </a>
 </p>
 
+最新bce-embedding-base_v1相关信息，以及更多MTEB和RAG相关评测细节，请移步：
 <p align="left">
   <a href="https://github.com/netease-youdao/BCEmbedding">GitHub</a>
 </p>
 
+主要特点：
+1、中英双语，以及中英跨语种能力；
+2、RAG优化，适配更多真实业务场景；
+3、方便集成进langchain和llamaindex。
+
+-----------------------------------------
 <details open="open">
 <summary>Click to Open Contents</summary>
 
@@ -33,7 +40,8 @@ license: apache-2.0
 - <a href="#-model-list" target="_Self">🍎 Model List</a>
 - <a href="#-manual" target="_Self">📖 Manual</a>
   - <a href="#installation" target="_Self">Installation</a>
-  - <a href="#quick-start" target="_Self">Quick Start</a>
+  - <a href="#quick-start" target="_Self">Quick Start (`transformers`, `sentence-transformers`)</a>
+  - <a href="#integrations-for-rag-frameworks" target="_Self">Integrations for RAG Frameworks (`langchain`, `llama_index`)</a>
 - <a href="#%EF%B8%8F-evaluation" target="_Self">⚙️ Evaluation</a>
   - <a href="#evaluate-semantic-representation-by-mteb" target="_Self">Evaluate Semantic Representation by MTEB</a>
   - <a href="#evaluate-rag-by-llamaindex" target="_Self">Evaluate RAG by LlamaIndex</a>
@@ -127,17 +135,20 @@ Existing embedding models often encounter performance challenges in bilingual an
 ### Installation
 
 First, create a conda environment and activate it.
+
 ```bash
 conda create --name bce python=3.10 -y
 conda activate bce
 ```
 
-Then install `BCEmbedding`:
+Then install `BCEmbedding` for minimal installation:
+
 ```bash
-pip install git+https://github.com/netease-youdao/BCEmbedding.git
+pip install BCEmbedding==0.1.1
 ```
 
 Or install from source:
+
 ```bash
 git clone git@github.com:netease-youdao/BCEmbedding.git
 cd BCEmbedding
@@ -146,7 +157,9 @@ pip install -v -e .
 
 ### Quick Start
 
-Use `EmbeddingModel` by `BCEmbedding`, and `cls` [pooler](https://github.com/netease-youdao/BCEmbedding/blob/master/BCEmbedding/models/embedding.py#L24) is default.
+#### 1. Based on `BCEmbedding`
+
+Use `EmbeddingModel`, and `cls` [pooler](./BCEmbedding/models/embedding.py#L24) is default.
 
 ```python
 from BCEmbedding import EmbeddingModel
@@ -161,7 +174,7 @@ model = EmbeddingModel(model_name_or_path="maidalun1020/bce-embedding-base_v1")
 embeddings = model.encode(sentences)
 ```
 
-Use `RerankerModel` by `BCEmbedding` to calculate relevant scores and rerank:
+Use `RerankerModel` to calculate relevant scores and rerank:
 
 ```python
 from BCEmbedding import RerankerModel
@@ -183,6 +196,164 @@ scores = model.compute_score(sentence_pairs)
 rerank_results = model.rerank(query, passages)
 ```
 
+NOTE:
+
+- In [`RerankerModel.rerank`](./BCEmbedding/models/reranker.py#L137) method, we provide an advanced preproccess that we use in production for making `sentence_pairs`, when "passages" are very long.
+
+#### 2. Based on `transformers`
+
+For `EmbeddingModel`:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+# list of sentences
+sentences = ['sentence_0', 'sentence_1', ...]
+
+# init model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained('maidalun1020/bce-embedding-base_v1')
+model = AutoModel.from_pretrained('maidalun1020/bce-embedding-base_v1')
+
+device = 'cuda'  # if no GPU, set "cpu"
+model.to(device)
+
+# get inputs
+inputs = tokenizer(sentences, padding=True, truncation=True, max_length=512, return_tensors="pt")
+inputs_on_device = {k: v.to(self.device) for k, v in inputs.items()}
+
+# get embeddings
+outputs = model(**inputs_on_device, return_dict=True)
+embeddings = outputs.last_hidden_state[:, 0]  # cls pooler
+embeddings = embeddings / embeddings.norm(dim=1, keepdim=True)  # normalize
+```
+
+For `RerankerModel`:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+# init model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained('maidalun1020/bce-reranker-base_v1')
+model = AutoModelForSequenceClassification.from_pretrained('maidalun1020/bce-reranker-base_v1')
+
+device = 'cuda'  # if no GPU, set "cpu"
+model.to(device)
+
+# get inputs
+inputs = tokenizer(sentence_pairs, padding=True, truncation=True, max_length=512, return_tensors="pt")
+inputs_on_device = {k: v.to(device) for k, v in inputs.items()}
+
+# calculate scores
+scores = model(**inputs_on_device, return_dict=True).logits.view(-1,).float()
+scores = torch.sigmoid(scores)
+```
+
+#### 3. Based on `sentence_transformers`
+
+For `EmbeddingModel`:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# list of sentences
+sentences = ['sentence_0', 'sentence_1', ...]
+
+# init embedding model
+## New update for sentence-trnasformers. So clean up your "`SENTENCE_TRANSFORMERS_HOME`/maidalun1020_bce-embedding-base_v1" or "～/.cache/torch/sentence_transformers/maidalun1020_bce-embedding-base_v1" first for downloading new version.
+model = SentenceTransformer("maidalun1020/bce-embedding-base_v1")
+
+# extract embeddings
+embeddings = model.encode(sentences, normalize_embeddings=True)
+```
+
+For `RerankerModel`:
+
+```python
+from sentence_transformers import CrossEncoder
+
+# init reranker model
+model = CrossEncoder('maidalun1020/bce-reranker-base_v1', max_length=512)
+
+# calculate scores of sentence pairs
+scores = model.predict(sentence_pairs)
+```
+
+### Integrations for RAG Frameworks
+
+#### 1. Used in `langchain`
+
+```python
+from langchain.embeddings import HuggingFaceEmbeddings
+from langchain_community.vectorstores import FAISS
+from langchain_community.vectorstores.utils import DistanceStrategy
+
+query = 'apples'
+passages = [
+        'I like apples', 
+        'I like oranges', 
+        'Apples and oranges are fruits'
+    ]
+  
+# init embedding model
+model_name = 'maidalun1020/bce-embedding-base_v1'
+model_kwargs = {'device': 'cuda'}
+encode_kwargs = {'batch_size': 64, 'normalize_embeddings': True, 'show_progress_bar': False}
+
+embed_model = HuggingFaceEmbeddings(
+    model_name=model_name,
+    model_kwargs=model_kwargs,
+    encode_kwargs=encode_kwargs
+  )
+
+# example #1. extract embeddings
+query_embedding = embed_model.embed_query(query)
+passages_embeddings = embed_model.embed_documents(passages)
+
+# example #2. langchain retriever example
+faiss_vectorstore = FAISS.from_texts(passages, embed_model, distance_strategy=DistanceStrategy.MAX_INNER_PRODUCT)
+
+retriever = faiss_vectorstore.as_retriever(search_type="similarity", search_kwargs={"score_threshold": 0.5, "k": 3})
+
+related_passages = retriever.get_relevant_documents(query)
+```
+
+#### 2. Used in `llama_index`
+
+```python
+from llama_index.embeddings import HuggingFaceEmbedding
+from llama_index import VectorStoreIndex, ServiceContext, SimpleDirectoryReader
+from llama_index.node_parser import SimpleNodeParser
+from llama_index.llms import OpenAI
+
+query = 'apples'
+passages = [
+        'I like apples', 
+        'I like oranges', 
+        'Apples and oranges are fruits'
+    ]
+
+# init embedding model
+model_args = {'model_name': 'maidalun1020/bce-embedding-base_v1', 'max_length': 512, 'embed_batch_size': 64, 'device': 'cuda'}
+embed_model = HuggingFaceEmbedding(**model_args)
+
+# example #1. extract embeddings
+query_embedding = embed_model.get_query_embedding(query)
+passages_embeddings = embed_model.get_text_embedding_batch(passages)
+
+# example #2. rag example
+llm = OpenAI(model='gpt-3.5-turbo-0613', api_key=os.environ.get('OPENAI_API_KEY'), api_base=os.environ.get('OPENAI_BASE_URL'))
+service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
+
+documents = SimpleDirectoryReader(input_files=["BCEmbedding/tools/eval_rag/eval_pdfs/Comp_en_llama2.pdf"]).load_data()
+node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
+nodes = node_parser.get_nodes_from_documents(documents[0:36])
+index = VectorStoreIndex(nodes, service_context=service_context)
+query_engine = index.as_query_engine()
+response = query_engine.query("What is llama?")
+```
+
+
 ## ⚙️ Evaluation
 
 ### Evaluate Semantic Representation by MTEB
@@ -193,9 +364,9 @@ We provide evaluateion tools for `embedding` and `reranker` models, based on [MT
 
 #### 1. Embedding Models
 
-Just run following cmd to evaluate `your_embedding_model` (e.g. `maidalun1020/bce-embedding-base_v1`) in **monolingual, bilingual and crosslingual settings** (e.g. `["en", "zh", "en-zh", "zh-en"]`).
+Just run following cmd to evaluate `your_embedding_model` (e.g. `maidalun1020/bce-embedding-base_v1`) in **bilingual and crosslingual settings** (e.g. `["en", "zh", "en-zh", "zh-en"]`).
 
-  运行下面命令评测`your_embedding_model`（比如，`maidalun1020/bce-embedding-base_v1`）。评测任务将会在**单语种，双语种和跨语种**（比如，`["en", "zh", "en-zh", "zh-en"]`）模式下评测：
+  运行下面命令评测`your_embedding_model`（比如，`maidalun1020/bce-embedding-base_v1`）。评测任务将会在**双语和跨语种**（比如，`["en", "zh", "en-zh", "zh-en"]`）模式下评测：
 
 ```bash
 python BCEmbedding/tools/eval_mteb/eval_embedding_mteb.py --model_name_or_path maidalun1020/bce-embedding-base_v1 --pooler cls
@@ -206,8 +377,11 @@ The total evaluation tasks contain ***114 datastes*** of **"Retrieval", "STS", "
   评测包含 **"Retrieval"， "STS"， "PairClassification"， "Classification"， "Reranking"和"Clustering"** 这六大类任务的 ***114个数据集***。
 
 ***NOTE:***
-- All models are evaluated in their **recommended pooling method (`pooler`)**. "jina-embeddings-v2-base-en", "m3e-base" and "m3e-large" use `mean` pooler, while the others use `cls`.
+- **All models are evaluated in their recommended pooling method (`pooler`)**.
+  - `mean` pooler: "jina-embeddings-v2-base-en", "m3e-base", "m3e-large", "e5-large-v2", "multilingual-e5-base", "multilingual-e5-large" and "gte-large".
+  - `cls` pooler: Other models.
 - "jina-embeddings-v2-base-en" model should be loaded with `trust_remote_code`.
+
 ```bash
 python BCEmbedding/tools/eval_mteb/eval_embedding_mteb.py --model_name_or_path {moka-ai/m3e-base | moka-ai/m3e-large} --pooler mean
 
@@ -215,14 +389,14 @@ python BCEmbedding/tools/eval_mteb/eval_embedding_mteb.py --model_name_or_path j
 ```
 
   ***注意：***
-  - 所有模型的评测采用各自推荐的`pooler`。"jina-embeddings-v2-base-en"， "m3e-base"和"m3e-large"的 `pooler`采用`mean`，其他模型的`pooler`采用`cls`.
+  - 所有模型的评测采用各自推荐的`pooler`。"jina-embeddings-v2-base-en", "m3e-base", "m3e-large", "e5-large-v2", "multilingual-e5-base", "multilingual-e5-large"和"gte-large"的 `pooler`采用`mean`，其他模型的`pooler`采用`cls`.
   - "jina-embeddings-v2-base-en"模型在载入时需要`trust_remote_code`。
 
 #### 2. Reranker Models
 
-Run following cmd to evaluate `your_reranker_model` (e.g. "maidalun1020/bce-reranker-base_v1") in **monolingual, bilingual and crosslingual settings** (e.g. `["en", "zh", "en-zh", "zh-en"]`).
+Run following cmd to evaluate `your_reranker_model` (e.g. "maidalun1020/bce-reranker-base_v1") in **bilingual and crosslingual settings** (e.g. `["en", "zh", "en-zh", "zh-en"]`).
 
-  运行下面命令评测`your_reranker_model`（比如，`maidalun1020/bce-reranker-base_v1`）。评测任务将会在**单语种，双语种和跨语种**（比如，`["en", "zh", "en-zh", "zh-en"]`）模式下评测：
+  运行下面命令评测`your_reranker_model`（比如，`maidalun1020/bce-reranker-base_v1`）。评测任务将会在 **双语种和跨语种**（比如，`["en", "zh", "en-zh", "zh-en"]`）模式下评测：
 
 ```bash
 python BCEmbedding/tools/eval_mteb/eval_reranker_mteb.py --model_name_or_path maidalun1020/bce-reranker-base_v1
@@ -323,25 +497,30 @@ The summary of multiple domains evaluations can be seen in <a href=#1-multiple-d
 
 #### 1. Embedding Models
 
-| Model | Retrieval | STS | PairClassification | Classification | Reranking | Clustering | Avg |  
-|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|  
-| bge-base-en-v1.5 | 37.14 | 55.06 | 75.45 | 59.73 | 43.05 | 37.74 | 47.20 |  
-| bge-base-zh-v1.5 | 47.60 | 63.72 | 77.40 | 63.38 | 54.85 | 32.56 | 53.60 |  
-| bge-large-en-v1.5 | 37.15 | 54.09 | 75.00 | 59.24 | 42.68 | 37.32 | 46.82 |  
-| bge-large-zh-v1.5 | 47.54 | 64.73 | **79.14** | 64.19 | 55.88 | 33.26 | 54.21 |  
-| jina-embeddings-v2-base-en | 31.58 | 54.28 | 74.84 | 58.42 | 41.16 | 34.67 | 44.29 |  
-| m3e-base | 46.29 | 63.93 | 71.84 | 64.08 | 52.38 | 37.84 | 53.54 |  
-| m3e-large | 34.85 | 59.74 | 67.69 | 60.07 | 48.99 | 31.62 | 46.78 |  
-| ***bce-embedding-base_v1*** | **57.60** | **65.73** | 74.96 | **69.00** | **57.29** | **38.95** | **59.43** |  
+| Model | Dimensions | Pooler | Instructions | Retrieval (47) | STS (19) | PairClassification (5) | Classification (21) | Reranking (12) | Clustering (15) | ***AVG*** (119) |  
+|:--------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|  
+| bge-base-en-v1.5 | 768 | `cls` | Need | 37.14 | 55.06 | 75.45 | 59.73 | 43.00 | 37.74 | 47.19 |  
+| bge-base-zh-v1.5 | 768 | `cls` | Need | 47.63 | 63.72 | 77.40 | 63.38 | 54.95 | 32.56 | 53.62 |  
+| bge-large-en-v1.5 | 1024 | `cls` | Need | 37.18 | 54.09 | 75.00 | 59.24 | 42.47 | 37.32 | 46.80 |  
+| bge-large-zh-v1.5 | 1024 | `cls` | Need | 47.58 | 64.73 | 79.14 | 64.19 | 55.98 | 33.26 | 54.23 |  
+| e5-large-v2 | 1024 | `mean` | Need | 35.98 | 55.23 | 75.28 | 59.53 | 42.12 | 36.51 | 46.52 |  
+| gte-large | 1024 | `mean` | Free | 36.68 | 55.22 | 74.29 | 57.73 | 42.44 | 38.51 | 46.67 |  
+| gte-large-zh | 1024 | `cls` | Free | 41.15 | 64.62 | 77.58 | 62.04 | 55.62 | 33.03 | 51.51 |  
+| jina-embeddings-v2-base-en | 768 | `mean` | Free | 31.58 | 54.28 | 74.84 | 58.42 | 41.16 | 34.67 | 44.29 |  
+| m3e-base | 768 | `mean` | Free | 46.29 | 63.93 | 71.84 | 64.08 | 52.38 | 37.84 | 53.54 |  
+| m3e-large | 1024 | `mean` | Free | 34.85 | 59.74 | 67.69 | 60.07 | 48.99 | 31.62 | 46.78 |  
+| multilingual-e5-base | 768 | `mean` | Need | 54.73 | 65.49 | 76.97 | 69.72 | 55.01 | 38.44 | 58.34 |  
+| multilingual-e5-large | 1024 | `mean` | Need | 56.76 | 66.79 | 78.80 | 71.61 | 56.49 | 43.09 | 60.50 |  
+| ***bce-embedding-base_v1*** | 768 | `cls` | Free | 57.60 | 65.73 | 74.96 | 69.00 | 57.29 | 38.95 | 59.43 |  
 
 ***NOTE:***
-- Our ***bce-embedding-base_v1*** outperforms other opensource embedding models with various model size.
+- Our ***bce-embedding-base_v1*** outperforms other opensource embedding models with comparable model size.
 - ***114 datastes*** of **"Retrieval", "STS", "PairClassification", "Classification", "Reranking" and "Clustering"** in `["en", "zh", "en-zh", "zh-en"]` setting.
 - The [crosslingual evaluation datasets](https://github.com/netease-youdao/BCEmbedding/blob/master/BCEmbedding/evaluation/c_mteb/Retrieval.py) we released belong to `Retrieval` task.
 - More evaluation details please check [Embedding Models Evaluation Summary](https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/EvaluationSummary/embedding_eval_summary.md).
 
   ***要点：***
-  - 对比所有开源的各种规模的embedding模型，***bce-embedding-base_v1*** 表现最好。
+  - 对比���他开源的相同规模的embedding模型，***bce-embedding-base_v1*** 表现最好，效果比最好的large模型稍差。
   - 评测包含 **"Retrieval"， "STS"， "PairClassification"， "Classification"， "Reranking"和"Clustering"** 这六大类任务的共 ***114个数据集***。
   - 我们开源的[跨语种语义表征评测数据](https://github.com/netease-youdao/BCEmbedding/blob/master/BCEmbedding/evaluation/c_mteb/Retrieval.py)属于`Retrieval`任务。
   - 更详细的评测结果详见[Embedding模型指标汇总](https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/EvaluationSummary/embedding_eval_summary.md)。
@@ -368,16 +547,8 @@ The summary of multiple domains evaluations can be seen in <a href=#1-multiple-d
 
 #### 1. Multiple Domains Scenarios
 
-| Embedding Models | WithoutReranker <br> [*hit_rate/mrr*] | CohereRerank <br> [*hit_rate/mrr*] | bge-reranker-large <br> [*hit_rate/mrr*] | ***bce-reranker-base_v1*** <br> [*hit_rate/mrr*] | 
-|:-------------------------------|:--------:|:--------:|:--------:|:--------:| 
-| OpenAI-ada-2 | 81.04/57.35 | 88.35/67.83 | 88.89/69.64 | **90.71/75.46** |  
-| bge-large-en-v1.5 | 52.67/34.69 | 64.59/52.11 | 64.71/52.05 | **65.36/55.50** |  
-| bge-large-zh-v1.5 | 69.81/47.38 | 79.37/62.13 | 80.11/63.95 | **81.19/68.50** |  
-| llm-embedder | 50.85/33.26 | 63.62/51.45 | 63.54/51.32 | **64.47/54.98** |  
-| CohereV3-en | 53.10/35.39 | 65.75/52.80 | 66.29/53.31 | **66.91/56.93** |  
-| CohereV3-multilingual | 79.80/57.22 | 86.34/66.62 | 86.76/68.56 | **88.35/73.73** |  
-| JinaAI-v2-Base-en | 50.27/32.31 | 63.97/51.10 | 64.28/51.83 | **64.82/54.98** |  
-| ***bce-embedding-base_v1*** | **85.91/62.36** | **91.25/69.38** | **91.80/71.13** | ***93.46/77.02*** |  
+
+![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/64745e955aba8edfb2ed561a/NyV_6ZrsaqUluUnxHKR_m.jpeg)
 
 ***NOTE:***
 - In `WithoutReranker` setting, our `bce-embedding-base_v1` outperforms all the other embedding models.
@@ -401,7 +572,8 @@ Welcome to scan the QR code below and join the WeChat group.
 
   欢迎大家扫码加入官方微信交流群。
 
-<img src="https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/assets/Wechat.jpg" width="20%" height="auto">
+
+![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/64745e955aba8edfb2ed561a/mMlIkYn2qPXlivq4wtvyy.jpeg)
 
 ## ✏️ Citation