Update README.md

README.md

@@ -110,6 +110,30 @@ print(query_vectors)
 
 Complete working Colab Notebook is [here](https://colab.research.google.com/drive/1-5WGEYPSBNBg-Z0bGFysyvckFuM8imrg)
 
+### Reranking Using ColBERT
+
+```python
+from colbert.modeling.checkpoint import Checkpoint
+from colbert.infra import ColBERTConfig
+
+query = ["How to use ColBERT for indexing long documents?"]
+documents = [
+    "ColBERT is an efficient and effective passage retrieval model.",
+    "Jina-ColBERT is a ColBERT-style model but based on JinaBERT so it can support both 8k context length.",
+    "JinaBERT is a BERT architecture that supports the symmetric bidirectional variant of ALiBi to allow longer sequence length.",
+    "Jina-ColBERT model is trained on MSMARCO passage ranking dataset, following a very similar training procedure with ColBERTv2.",
+]
+
+config = ColBERTConfig(query_maxlen=32, doc_maxlen=512)
+ckpt = Checkpoint(args.reranker, colbert_config=colbert_config)
+Q = ckpt.queryFromText([all_queries[i]])
+D = ckpt.docFromText(all_passages, bsize=32)[0]
+D_mask = torch.ones(D.shape[:2], dtype=torch.long)
+scores = colbert_score(Q, D, D_mask).flatten().cpu().numpy().tolist()
+ranking = numpy.argsort(scores)[::-1]
+print(ranking)
+```
+
 ## Evaluation Results
 
 **TL;DR:** Our Jina-ColBERT achieves the competitive retrieval performance with [ColBERTv2](https://huggingface.co/colbert-ir/colbertv2.0) on all benchmarks, and outperforms ColBERTv2 on datasets in where documents have longer context length.
@@ -157,14 +181,41 @@ We also evaluate the zero-shot performance on datasets where documents have long
 | Jina-ColBERT-v1 | 8192 | 8192 | 83.7 |
 | Jina-embeddings-v2-base-en | 8192 | 8192 | **85.4** |
 
-\* denotes that we truncate the context length to the length of 512 for document but the query length is still 512.
+\* denotes that we truncate the context length to 512 for documents. The context length of queries is all 512.
+
+**To summarize, Jina-ColBERT achieves the comparable retrieval performance with ColBERTv2 on all benchmarks, and outperforms ColBERTv2 on datasets in where documents have longer context length.**
+
+### Reranking Performance
+
+We evaluate the reranking performance of ColBERTv2 and Jina-ColBERT on BEIR. We use BM25 as the first-stage retrieval model. The full evaluation code can be found in [this repo](https://github.com/liuqi6777/eval_reranker). 
+
+In summary, Jina-ColBERT outperforms ColBERTv2, even achieving comparable performance with some cross-encoder.
+
+The best model, jina-reranker, will be open-sourced soon!
+
+|BM25|ColBERTv2|Jina-ColBERT|MiniLM-L-6-v2|BGE-reranker-base-v1|BGE-reranker-large-v1|Jina-reranker-base-v1|
+| --- | :---: | :---: | :---: | :---: | :---: | :---: |
+Arguana         |29.99|33.42|33.95|30.67|23.26|25.42|42.59|
+Climate-Fever   |16.51|20.66|21.87|24.70|31.60|31.98|25.49|
+DBPedia         |31.80|42.16|41.43|43.90|41.56|43.79|43.68|
+FEVER           |65.13|81.07|83.49|80.77|87.07|89.11|86.10|
+FiQA            |23.61|35.60|36.68|34.87|33.17|37.70|41.38|
+HotpotQA        |63.30|68.84|68.62|72.65|79.04|79.98|75.61|
+NFCorpus        |33.75|36.69|36.38|36.48|32.71|36.57|37.73|
+NQ              |30.55|51.27|51.01|52.01|53.55|56.81|56.82|
+Quora           |78.86|85.18|82.75|82.45|78.44|81.06|87.31|
+SCIDOCS         |14.90|15.39|16.67|16.28|15.06|16.84|19.56|
+SciFact         |67.89|70.23|70.95|69.53|70.62|74.14|75.01|
+TREC-COVID      |59.47|75.00|76.89|74.45|67.46|74.32|82.09|
+Webis-touche2020|44.22|32.12|32.56|28.40|34.37|35.66|31.62|
+Average         |43.08|49.82|50.25|49.78|49.84|52.57|**54.23**|
+
+ColBERT
 
-**To summarize, Jina-ColBERT achieves the comparable performance with ColBERTv2 on all benchmarks, and outperforms ColBERTv2 on datasets in where documents have longer context length.**
 
 ## Plans
 
-- We will evaluate the performance of Jina-ColBERT as a reranker in a retrieval pipeline, and add the usage examples.
-- We are planning to improve the performance of Jina-ColBERT by fine-tuning on more datasets in the future.
+We are planning to improve the performance of Jina-ColBERT by fine-tuning on more datasets in the future.
 
 ## Other Models