Repository: beir-cellar/beir
Issue: Different results by retriever.encode_and_retrieve and retriever.retrieve
Root cause: Algorithmic Error

Hi, I encountered an issue while evaluating the performance of contriever-msmarco on the Arguana dataset using the official example.

When running the following code:
```
results = retriever.retrieve(corpus, queries)
results = retriever.encode_and_retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values, ignore_identical_ids=False)
```
I noticed that the results output might differ depending on which method (`retrieve `or `encode_and_retrieve`) is used. Specifically, `encode_and_retrieve `may include document IDs that are the same as the query IDs.

I always set `ignore_identical_ids=False`, and when using retrieve, I get a normal ndcg@10=44 by `retrieve`. However, when using `encode_and_retrieve`, the ndcg@10 is much lower at only 33.4. After comparing the results from both methods, I found that `encode_and_retrieve `includes query-document similarity with the query itself, which causes the issue.

I would like to know how I can fix this problem, as I intend to save the embeddings and use them.

Thank you in advance!

Hi @liyongkang123, 

Thanks for catching the error; you are correct. I missed implementing the logic to remove `query_id` that are identical to itself in the document list. This is why your scores are coming out to be lower than expected. 

I'll write a bugfix script to resolve this by tomorrow.

Regards,
Nandan

Hi Nandan, I’m not sure if this is the right approach. If there are cases where a `doc_id `is the same as a `query_id `in the corpus, I wonder whether this should be handled under the `ignore_identical_ids `parameter?

Perhaps we could compare the text of the `doc_id` and the `query_id`, and if they are the same, we could remove it.

Hi Nandan,   I also found the following difference:
```
retriever = EvaluateRetrieval(model, score_function=score_function, k_values=[1, 5, 10, 50, 100, 1000])  # "dot" for dot product cos_sim
results = retriever.encode_and_retrieve(corpus, queries, encode_output_path=embedding_save_path, overwrite=True)
results = retriever.retrieve(corpus, queries)
```
If we set a `score_function `for the retriever, it will only affect the `retriever.retrieve` function and not `retriever.encode_and_retrieve`.

This is because `retriever.encode_and_retrieve` first encodes and saves the embeddings, while in the `search_from_files `function, it directly calls `faiss.IndexFlatIP` for inner product-based indexing. In short, if you use `retriever.encode_and_retrieve`, you must normalize the embeddings directly when the model outputs them if you intend to compute cosine similarity.

Hi @liyongkang123, 

This is really useful. Will you be able to create a PR to include the score function within the `retriever.encode_and_retrieve` function? 

Hi Nandan, I can try doing it this way. However, I'm not sure when the normalization is performed. We have two strategies:

1. Perform normalization before saving the embedding, then continue using `faiss.IndexFlatIP`.
2. Save the embedding with user-controlled model output, and normalize all embeddings before using` faiss.IndexFlatIP`.
   Do you have a preference?


If we would like to do cosine-similarity, then (1), then normalizing before saving the embedding makes sense to me.