Paper: Adder: Adapted Dense Retrieval (ICLR-2024)

Reviewer comment (Novelty):
The proposed approach has limited novelty and is similar to existing work of training task/domain specific parameters while keeping model parameters fixed. In this case, the model is actually served through an API and users can only access final embedding outputs from the model.

Author response:
The review correctly points out that our approach uses some well-known ideas, like residuals and adapters. The key novelty is the combination of these ideas in the right way and applying them on embeddings. We are thus able to tune retrievals to solve a problem that the review agrees is well motivated. In short, the contributions include the strict heterogeneous retrieval problem definition that arises from new RAG-like applications, the idea of adapting embeddings, the idea of using residual low-rank adaptations for embeddings and the evaluation showing that the idea works.

Reviewer comment (Novelty):
This proposed approach is similar to adding additional trainable parameters on top of the final layer of a model. Even the choice of transformation function used in this paper is explored in prior work.

Author response:
The review correctly points out that our approach uses some well-known ideas, like residuals and adapters. The key novelty is the combination of these ideas in the right way and applying them on embeddings. We are thus able to tune retrievals to solve a problem that the review agrees is well motivated. In short, the contributions include the strict heterogeneous retrieval problem definition that arises from new RAG-like applications, the idea of adapting embeddings, the idea of using residual low-rank adaptations for embeddings and the evaluation showing that the idea works.

Reviewer comment (Experiments):
In terms of the transformation function, only one formulation is tried and no ablation or alternate variations are tested.

Author response:
The review also correctly points out that evaluation is limited in the paper. We performed extensive experiments with various modeling choices, such as not using residuals, not using contrastive loss, not using global hard negative, etc., before we arrived at the right solution presented in this paper. Unfortunately, we did not include those results in the paper. We plan to fix this oversight and include the results for these variants: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss. We also experimentally observed that Adder and Adder2 adapters over SBert embeddings show an even greater relative gain in nDCG metrics than what we report for OpenAI embeddings. Please see the table in the response above to Reviewer Y6UA.

Reviewer comment (Experiments):
There can be other simple variations to the transformation function such as using a shallow multi-layer perceptron model or using a trainable skip connection weight parameter. Adding such other choices could further strengthen the experiments in this paper.

Author response:
The review also correctly points out that evaluation is limited in the paper. We performed extensive experiments with various modeling choices, such as not using residuals, not using contrastive loss, not using global hard negative, etc., before we arrived at the right solution presented in this paper. Unfortunately, we did not include those results in the paper. We plan to fix this oversight and include the results for these variants: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss. We also experimentally observed that Adder and Adder2 adapters over SBert embeddings show an even greater relative gain in nDCG metrics than what we report for OpenAI embeddings. Please see the table in the response above to Reviewer Y6UA.

Reviewer comment (Experiments):
The domains used for evaluation are small-scale, containing up to 57K items, which is rather small for information retrieval tasks.

Author response:
Regarding the limited corpus size, the choice is guided purely by our end use case for RAG.  Also in our target application, we have limited training data. We wanted to design a solution that can work with limited training data, even if it required small corpus sizes, and used limited computational resources. That was the goal of our work. We agree that for larger scale datasets and enough training data, other approaches, such as finetuning, could come into play, but they would also incur a significant cost and time. ADDER only takes 8secs for 1 epoch (668 batches for a batch size 16) on a single gpu (k-80) machine.

Reviewer comment (Experiments):
Large-scale datasets from the BeIR benchmark, which contain domains with up to 8 million items, are not used and their absence is not explained.

Author response:
We would like to clarify that the training data is limited by design in our proposed technique & experiments to cater to low-resource domains (e.g. downstream applications of RAG) where there is limited access to data. Our whole approach and the design choices -- adapters and hard global negatives -- are guided by this limited data setting. Hence, the evaluation is also focused on the scenarios where data is limited.
 
The immediate end goal is not to perform general-purpose information retrieval, but a very specific application of it needed for LLM-based solutions for tasks such as NL2Code; please also see our overall response at https://openreview.net/forum?id=n3kFlvVhJM&noteId=UiCjIw5BCv.  We recently obtained experimental results that show the gains from using ADDER and ADDER2 for NL2Code tasks performed using LLMs where we (a) use ADDER to retrieve few-shot examples and (b) use ADDER to rerank the candidates returned by the LLM. This application was one of our main motivations for designing ADDER and ADDER2. On the Excel's PowerQuery Mashup dataset, we observe that using only OpenAI embeddings gives us a sketch match performance of 0.72, whereas using ADDER gives us a performance of 0.74.  The important point to note here is that the Mashup is a low-resource language with limited data availability, thus it is not amenable to techniques for domain-adaptation that require high data requirement.
 
We would also like to point the author to some qualitative examples for the same task before & after we apply ADDER to the pipeline in the [official comment on the main thread](https://openreview.net/forum?id=n3kFlvVhJM&noteId=HHbXMvYBvT). The reviewer may note the difference in quality for the top-ranked examples.
 
If the training data was not a constraint, many existing approaches would be available -- finetune/retrain white-box models, training dense layers on top of the black-box model --  which could potentially perform better. ADDER employs residual adaptation to embeddings; this allows us to massage the inherent distribution of data to adapt better to the task but not transform it completely. Over larger datasets, we suspect that transformation of the embeddings would be of a much higher degree and denser to learn, but that is not the goal here.

Reviewer comment (Experiments):
The paper can be stronger with more ablations and analysis. One important experiment is to test if the proposed method can work with different base embeddings models, not just OpenAI embeddings.

Author response:
W1, W2: We realize now that although we did a lot of experiments with other modeling choices, we only reported the numbers with the best architecture we found. We will include performance numbers for following ablations in the final version: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss.

Regarding using other base embeddings, thank you for the suggestion because we ran experiments with sBert and noticed that the relative gains provided by Adder are huge (compared to the relative gains we get over OpenAI embeddings). Although, the absolute numbers are poorer than what we get with OpenAI embeddings. 

Here are the results:  
For the same BEIR benchmarks reported in Table 2, here are the results with using SBert as base. (nDCG@k values)

| Dataset |   Base@1  |   ADDER@1 |   Base@3  |   ADDER@3 |   Base@5  |   ADDER@5 |   Base@10 |  ADDER@10| 
| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | 
| Scifact    |   0.01    |   0.17 |   0.02  |   0.19 |   0.02 |   0.22 |   0.03 |   0.24 | 
| Arguana    |   0.04 |   0.11 |   0.07 |   0.19 |   0.08 |   0.22 |   0.10 |   0.25 | 
| FiQA   |   0.00 |   0.06 |   0.00 |   0.05 |   0.00 |   0.06  |   0.01 |   0.07 | 
| NFCorpus   |   0.03 |   0.07  |   0.02 |   0.07 |   0.02 |   0.07 |   0.03 |   0.06 | 

For the same task reported in Table 3, here are the results when using SBert as base: (nDCG@k values)

| Dataset   |   Base@1  |   ADDER@1 |   Base@3  |   ADDER@3 |   Base@5  |   ADDER@5 |   Base@10 |   ADDER@10 | 
| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
|   M   |   0.04    |   0.76    |   0.08    |   0.86    |   0.09    |   0.88    |   0.11    |   0.89 | 
|   Conala  |   0.07    |   0.92    |   0.13    |   0.96    |   0.15    |   0.97    |   0.17    |   0.97 | 
|   Bash    |   0.05    |   0.81    |   0.07    |   0.89    |   0.09    |   0.9 |   0.1 |   0.91 | 
|   SMCalFlow   |   0.01    |   0.95    |   0.02    |   0.98    |   0.02    |   0.98    |   0.03    |   0.98 |

Reviewer comment (Experiments):
Why using the attention + residual adaptation? Have you tried other options, e.g., a linear adapter or MLP?

Author response:
Q1: It is a good question about why attention + residual. Indeed, we started with a linear adapter and an MLP adapter. We observed that their performance was often much worse than the raw openAI embeddings, especially when there was limited training data. We realized that OpenAI embeddings are actually very good, and we wanted to keep their “goodness” and only perturb/tweak them slightly for a particular retrieval task. If the reviewer would find it useful, we can rerun those early experiments and report the results. 

W1, W2: We realize now that although we did a lot of experiments with other modeling choices, we only reported the numbers with the best architecture we found. We will include performance numbers for following ablations in the final version: (1) not using residual adaptation and instead using direct MLP adapter, (2) not using global hard negatives and instead using batch negatives, (3) not using contrastive loss.

Reviewer comment (Evaluation):
While the idea of adapter is interesting, a key problem of the paper is that it fails to demonstrate that it can improve IR performance. As the experimental results show, the method can only improve on some of the datasets. The advantage of the method is not demonstrated.

Author response:
We agree with the reviewer that “[our] method can only improve on some of the datasets”. The important point is that we understand when it improves and when it does not. The point is that we should use an adapter only when it is needed.  We are not claiming that one should always attach an adapter for every kind of retrieval task. Only when the base embeddings perform poorly, should we attach an adapter. Our explanation for when adapters are likely to improve, and when not, can guide this choice. The fact that the adapter is not improving the metrics for some benchmarks should be viewed as a learning contribution of our paper. We further note that the improvements due to ADDER are dependent also on the base model, and OpenAI embedding model is a particularly powerful model that is already proficient on retrievals over several different classes of entities. This fact is not true for other base models, such as SBert. ADDERT over SBert consistently improves retrieval performance because SBert embeddings are not “trained” for diverse retrieval tasks; please see the table in the response to Reviewer Tp6G.

Reviewer comment (Experiments):
The experiments have been carried out on small datasets. The authors argue that retrieval on a small dataset is a particular problem that warrants more explorations. It is unclear why this is the case, and why a retriever that works on large datasets may not be directly applied to small datasets.

Author response:
We also want to address the concern that the reviewer points out regarding why we perform well on smaller datasets. We discuss this in Section 4.1 of the paper. The proposed technique is more effective in strict retrieval settings given the limited amount of data available and used in the training process of adapters (e.g. Retrieval Augmented Generation). In the limited data setting, we tend to model the training of our adapters with a hard global negatives optimization which helps pick the ideal corpus element for a given query by contrasting it against other corpus elements. This exercise learns better to differentiate between top elements; however, as you go down in the ranking, the relations are not ordered well enough to learn fine differences to prefer one against another. Hence, resulting at better retrievals at smaller `k`.

If the training data was not a constraint, we would be able to finetune/retrain white-box models as well as training dense layers on top of the black-box model which could perform better at all `k` and be able to learn a new embedding distribution for your task. ADDER employs residual adaptation to embeddings; this allows us to massage the inherent distribution of data and not transform it completely. Over larger datasets, we suspect that transformation of the embeddings would be of a much higher degree and denser to learn.

Reviewer comment (Novelty):
The authors also argue that the existing investigations in IR have focused on large ranked lists, and this paper targets small ranked lists (considered to be more strict). It is unclear why this size of output is a particular problem. In many previous studies, evaluations have looked at not only top-K with large K, but also top-K with small K (NDCG@1, NDCG@5, ...). They are not so different from the measures used in this paper. In addition, the paper does not propose a specific method for retrieving only a few documents. So, the difference between large and small K is not so important, and the difference (if any) is not addressed in the paper.

Author response:
Regarding the discussion on large vs small corpus, we only wanted to make the point that retrieval from small corpus is also of interest in certain applications. Of course, one can use retrieval methods that work on large corpus also on small corpus. Similarly for the issue with small versus large K. We wanted to improve metrics @k for k=1,3. As out results show, when Adder improves over baseline, we usually get a higher relative improvement for small k than we get for large k. We want to point out that our Adder approach is almost at par with the state-of-the-art, and even better than the state-of-the-art for SciFact based on the leaderboard for BEIR.

Reviewer comment (Experiments):
The authors only compared the proposed method with a basic LLM-based retriever. The latter may not be a state of the art of IR. It would be interesting to compare with other baselines such as ANCE and DPR. It would be interesting to also compare the adapter-based method with a fine-tuning-based method. Although fine-tuning a LLM with limited examples may not be easy or effective, to support the argument that adapter is a better solution than fine-tuning, the comparison may be useful.

Author response:
We understand the reviewer's concern regarding the OpenAI embeddings not being tested on larger IR datasets. We would like to draw their attention to the following papers: 
- [Text and Code Embeddings by Contrastive Pre-Training](https://arxiv.org/abs/2201.10005): The previous generation of OpenAI embeddings already outperformed ANCE as shown in _Table 5_. _Table 7_ also shows that they generalize well with other large scale IR datasets.
- [Evaluating Embedding APIs for Information Retrieval](https://arxiv.org/pdf/2305.06300.pdf): This evaluates the predecessors mentioned in the first paper, the latest OpenAI embeddings (`ada-002` which we use in our experiments), and other Embeddings available as a service. The paper finds that the `ada-002` embeddings outperform others in almost all IR tasks on a very wide range of datasets.
- [Vector Search with OpenAI Embeddings](https://arxiv.org/pdf/2201.10005.pdf): This work builds upon both the previously mentioned works and further extends the evaluation to more datasets like MSMARCO \& TREC-DL19/20.

We hope this addresses the concerns about extensive evaluation of OpenAI Embeddings and their generality over IR benchmarks. An important point to note in all these papers is that the OpenAI embeddings are almost as good as the other leading baselines usually exceeding or trailing them by a very small margin. We would be happy to include results over other baselines in the final version, if the reviewer believes that it would reasonably add more value to this work.

Reviewer comment (Experiments):
The concept of heterogeneous datasets is misleading. One could understand that the dataset contains several types of data. The case of NL queries and code dataset is not a real heterogeneous dataset. This is more similar to cross-media retrieval. The proposed method does not seem to be capable of handling truly heterogeneous datasets.

Author response:
We will be happy to replace the use of “heterogeneous” since it seems it has a connotation different from what we meant by it.

Reviewer comment (Experiments):
How does the adapter-based approach compare to a fine-tuning-based method? Would it be possible to provide additional comparison between these methods?

Author response:
We assume black-box access to the base model. In other words, we do not access to the weights. Hence, adapter-based approach is the only possibility and fine-tuning is not an option.

Reviewer comment (Experiments):
How does it compare to the state-of-the-art IR methods?

Author response:
Regarding comparison to the state-of-the-art IR methods, we note that the general-purpose OpenAI embeddings are a very strong baseline. Looking at the leaderboard on BEIR benchmarks (https://eval.ai/web/challenges/challenge-page/1897/leaderboard/4475), which reports ndcg@10, we see that OpenAI embeddings (our baseline) is very close to the leaders, and even beats the top leader on one dataset (Arguana). For SciFact, we improve over openai embeddings, and thus beat the state-of-the-art (on the ndcg@10 metric). As we discussed in Q1 above, we get higher relative gains on ndcg@1, which is our focus. Further, we explained in the paper why we do not see gains across all benchmarks, and when we don’t, we do not deteriorate the performance of openai by much (and still beat the leaders on some benchmarks, such as Arguana).

Reviewer comment (Experiments):
What are the specific solutions proposed in the paper for: (1) retrieval on small datasets, (2) for top-K with small K?

Author response:
The specific solution proposed in the paper is the idea of learning a low-rank residual adapter for embeddings.

Reviewer comment (Writing):
What are the general characteristics of NL2X? The description is missing.

Author response:
We also obtained experimental results that show the gains from using ADDER and ADDER2 for NL2Code tasks performed using LLMs for where we (a) use ADDER to retrieve few-shot examples and (b) use ADDER to rerank the candidates returned by the LLM. This application was one of our main motivations for designing ADDER and ADDER2. On the Excel's PowerQuery Mashup dataset, we observe that using only OpenAI embeddings gives us a sketch match performance of 0.72, whereas using ADDER gives us a performance of 0.74. As part of ongoing work, we continue to perform experiments on NL2X tasks using ADDER.

Reviewer comment (Reproducibility):
There is no specification of the data sets used to tune the hyperparameters (e.g., the dimensions, but also $\gamma$, which I assume is part of the loss?).

Author response:
No, we do not perform evaluation (for hyperparameter selection) on the full, or even 80%, of the test set. It is always on the dev set. In case of one benchmark set (Arguana), the train+dev set was picked to be 20% of the test set (because Arguana had no train set).

For the NL2X tasks we had access to well defined train and test sets.

Reviewer comment (Experiments):
It is noted that using batch-local instead of global negative examples leads to worse results, but no performance number is given for this approach.

Author response:
Regarding batch local negatives vs global negatives, we observed that using local negatives reduces performance over the base embeddings by over 10% or more over the datasets. Since we saw this in initial experimentation, we did not record the exact numbers. We would be happy to perform those experiments and report the numbers.

Reviewer comment (Writing):
There is a high-level description of a loss function (should be zero if the positive example is closer than the negative), but no concrete instantiation in the form of a formula is given.

Author response:
The loss function is as follows: if q, c, and c’ are the transformed embeddings of the query, true positive corpus element, and the global negative corpus element, then the loss l(q,c,c’) = max(0, dot(q,c’) - dot(q,c)), where dot is the cosine similarity function.

Reviewer comment (Experiments):
The subtitle of Table 2 mentions SBert, but its results are not shown further because they are "uniformly worse than those for OpenAI embeddings". It would be interesting (1) to see figures showing this, and (2) to find out if the presented method also works for worse base embeddings.

Author response:
We thank the reviewer for their comment on SBert, and include the following results for SBERT for both the default and adapted embeddings scenarios: 

 For the same BEIR benchmarks reported in Table 2, here are the results with using SBert as base. (nDCG@k values)    

| Dataset |   Base@1  |   ADDER@1 |   Base@3  |   ADDER@3 |   Base@5  |   ADDER@5 |   Base@10 |  ADDER@10| 
| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |  
| Scifact | 0.01 |   0.17 |   0.02  | 0.19 | 0.02 | 0.22 | 0.03 | 0.24 |  
| Arguana |   0.04 |   0.11 |   0.07 | 0.19 | 0.08 | 0.22 | 0.10 | 0.25 |   
| FiQA | 0.00 |   0.06 |   0.00 |   0.05 | 0.00 | 0.06 | 0.01 | 0.07 |   
| NFCorpus | 0.03 | 0.07 | 0.02 | 0.07 | 0.02 | 0.07 | 0.03 | 0.06 |  

 For the same task reported in Table 3, here are the results when using SBert as base: (nDCG@k values)      

| Dataset   |   Base@1  |   ADDER@1 |   Base@3  |   ADDER@3 |   Base@5  |   ADDER@5 |   Base@10 |   ADDER@10 |     
| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |    
|   M   |   0.04    | 0.76  |   0.08    |   0.86    |   0.09    |   0.88    |   0.11    |   0.89 |     
|   Conala  | 0.07  | 0.92  |   0.13    |   0.96    |   0.15    |   0.97    |   0.17    |   0.97 |     
|   Bash    | 0.05 |   0.81    |   0.07    |   0.89    |   0.09    |   0.9 |   0.1 |   0.91 |     
|   SMCalFlow   |  0.01 | 0.95 | 0.02 | 0.98 | 0.02 | 0.98 | 0.03 | 0.98 |

Reviewer comment (Novelty):
The paper lacks a comparison of competing methods, and its distinction from related work is not very precise.

Author response:
Regarding comparison to competing methods, we observed that OpenAI embeddings were giving state-of-the-art results in most cases; in some cases, OpenAI embeddings performed even better than the leader on the BEIR leaderboard. That was the reason to start with OpenAI embeddings. The ADDER adapter works over those embeddings, and even improves over them in some cases.

Reviewer comment (Experiments):
A comparison with adaptive similarity methods as investigated in the field of similarity search is missing, e.g. [Seidl97].

Author response:
We also thank the reviewer for the interesting [Seidl97] reference that we will discuss in the revised version.

Reviewer comment (Experiments):
The experimental setup description seems to contradict itself. One statement says: "We carried out our experiments on regular laptops and desktops, and used no special purpose hardware for training or inference except for the black-box rest API calls to the OpenAI embedding endpoint." Another statement says: "To be precise, for our experiments over ADDER & ADDER2, we use a virtual machine with a single Nvidia K80 GPU (with 24GiB of vRAM), 4 CPU cores and 28 GB of RAM."

Author response:
Thanks for pointing out the contradictory description of our experimental setup. Our initial experiments were done on CPU machines, but later when we had to perform multiple runs to generate numbers for our tables, we switched to a virtual machine with GPU. We will fix this error. To provide evidence for the fact that the computational requirement is low, we report here that the time taken to train our ADDER for 1 epoch (containing 886 batches with batch size 16) is 8.97s on GPU, whereas the same took 71.73s on CPU. So, the experiments are doable on CPU, they just take a little longer but still reasonable time.

Reviewer comment (Presentation):
The tables would be more readable if the numbers of competing approaches were next to each other (e.g., by having column groups for each k=1,3,5,10 value and the approaches next to each other).

Author response:
Thanks also for the suggestion to rearrange the columns of the tables. We will do so.

Reviewer comment (Evaluation):
Though the pretrained embedding model is treated as a black box, adaptation is still strictly tied to the specific pretrained embedding model and needs to be done separately for each pretrained embedding model. As the pretrained embedding model changes over time due to retraining/continual updating, the adapters also need to be retrained.

Author response:
The reviewer is correct when they state that the adaptation is tied to the specific blackbox model. When the blackbox embedding model changes, the adapters also need to be retrained. We would like to point out that even though this is true, there is minimal effort in retraining the adaptors as compared to other techniques like finetuning. For 1 epoch consisting of 668 batches (batch size 16) our technique only takes ~71 seconds on the largest dataset FiQA-2018 (corpus size 57k), that too on a CPU only compute (on a single K-80 GPU this takes only 8 secs). Hence the effort in training adapters is very minimal and not compute-intensive.

Reviewer comment (Experiments):
The work assumes that in many applications, the target corpus is of small size (a few thousand documents) and that a limited number of examples of "good" retrievals (a few thousand) are available for this corpus. This is however a restrictive assumption. Typically the corpus is several orders larger in size than the available number of "good" retrievals.

Author response:
We agree with the reviewer that working on only small corpus sizes is restrictive. We do report numbers for FiQA and NFCorpus, where the corpus sizes are larger, and the number of good retrievals is less. There are applications where corpus sizes are small and training data is sparse, which serve as motivation for this work.

Reviewer comment (Evaluation):
Improvements in retrieval efficacy is not impressive. The proposed approach is not a clear winner always. It is also clear what is the best improvement one can hope to get from this line of attack. Is the lack of large training data sets the major hurdle for significant further improvement or there are inherent limitations in the methodology? Even if large amount of training data were available, the global hard negatives idea employed would not work as is.

Author response:
We agree that we cannot bring about large improvements since our technique employs a residual approach. It is limited to adapting the embeddings and not changing them entirely. Further, we envision our method to be employed in cases specifically where there is a scarcity of training data and larger models might be overfit to this. In case there is sufficient training data one can finetune a whitebox model or learn a denser layer on top of the blackbox embedding and would see better improvements since the inherent quality of dense embeddings would improve. Another benefit of our approach is the low computational requirement, which allows one to do online learning to tune ADDERs to handle new data/ outliers. Finally, the improvements in retrieval efficacy are dependent on the base model -- we observed significantly more pronounced gains SBert; please see the table in comment to Reviewer LpK4.

Reviewer comment (Experiments):
There is no study of the improvement brought by the proposed adapter-based retrieval approach to RAG tasks.

Author response:
Thank you for asking about results on RAG. We report results on the M dataset proposed in [1], we would be happy to include these results in the final version of the paper as well. We observe the using only OpenAI embeddings gives us a sketch match performance of 0.72 on the M dataset. Whereas when we use ADDER to select the fewshot prompt and perform reranking over generated candidates, we obtain a Top-1 performance of 0.74.

[1] Anirudh Khatry, Joyce Cahoon, Jordan Henkel, Shaleen Deep, Venkatesh Emani, Avrilia Floratou, Sumit Gulwani, Vu Le, Mohammad Raza, Sherry Shi, Mukul Singh, Ashish Tiwari. From Words to Code: Harnessing Data for Program Synthesis from Natural Language (arXiv:2305.01598)

Reviewer comment (Experiments):
One limitation of the paper is the lack of baselines with task-specific adaptation. The only baseline shown in the evaluation is based on general-purpose embeddings, which may not be a fair comparison.

Author response:
Weakness1: General-purpose OpenAI embeddings are very strong baselines. Looking at the leaderboard on BEIR benchmarks (https://eval.ai/web/challenges/challenge-page/1897/leaderboard/4475), which reports ndcg@10, we see that OpenAI embeddings (our baseline) is very close to the leaders, and even beats the top leader on one dataset (Arguana). For SciFact, we improve over openai embeddings, and thus beat the state-of-the-art (on the ndcg@10 metric). As we discussed in Q1 above, we get higher relative gains on ndcg@1, which is our focus. Further, we explained in the paper (and Weakness 3, below) why we do not see gains across all benchmarks, and when we don’t, we do not deteriorate the performance of openai by much (and still beat the leaders on some benchmarks, such as Arguana).

Reviewer comment (Evaluation):
The paper claims to enable improved heterogeneous and strict retrieval, but it was not clear to me from the experimental results whether the improvement in strict retrieval has been substantiated with evidence.

Author response:
Q1. We claimed that our approach, Adder, improves strict retrieval based on the evidence that the relative improvement on ndcg@1 metric is mostly greater than the relative improvements we get on ndcg@10 (and ndcg@k for larger k). This can be verified for the data presented in Table 2 and 3. For example, in Table 3, Row 1 for smcalflow, we see a 50% relative improvement on ndcg@1 and a smaller 30% relative improvement on ndcg@10. In general, we observed that whenever there are gains, the relative gains on ndcg@k are higher for a smaller k. That is why we said that our approach is more suited for strict retrieval tasks. We will suitably justify our claims in the revised version.

Reviewer comment (Evaluation):
The improvements resulting from the proposed methodology appear limited in some instances, and it even underperforms compared to the baseline in the cases of FiQA and CONALA.

Author response:
Weakness 3: It is true that our approach does not uniformly improve performance across all benchmarks. However, there is good reason for it, as we discuss in the last paragraph of Section 4.1 and the last two paragraphs of Section 4.2. We believe that knowing when our approach improves over the baseline, and when it does not, is a strength of our work. It informs us on when to use an embedding adapter or when not. So, while the first impression from our results (Tables 2,3) may be that our approach does not work consistently, if we look deeper at the class of benchmarks where it does not improve over the base openai embeddings, one realizes that the results validate our hypothesis – use adapters when the retrieval task has nuances (code, semi-structured text) that the base model is not likely to exploit.  

Given that the OpenAI embeddings are widely trained on public data, including language and code, it is possible that they were also trained on some of the tasks/data from the BEIR dataset. That can make it difficult to further improve upon the OpenAI embeddings by tuning them on the same data. On the other hand, a weaker embedding model, like sBERT,  provides more opportunity for improvement by tuning it for the task. In the results presented below for SBert + Adder, we do see a consistent gain across all IR tasks ranging from 2x to 10x in nDCG@1 across datasets. Thus, the overall gains that we can expect from our approach depends on the initial pretraining of the underlying model.  

For the same BEIR benchmarks reported in Table 2, here are the results with using SBert as base. (nDCG@k values)  

| Dataset |   Base@1  |   ADDER@1 |   Base@3  |   ADDER@3 |   Base@5  |   ADDER@5 |   Base@10 |  ADDER@10|   
| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| Scifact | 0.01 |   0.17 |   0.02  | 0.19 | 0.02 | 0.22 | 0.03 | 0.24 |
| Arguana |   0.04 |   0.11 |   0.07 | 0.19 | 0.08 | 0.22 | 0.10 | 0.25 | 
| FiQA | 0.00 |   0.06 |   0.00 |   0.05 | 0.00 | 0.06 | 0.01 | 0.07 | 
| NFCorpus | 0.03 | 0.07 | 0.02 | 0.07 | 0.02 | 0.07 | 0.03 | 0.06 |

For the same task reported in Table 3, here are the results when using SBert as base: (nDCG@k values)    

| Dataset   |   Base@1  |   ADDER@1 |   Base@3  |   ADDER@3 |   Base@5  |   ADDER@5 |   Base@10 |   ADDER@10 |   
| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |  
|   M   |   0.04    | 0.76  |   0.08    |   0.86    |   0.09    |   0.88    |   0.11    |   0.89 |   
|   Conala  | 0.07  | 0.92  |   0.13    |   0.96    |   0.15    |   0.97    |   0.17    |   0.97 |   
|   Bash    | 0.05 |   0.81    |   0.07    |   0.89    |   0.09    |   0.9 |   0.1 |   0.91 |   
|   SMCalFlow   |  0.01 | 0.95 | 0.02 | 0.98 | 0.02 | 0.98 | 0.03 | 0.98 |

Reviewer comment (Evaluation):
How does the proposed approach improve strict retrieval compared to baselines? How do the experimental results support this improvement?

Author response:
Q1. We claimed that our approach, Adder, improves strict retrieval based on the evidence that the relative improvement on ndcg@1 metric is mostly greater than the relative improvements we get on ndcg@10 (and ndcg@k for larger k). This can be verified for the data presented in Table 2 and 3. For example, in Table 3, Row 1 for smcalflow, we see a 50% relative improvement on ndcg@1 and a smaller 30% relative improvement on ndcg@10. In general, we observed that whenever there are gains, the relative gains on ndcg@k are higher for a smaller k. That is why we said that our approach is more suited for strict retrieval tasks. We will suitably justify our claims in the revised version.

Reviewer comment (Experiments):
How is the dimension size of h selected? It was mentioned that h ranges from 16 to 128, but I could not find a discussion on this in the experiments. What are the pros and cons of choosing higher or lower values for h?

Author response:
Q2: We treat h, which is the rank of the LoRA transformation matrix, as a hyperparameter in our experiments. We selected the value of h from the set [16, 32, 64, 128, 256, 512, 768, 1024] through validation over the train/dev sets. We observed that the best values for h across the datasets we reported emerged to be around 256 (for ADDER) & 512 (for ADDER2). We also observed some robustness to the choice of h in our experiments – the reported metrics did not change very significantly when varying h in a range centred around the ‘best’ h determined by validation over the available data.