Paper: Polarity-Aware Semantic Retrieval with Fine-Tuned Sentence Embeddings (ICLR-2024)

Reviewer comment (Writing):
The paper is poorly written, and the motivation and problem definition are not clear.

Author response:
I agree that our motivation and problem definition could be much better stated. And for that I thank you and some of the other reviewers in pointing out.

Reviewer comment (Novelty):
The problem studied is not insightful and the findings are trivial.

Author response:
There is a real use-case to retrieving with multiple objectives, and, as others have stated, relates to the catastrophic forgetting when fine-tuning these models. We showed that you can get a better trade-off than what is currently available with the architecture as-is, but by augmenting input data instead. I have since the reviews came in studied this in more depth and we out-perform alternatives such as SetFit in all related benchmarks with SentEval https://github.com/facebookresearch/SentEval, although we did not train for any of these objectives.

Reviewer comment (Writing):
Could you give a formal definition of the polarity? From my understanding, it is according to the label in the original dataset.

Author response:
The polarity is the label. If this was unclear, we are strictly using binary labeled datasets.

Reviewer comment (Novelty):
What is your motivation for this study and what is your takeaway conclusion? The experimental results are kind of trivial to me. “Polarity” and “Semantic” can have distribution overlap but are not exactly the same. So if you finetune the sentence embedding methods based on polarity perspective, it is quite common that it will suffer on the semantic dimension.

Author response:
The main motivation is rooted in a real-world application of e.g. retrieving K items that are similar on multiple objectives. An example would be a news article that has high similarity with another, while they may be connected to two entirely different topics (e.g. by sentiment). The entire point shown here is that we can increase the polarity far above the reduction of similarity, and this trade-off can be reduced by the configurations we evaluated.

Reviewer comment (Writing):
What is w_i in sec 3.2.2? Is it the same as that in sec 3.2.1?

Author response:
They are shared. We have since clarified this. The paper will be updated.

Reviewer comment (Reproducibility):
What is the reference model for semantic calculation used in sec 3.3 and sec 4? If I am understanding correctly, in section 4, you are using the base sentence embedding model without further finetuning as the reference model. Is it the same in sec 3.3?

Author response:
The reference model is always the respective baseline model (i.e. if the fine-tuned model is based on model A, the reference is A)

Reviewer comment (Reproducibility):
How do you set \beta in sec 3.2.2?

Author response:
As mentioned, we opted for removing this common metric, as it provided little value in hindsight.

Reviewer comment (Novelty):
This paper lacks technical contribution. I can't see an originally proposed model architecture for sentence embedding problem. Instead, this paper looks more like a summary of why fine-tuning is an important technique for sentence embedding, but fine-tuning is also not a new concept and has been adopted by many previous works, including most LLMs.

Author response:
I have questions regarding your definition of "technical contribution". While we do not propose a change in model architecture, we have shown that a change in the data, namely by augmenting input data on supervised data not related to sentence similarity, leads to improved retrieval scores for specific configurations. Fine-tuning on multiple objectives is vastly different from studying a single metric on existing datasets when fine-tuning. Otherwise, refer to the other comments.

Reviewer comment (Evaluation):
When we do experiments, we usually need to repeat the same experiment multiple times and report both mean and standard deviation, in order to verify that the proposed model indeed significantly outperforms baselines. However, I can see mean but not standard deviation in the experiments.

Author response:
Thanks for this. Will update the paper. From initial inspection, there are no changes to our conclusions based on the std, but is surely a much-needed addition.

Reviewer comment (Novelty):
Although the paper proposed a new formulation of the text retrieval task, it did not mention why the task was important in the downstream applications or in further investigation of model capacity. Thus, the results of this paper provided limited implications.

Author response:
After the reviews came in, we attempted to make our statements more clear by its transferability to other applications, and did so using the established evaluations found in SentEval. We found our approach to outperform in all cases compared to e.g. SetFit on the same data with the same sample size. This should likely have been done beforehand, but we did not consider severely out-of-domain data as we were strictly focused on in-domain retrieval from an existing data source in the real world.

Reviewer comment (Experiments):
In the direction of the model investigation, the experiment setup was based on a single dataset, i.e., fine-tuning and retrieving the same dataset. Consequently, the author could only conclude that the proposed method retained sentence similarity performance within the fine-tuned dataset. It is unclear how this is better than the standard benchmarks that included zero-shot classifications and unsupervised sentence retrieval.

Author response:
After the reviews came in, we attempted to make our statements more clear by its transferability to other applications, and did so using the established evaluations found in SentEval. We found our approach to outperform in all cases compared to e.g. SetFit on the same data with the same sample size. This should likely have been done beforehand, but we did not consider severely out-of-domain data as we were strictly focused on in-domain retrieval from an existing data source in the real world.

Reviewer comment (Novelty):
The problem addressed in this paper is relevant to the "forgetting effect" and its solutions, such as Chen et al., 2020 or Luo et al., 2023. The author should discuss this in the related work.

Author response:
Thanks for these resources.

Reviewer comment (Evaluation):
Figure 4 was quite insightful, did the "max" models remain consistent?

Author response:
The max models is the ideal result, and is nearly achieved by the best configurations shown in the end of the article, in table 7/8. We opted for changing this figure to box plots of similarity and polarity, giving more insights into the deviation in metrics per sample size.

Reviewer comment (Experiments):
Could you clarify why you did not test the fine-tuned models on STS using a standard benchmark?

Author response:
As our goal is not generalized STS, I agree that we should have performed tests on established benchmarks, if not only for the sake of comparison to approach similar scores as the baseline while increasing scores on retrieval-based tasks within the same domains. As explained in comments below, we ended up rerunning experiments with our models for the tasks provided by SentEval (https://github.com/facebookresearch/SentEval) on various problems. Results are consistently better than the pre-trained alternatives for similar tasks like opinionated texts and reviews when fine-tuned on SST-2, and also outperforms the alternative SetFit model.