Paper: SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings (ICLR-2024)

Reviewer comment (Novelty):
Representing set operation with embedding is not novel, but the comparison and discussion compared with existing methods are missing.  
  - Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, Nils Y. Hammerla Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors. ICLR 2019.  
  - Siddharth Bhat, Alok Debnath, Souvik Banerjee, and Manish Shrivastava. Word Embeddings as Tuples of Feature Probabilities. RepL4NLP. 2020.  
  - Shib Dasgupta, Michael Boratko, Siddhartha Mishra, Shriya Atmakuri, Dhruvesh Patel, Xiang Li, and Andrew McCallum. Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings. ACL2022.

Author response:
1. **Representing set operation with embedding is not novel, but the comparison and discussion compared with existing methods are missing.**

   We would like to thank the reviewer for sharing those papers that combines word embeddings with set theory or set-theoretic operations. Our original review focused more on the works that apply set theory in word representation and comprehension, however, research that employees set theory for word embedding quality improvements are not addressed. The citations suggested by the reviewer have been incorporated into the manuscript. 

   Additionally, in the updated manuscript, we include a detailed discussion comparing SetCSE with these existing methods. Specifically, we aim to underscore the following as the novelty of our work:
   * SetCSE employs **sentence embeddings** for semantic representation and information retrieval, diverging from the prior focus of the mentioned works on using and improving **word embeddings**.
   * SetCSE utilizes sets of sentences and its learning mechanism to recognize and represent complex and intricate semantics for information querying. This approach differs from previous works, which did not consider the collective use of words to represent complex semantics.
   * SetCSE integrates set-theoretic operations for expressing complex queries in practical sentence retrieval tasks, distinguishing it from previous works that used set operations to uncover word relationships.

Reviewer comment (Experiments):
The quantitative evaluation is performed only in artificial settings, and there are only case studies for the application results. It is unclear how the method can be stably used for the application.

Author response:
3. **The quantitative evaluation is performed only in artificial settings, and there are only case studies for the application results. It is unclear how the method can be stably used for the application.**

   We appreciate the reviewer's suggestion. In order to better demonstrate the stability of SetCSE performance in serial operations, we have included the following experiments and evaluations:
   * Series of two SetCSE intersection operations.
   * Series of two SetCSE difference operations.
   * Series of SetCSE intersection and difference operations.
   
   The experiment are described in the Appendix C.4 of the updated manuscript. Since we are evaluating the performance of SetCSE serial operations, the multi-label datasets are employed. To encompass diverse contexts, we consider the datasets GitHub Issue (Ismael, 2022), English Quotes (Eltaief, 2022), and Reuters-21578 (Lewis, 1997). The detailed results can be found in the manuscript Tables 8 to 10 (we include one set of results below). As those results indicate, SetCSE improves the aforementioned serial operations by 26%, 39%, and 34%, respectively. In most of the cases, the accuracy and F1 in these experiments are above 70%. 

   **Table 8. Evaluation results for series of SetCSE intersections.**
   |      | GitHub-HD |   Quotes-FL   | Quotes-FI  | Reuters-SG | Reuters-SC |
   | :---        |    :----:   |   :----:   |   :----:   |   :----:   |        :---: |
   | Metrics     | Acc  F1 | Acc  F1 | Acc  F1 | Acc  F1 | Acc  F1 |
   | Ave. Improvement   | 55% 51% | 22% 22% | 8% 7% | 38% 34% | 13% 11% |


   In summary, the stability of SetCSE can be evaluated through four assessments:
   * Performance on single SetCSE operations (Section 5)
   * Performance on series of SetCSE operations (Appendix C.4)
   * Case studies presented in the application section (Section 6)
   * The error analysis corresponding to the case studies (Appendix D.1)

   We hope the above quantitative and qualitative results could provide more insights into the performance of SetCSE.

Reviewer comment (Experiments):
The paper would benefit from an additional experiment on sentence retrieval, similar to the one in Section 6.1. In this setting, the paper could compare its performance against traditional retrieval-based methods such as DPR and BM25 to better illustrate the model's improvements. Furthermore, in Table 2, certain baseline models do not show significant improvements with SetCSE. For instance, the improvements for SGPT are marginal when compared to other baselines. The paper should include an analysis explaining the variations in improvements among different models.

Author response:
**The paper would benefit from an additional experiment on sentence retrieval, similar to the one in Section 6.1. In this setting, the paper could compare its performance against traditional retrieval-based methods such as DPR and BM25 to better illustrate the model's improvements.** 

    We appreciate the suggestion. In response, we have added BM25 and DPR as two baseline and compared with SetCSE. The results are included in **Tables 1** and **2** of the updated manuscript. We find that the above methods are less performant compared to SetCSE embedding models.
   
   **Table 1. Evaluation results for intersection operation.**
   |      | AGT |   AGD   | FPB  | Banking77 | FMTOD |
   | :---        |    :----:   |   :----:   |   :----:   |   :----:   |          ---: |
   | BM25     | 24.90 24.90 | 25.02 25.02 | 33.40 38.91 | 41.25 41.32 | 37.59 41.19 |
   | DPR   | 25.00 25.00 | 25.00 25.00 | 33.33 38.81 | 41.30 41.35 | 38.33 42.87 |

   **Table 2. Evaluation results for difference operation.**
   |      | AGT |   AGD   | FPB  | Banking77 | FMTOD |
   | :---        |    :----:   |   :----:   |   :----:   |   :----:   |          ---: |
   | BM25     | 57.34 57.47 | 59.95 57.57 | 69.22 60.77 | 68.76 68.14 | 73.97 71.74 |
   | DPR   | 56.50 54.45 | 56.98 54.76 | 66.67 58.06 | 68.70 68.06 | 71.67 68.43 |

   $ $

2. **Furthermore, in Table 2, certain baseline models do not show significant improvements with SetCSE. For instance, the improvements for SGPT are marginal when compared to other baselines. The paper should include an analysis explaining the variations in improvements among different models.**
   
   We have included a detailed discussion regarding the performance of SGPT embedding model and potential resolution in Appendix C.3 of our updated manuscript. Specifically, the following are addressed:

   In Section 5, our evaluations demonstrate that the decoder-only SGPT-125M performs less effectively compared to encoder-based models of similar sizes, both before and after inter-set contrastive learning stages. This observation aligns with findings from other studies that compare embeddings produced by BERT-based models and GPT in benchmark word embedding tasks (Ethayarajh, 2019; Liu et al., 2020; Cai et al., 2020).

   Since our evaluation indicates that SGPT benefits less from inter-set fine-tuning, future studies may consider other contrastive learning methods (Jian et al., 2022; Jain et al., 2023) to enhance the context awareness and discriminatory capabilities of decoder-based models.

   $ $

Reviewer comment (Experiments):
The core concept is to create clusters of sentences with semantic meaning, and this can limit the generalization ability of the proposed framework. In comparison to other baselines, the incorporation of semantic meaning within sets naturally provides additional information for training.

Author response:
**The core concept is to create clusters of sentences with semantic meaning, and this can limit the generalization ability of the proposed framework. In comparison to other baselines, the incorporation of semantic meaning within sets naturally provides additional information for training.**

   We appreciate this comment from the reviewer. Indeed, the inter-set contrastive learning proposed in our framework aims to reinforce the semantics represented by sample sets. Hence, the learning mechanism introduced context-specific fine-tuning to the model. Evaluation presented in **Tables 1**, **2**, and **Tables 8** to **10** of the updated manuscript show that this improvements on the awareness of the query task specific semantics do enhance the query performance. In addition, the fine-tuning computation is not costly.

   Furthermore, we incorporated a study in the updated manuscript to evaluate the inter-set fine-tuned models' capability to conduct general purpose NLU tasks. We conducted 7 STS tasks for the inter-set fine-tuned model, and find that the context-specific learning does not significantly impact the generalization of the underlying model, only decreasing the task performance by 7%.
    
   Combining the fact that the fine-tuning significantly improves the querying performance, being computationally efficient, and does not create a large impact to the model's generalization capability, we think the proposed learning is worthwhile.

   $ $

Reviewer comment (Reproducibility):
The paper fails to provide code.

Author response:
**The paper fails to provide code.**
   
   We appreciate the interest of the reviewer regarding codes. The codes have been uploaded to the supplementary material section.

   $ $

Reviewer comment (Theory):
How to adapt the proposed method to a situation where sentences are not clustered with semantic meaning or where the clusters do not exist?

Author response:
**How to adapt the proposed method to a situation where sentences are not clustered with semantic meaning or where the clusters do not exist?**

   This potential challenge could be resolved by incorporating samples created by domain experts or generative models. For instance, if in some cases samples for certain semantics are less than say 20 or 30, and those semantics one wants to query are complex and intricate in natural, one could consult the domain expert for more samples. If the semantics are less involved, one can manually create or even query generative models for more samples. In Section 7, we also indicate that the samples needed for SetCSE to yield good results are not very large, practically, 20 to 30 samples with diversity should be sufficient.

Reviewer comment (Evaluation):
There is no mention of an error analysis which would be beneficial in understanding the limitations of SetCSE in certain scenarios.

Author response:
1. **There is no mention of an error analysis which would be beneficial in understanding the limitations of SetCSE in certain scenarios.**

   We appreciate reviewer's constructive suggestion. In response, we have include the error analysis the error analysis with respect to Section 6.1 use cases in **Table 11** of the updated manuscript. Specifically, **Tables 11a** and **11b present** the less preferable query results compared to **Tables 3b** and **3c**, respectively. It’s important to note that the presented results are not ranked among the top sentences in the corresponding SetCSE querying outputs. Instead, they are listed between the 30th to 50th positions within the sentences of the S&P500 earnings call dataset (Qin & Yang, 2019). See below as a highlight.

   **Table 11a. Error analysis for the query ''*using technology to solve Social issues, while neglecting its potential negative impact*".**
   | |
   | :---------- |
   | Finally, there is one commonality our **customers** have, it’s that they live in a hybrid **IT world**. |
   | We had good **bio-security** to begin with, but we amped it up to an all-time high level of **discipline and scrutiny**, frankly, and it hasn’t stopped.|
   | It is fairly incremental in terms of adding things like **customer support**, **field application engineering, software support**, given that we’re familiarizing people with our architecture.|
   | |

   Based on the findings in **Table 11**, we identify the following issues that may lead to less preferred results:
   * Certain words in the query results closely match sample sentences in terms of represented semantics, leading to their respective rankings. However, the entire sentences show less relevance to those specific semantics. For instance, word “*customer*” may align with the sample phrase “*customer satisfaction*” yet the entire sentence might be less associated with “*Social issues*”.

   * Some sentences are chosen due to their high relevance to a single presented semantic, while an intersection of multiple semantics is anticipated. For example, the sentence “*We had good **bio-security** to begin with, but we amped it up to an **all-time high level of discipline and scrutiny**, frankly, and it hasn’t stopped.* ” aligns closely with “*Social issue*” but displays less relevance to “new technology”.

   It’s essential to note that the mentioned examples are not ranked among the top sentences in the query outputs, thereby naturally leading to errors. We aim to improve the assessment of the closeness between a sentence and sets of semantics to mitigate some of these aforementioned errors.

   $ $

Reviewer comment (Experiments):
The applications of complex semantic search, data annotation, and new topic discovery are very cool with the detailed examples, but there is not quantification here or comparison with others with existing set methods from the literature (same with Table 1 and 2 as well). Do you have comparisons with other methods from the literature on this topic?

Author response:
2. **The applications of complex semantic search, data annotation, and new topic discovery are very cool with the detailed examples, but there is not quantification here or comparison with others with existing set methods from the literature (same with Table 1 and 2 as well). Do you have comparisons with other methods from the literature on this topic?**

   We would like to thank the reviewer for the two suggestions: (1). further quantifying SetCSE performance, (2). compare SetCSE with more works presented in the literature. 

   **In response of the first suggestion**, an extensive evaluation on SetCSE serial operations performance is presented in Appendix C.4, which considers the following three SetCSE serial operations:
   * two SetCSE intersection operations.
   * two SetCSE difference operations.
   * SetCSE intersection and difference operations.

   We employ the multi-label datasets for serial operation evaluation. To encompass diverse contexts, we consider GitHub Issue (Ismael, 2022), English Quotes (Eltaief, 2022), and Reuters-21578 (Lewis, 1997) datasets. Detailed results can be found in the Tables 8-10 of the updated manuscript. As an example, the results indicate an average improvement of 34% in the performance of serial operations using SetCSE.

   **Table 8. Evaluation results for series of two SetCSE intersections.**
   |      | GitHub-HD |   Quotes-FL   | Quotes-FI  | Reuters-SG | Reuters-SC |
   | :---        |    :----:   |   :----:   |   :----:   |   :----:   |        :---: |
   | Metrics     | Acc  F1 | Acc  F1 | Acc  F1 | Acc  F1 | Acc  F1 |
   | Ave. Improvement   | 55% 51% | 22% 22% | 8% 7% | 38% 34% | 13% 11% |

   **In response to the second suggestion**, we included the following to the manuscript:

   * More related works that combines Set Theory with embeddings (Section 2). We indicate that these works only focus on at word level, with a end-goal of improving word embeddings or revealing word relationships, where SetCSE operates on sentence level, with a purpose of building a robust and practical querying framework for complex semantics.
   * A quantitative evaluation for the existing information retrieval methods (Section 5, Tables 1 and 2), including BM25 and DPR. We found that SetCSE performance is more than two times compared with these methods in terms of accuracy and F1.

Reviewer comment (Writing):
Typos: Section 7 "DISUCSSION"

Author response:
5. **Typos: Section 7 "DISUCSSION"**

   The typo has been corrected. We appreciate the reviewer's help.

Reviewer comment (Evaluation):
Can the authors discuss any observed limitations or frequent error patterns during SetCSE operations?

Author response:
3. **Can the authors discuss any observed limitations or frequent error patterns during SetCSE operations?**
   
   Please see response 2 for error analysis. In terms of other observed limitations, we want to point out that the SetCSE querying framework depends on the presented samples to improve the model discriminatory capability over complex semantics. This context-specific method is not designed for general NLU benchmark tasks, and we do observe a small decrease in model performance in standard STS tasks (see **Table 14** of the updated manuscript). However, this would not affect the usage of SetCSE as a querying framework, as we see that the inter-set contrastive learning significantly improves model performance in information retrieval when complex semantics and query structures are involved.

   $ $

Reviewer comment (Experiments):
How does SetCSE scale with the size of the dataset and the complexity of the query semantics?

Author response:
4. **How does SetCSE scale with the size of the dataset and the complexity of the query semantics?**

   We thank the reviewer for the question of SetCSE scalability. To illustrate, we include the following table regarding the five dataset used in our quantitative evaluations in Section 5 for more insights:

   **Table. Statistics for dataset AGT, AGD, FPB, Banking77, and FMTOD.**

   |              | AGT |   AGD   | FPB  | Banking77 | FMTOD |
   | :---        |    :----:   |   :----:   |   :----:   |   :----:   |   :---:   |
   | # of sentences     | 12,010 | 12,010 | 5,717 | 10,003 | 33,698 |
   | Ave. sentence length  | 41.97 | 193.55 | 122.71 | 59.47 | 35.70 |

   From the above table, we find that the AGD and FPB datasets are equipped with longer sentences, while FMTOD dataset contains most sentences among other datasets. In addition, the semantics presented in AGT, AGD and FPB are more general hence harder to "comprehend" by the model, whereas semantics in Banking77 and FMTOD are more specific, and easy to identify based on the key words presented. Based on the above, and the results presented in **Table 1** and **2** in the manuscript, we conclude that SetCSE performance are less impacted by the size of the dataset, while it generally performances better when semantics are specific instead of general.

   $ $