Paper: Personalized Language Generation via Bayesian Metric Augmented Retrieval (ICLR-2024)

Reviewer comment (Evaluation):
While authors aim to capture the characteristics of each user, they show that experimental results are search and textual quantitative evaluations, which do not lead to determine the level of achievement for this objective.

Author response:
Dear Reviewer, thank you for your feedback on our submission. Regarding the missing qualitative evaluation, we would like to provide an illustrative example from the **Law dataset**. Consider the query 'Explain Criminal Law to me', posed by a user whose profession is a `Dentist`. Our system knows nothing of this user but a few of his liked or disliked documents (a total of 18 documents in our experiment). Our system uniquely produces documents with analogies pertinent to medical or health-related fields, such as `patients` or `treatment or therapy`. This illustrates how our method tailors content to user-specific contexts, even with limited information about the user.

**Our method**: 
```
Criminal law refers to the body of laws that define and regulate behaviors that are considered to be crimes. You may encounter the concept of criminal law in your practice when dealing with **patients** who have engaged in illegal activities. Your expertise can be crucial in providing insights into their mental capacity, potential mitigating factors, or the need for **treatment or therapy**. 
```

**Euclidean**: 
```
Criminal law is a branch of law that deals with offenses committed against society as a whole, rather than individual disputes. This is similar to how physics principles, like the laws of motion, help maintain order and predict the movement of objects in the universe.
```

**BM25**: 
```
Criminal Law is like a set of guidelines that defines what actions are considered against the law and can result in punishment. Just like there are certain rules and principles you follow while creating your beautiful artwork, there are rules in society that everyone must abide by. You may find Criminal Law relevant to your work in developing AI systems that interact with society.
```
**MIPS**: 
```
Criminal law is a branch of law that deals with offenses committed against society as a whole, rather than individual disputes. By incorporating the principles of criminal law into your storytelling, you can create suspenseful plots, explore moral dilemmas, and delve into the psychological motivations behind criminal behavior. 
```

Reviewer comment (Evaluation):
The performance increase in Table 1 and 2 appears marginal and weak to support the claim.

Author response:
Dear reviewer,

We appreciate that you review our paper and provide insightful comments. We would like to address your concerns as follows:

> Did you perform statistical tests on Table 1 and 2? If so, please provide details.

We propose to look at the paired difference of the performance metrics: for each subject, we compute our method's HR@3, Rouge1 and the competing method’s HR@3, Rouge1. We propose to test the hypotheses:

Null hypothesis: Ours HR@3 and Rouge1 score equal competing method’s HR@3 and Rouge1

Alternative hypothesis: Ours HR@3 and Rouge1 are greater than competing method’s HR@3 and Rouge1.

In order to test the above hypothesis, we use a one-sided Wilcoxon signed-rank test to compare the paired performance metric values. The p-value of the test between our method and the baselines is reported in the following table.

Suppose we choose the significant level at 0.05. The table indicates that our method significantly outperforms baselines across all datasets. 

| | Datasets  | Politics               |   Law             | Technology |
|---------- |----------|--------------------|---------------------|---------------------|
|HR@3 |Ours - Euclidean | 1e-15 | 8e-16  | 8e-14 |
| |Ours - BM25 | 2e-18 | 2e-17 | 2e-18 |
| |Ours - MIPS | 7e-18  | 2e-17 | 2e-16 |
| |Ours - Logistic Regression | 7e-16  |5e-17 | 4e-16 |
|Rouge1 |Ours - Euclidean | 1e-12 | 1e-14 | 5e-15 |
| |Ours - BM25 | 1e-18 | 6e-17 | 2e-18 |
| |Ours - MIPS | 3e-10  | 4e-16 | 1e-14 |
| |Ours - Logistic Regression | 4e-17  | 7e-18 | 3e-17 |

Reviewer comment (Experiments):
Simplistic synthetic data: The only data used for experiments are synthetic with the conceived form of user preference being somewhat simplistic (different user types seeking stylistically different concept definitions).

Author response:
Thank you for your feedback on our response. Regarding your concerns about the synthetic setup of our approach, we wish to underscore that our paper's primary objective is to generate text aligning with the preferences of users. Our experimental approach is supported by multiple studies indicating the similarity between text generated by ChatGPT and human-generated text:

- Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text (Dou et al., ACL 2022)
- All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (Clark et al., ACL-IJCNLP 2021)

Consequently, simulating human-like responses using ChatGPT in our experiments is a reasonable approach.

Reviewer comment (Experiments):
Am I understanding correctly that the 5000 large document collection contains the 50 concept definitions rephrased in 100 different ways?

Author response:
Yes, it is correct. As mentioned in Section 6.1, we synthesize 3 datasets (Political, Legal and Technology) with that setup.

Reviewer comment (Writing):
Am I understanding correctly that the proposed approach never uses the pre-trained summarization model in learning user preferences?

Author response:
Yes, it is correct. However, we did use a pretrained summarization model to shorten documents for practical purposes.

Reviewer comment (Experiments):
Is the user feedback for Phase 2 obtained from labelled data? Eg if the document corresponds to the correct concept+user it is a positive document and negative if not?

Author response:
A user likes a document if that document lies in the top 5 documents that are closest to the user’s true document in terms of Euclidean distance; A user dislikes a document otherwise.

Reviewer comment (Experiments):
Please clarify in greater detail what data is used for training and evaluation. "For each user type, we systematically exclude all documents associated with that user type from the database, treating them as reference documents" -- its not clear what this means? Is this implying that no documents corresponding to the user type are available for training?

Author response:
Yes, the document corresponding to the user type will not be used for training but only used as a reference document to compute the top-5 documents that will be liked by the user.

Reviewer comment (Novelty):
The paper contains no context on related work from personalized search, it would be useful to make some connection to related prior work. Starting points may be: https://dl.acm.org/doi/10.1145/1076034.1076111, https://dl.acm.org/doi/10.1145/3357384.3357980, https://dl.acm.org/doi/10.1145/2124295.2124348. Also consider discussing differences to the task/approach of: https://arxiv.org/abs/2211.09260

Author response:
The work in Task-aware Retrieval with Instructions by Asai et al. (2022) adapts its search results using an additional search task instruction $t$ in natural language, along with the user's query $q$ (see Section 3, 'Task Formulation'). Our work differs from that of Asai et al. (2022) in that we rely on the users' ratings (like/dislike) on a selection of summarized documents, instead of using the task instruction $t$.

Reviewer comment (Experiments):
Instead of the proposed complex model, a simple linear model could potentially be fit to the user preferences, but this comparison is not explored.

Author response:
> What happens when you fit a simple linear model to users original preferences and update in an online fashion?

As mentioned in Section 6.2, we only ask users to rate a total of 18 shortened documents (N_c = 3, C1 = 3, C2 = 3). This number of samples is much too low for fitting a simple logistic regression binary classifier (with two classes, `like` or `dislike`) and will lead to overfitting. We, anyways, have implemented the baseline that you proposed, in which we learn a logistic regression model to predict whether the user likes or dislikes each document. We conduct the comparison on the Politics, Law and Technology dataset. The table below presents the performance between two models.  
We can clearly observe that our proposed method dominates the simple logistic regression baseline.

|  Dataset  | Methods               |  HR@3             | Precision@3 | Rouge1 | Rouge2| 
|----------|--------------------|---------------------|---------------------|---------------------|---------------------|
| Politics| Logistic Regression | 0.59 | 0.3036  | 0.3472  |  0.0975 |
| | Ours | **0.6849** | **0.3751**  |  **0.3701**  |  **0.1109** |
| Law | Logistic Regression |  0.4809 | 0.2521  | 0.3640  |  0.0933 |
| | Ours | **0.6176** | **0.3199** |  **0.3868**  |  **0.1210** |
| Technology | Logistic Regression |   0.5485 | 0.2704  | 0.3455  |  0.0950 |
| | Ours | **0.6803** | **0.3715**  |  **0.3654**  |  **0.1130** |

Reviewer comment (Theory):
Many design choices in the proposed model are not justified, including the decision to model user preference as a Wishart probability distribution and the use of a determinantal point process.

Author response:
> Why are we modeling the user preference as a Wishart probability distribution?

Because a popular approach for document retrieval is the nearest neighbor approach. The standard, off-the-shelf implementation uses an Euclidean distance. Thus, a reasonable approach to cater the user preference in the retrieval procedure is to alternate the distance. We thus need to parametrize the distance so that we can perform the learning process. Mahalanobis distance is the most natural way to generalize the Euclidean distance, and a Mahalanobis distance is parametrized by a positive semidefinite matrix (this is the matrix $A$ in our paper). To invoke a Bayesian method, we need to impose a prior and a posterior on the parameter space, and the parameter space in our case is the set of all a positive semidefinite matrices. And the Wishart distribution is, again, the most natural distribution supported on the positive semidefinite matrix space. We hope that this line of argument clarifies your concern.

Reviewer comment (Experiments):
The benefits of using a determinantal point process over simpler methods such as random sampling are not explained or compared.

Author response:
> Why do we need a determinantal point process? How would that compare to random sampling?

Detrimental Point Process is needed because when a user arrives, there is no user preference information that we can use to personalize the retrieval process. This is the cold-start setting, and diversifying the retrieval via DPP is a well-known and powerful technique in this cold-start situation. Many papers have demonstrated that DPP delivers better performance than random sampling, including:  
[1] Wang et al. (2020) Personalized Re-ranking for Improving Diversity in Live Recommender Systems, arXiv:2004.06390  
[2] Liu et al. (2022) Determinantal Point Process Likelihoods for Sequential Recommendation, arXiv:2204.11562  
[3] Wilhelm et al. (2018) Practical Diversified Recommendations on YouTube with Determinantal Point Processes.  
When we design our framework, we have already optimized the choice of the component. That is why we use a determinantal point process. We will add this argument to the revision.

Reviewer comment (Experiments):
It is unclear what would happen if a simple linear model were fit to the users’ original preferences and updated in an online fashion.

Author response:
> What happens when you fit a simple linear model to users original preferences and update in an online fashion?

As mentioned in Section 6.2, we only ask users to rate a total of 18 shortened documents (N_c = 3, C1 = 3, C2 = 3). This number of samples is much too low for fitting a simple logistic regression binary classifier (with two classes, `like` or `dislike`) and will lead to overfitting. We, anyways, have implemented the baseline that you proposed, in which we learn a logistic regression model to predict whether the user likes or dislikes each document. We conduct the comparison on the Politics, Law and Technology dataset. The table below presents the performance between two models.  
We can clearly observe that our proposed method dominates the simple logistic regression baseline.

|  Dataset  | Methods               |  HR@3             | Precision@3 | Rouge1 | Rouge2| 
|----------|--------------------|---------------------|---------------------|---------------------|---------------------|
| Politics| Logistic Regression | 0.59 | 0.3036  | 0.3472  |  0.0975 |
| | Ours | **0.6849** | **0.3751**  |  **0.3701**  |  **0.1109** |
| Law | Logistic Regression |  0.4809 | 0.2521  | 0.3640  |  0.0933 |
| | Ours | **0.6176** | **0.3199** |  **0.3868**  |  **0.1210** |
| Technology | Logistic Regression |   0.5485 | 0.2704  | 0.3455  |  0.0950 |
| | Ours | **0.6803** | **0.3715**  |  **0.3654**  |  **0.1130** |