Paper: Learning Personalized Story Evaluation (ICLR-2024)

Reviewer comment (Presentation):
The description of the paper is too fragmented and less logical.

Author response:
A1. We have reorganized the paper to make it clearer. We refer the reviewer to the general response for more detailed information.

Reviewer comment (Presentation):
The dataset construction section is too cursory, it is recommended to refine it and provide a flowchart.

Author response:
A2. Thanks for the valuable suggestions! We add a flowchart for the data construction in Figure 2 and demonstrate our PerSE framework in Appendix Figure 11.

Reviewer comment (Evaluation):
The authors need further clarity on the practical implications of the research work as well as specific application scenarios.

Author response:
A3. In this paper, we mainly focus on 

1. *introduce two personalized story evaluation datasets.* By elaborating on the data construction, these two datasets are less affected by the contamination issue of LLM-based evaluation models. They can be used as better benchmarks for both training and testing of the personalized story evaluation models. The pipeline can be easily adapted to other tasks.

2. *propose PerSE to analyze the unseen reviewers’ preferences from their prior reviewers and predict specific reviews from their views*. By using a small amount of data as the instructions, we enhance LLMs’ capabilities of modeling the reviewer’s preference. The experimental results show that PerSE can predict personalized reviews for new users with only 3 prior reviews.

Thanks to this capability, PerSE can be used in **the recommendation system to select the most suitable story** for a new reviewer without finetuning the preference data of this reviewer. For example, given several stories and three reviews from a new user, we can use PerSE to rank them and provide suggestions for this user.
PerSE can further **tailor story generation models to align with the unique needs and preferences of different users** by using the score of PerSE as the reward.

Reviewer comment (Writing):
Lack of Clarity in Model Attribution in contribution statements: The paper needs to explicitly specify that "Current LLMs" refer to models like ChatGPT-4, and it should distinguish that the pronoun "they" in "with instruction-tuning on several thousands of data, they can" pertains to the proposed method based on LLaMA-2. Failing to do so may lead to ambiguity, making it difficult to understand the paper's contributions.

Author response:
**Q2. Lack of clarity in model attribution in contribution statements. "Current LLMs" refer to models like ChatGPT-4, and it should distinguish that the pronoun "they" in "with instruction-tuning on several thousands of data, they can" pertains to the proposed method based on LLaMA-2.**

A2. Here the “current LLMs” refer to the direct usage of LLMs as the evaluation model, and “they can” indicates the instruction-turned LLMs. Both of them can refer to LLaMA-based or GPT-4-based models. The main difference here is before and after instruction-turning. We have revised the contribution part.

Reviewer comment (Evaluation):
Ambiguity in Metric Basis: While the paper mentions reporting prediction accuracy as the primary metric for comparative evaluation, it doesn't make it clear what this accuracy is based on. The reference to "prediction accuracy" needs to be explicitly linked to the mentioned Pearson correlation or other relevant metrics, providing a clear understanding of what's being measured.

Author response:
**Q1.  Ambiguity in Metric Basis: clarify what specific metrics this "prediction accuracy" is based on. Is it linked to the previously mentioned Pearson correlation, or are there other relevant metrics involved?**

A1. The prediction accuracy is the accuracy of binary classification, which is calculated by $accu = \frac{\text{the number of true prediction}}{\text{the total example size}}$. We view the comparison of each aspect as a binary classification between the given two plots. We also clarify this point in the revised manuscript.

Reviewer comment (Experiments):
Limited Diversity in Baseline Models: The paper primarily relies on LLM-based approaches for score prediction, neglecting the inclusion of various other methods commonly used in the field, such as matrix factorization and other score (rating) prediction techniques. To provide a more comprehensive evaluation, it would be beneficial to incorporate these alternative baseline models for a more thorough comparison.

Author response:
**Q3. Limited Diversity in Baseline Models: For not including a wider range of commonly used methods in the field, such as matrix factorization or other score prediction techniques, as part of the baseline models for comparison?**

A2. We add matrix factorization baseline[1] on Per-MPST for the main Table 2 (previous Table 3) as below. However, on Per-Doc, both plot pairs and the annotators of the test set have no overlapping with the training set, so the matrix factorization cannot apply to this setting. MF baseline has a similar performance to the simple baseline, which averages the score in the preference. 

|             | Pearson | Spearman | Kendall |
|-------------|---------|----------|---------|
| simple baseline    |   0.301 |    0.302 |   0.230 |
| MF          |   0.308 |    0.313 |   0.269 |
| llama-2-7b  |   0.146 |    0.117 |   0.094 |
| llama-2-13b |   0.172 |    0.182 |   0.147 |
| llama-2-70b |   0.214 |    0.232 |   0.181 |
| GPT-4       |   0.315 |    0.312 |   0.253 |
| PerSE-7b    |   0.307 |    0.329 |   0.263 |
| PerSE-13b   |   0.345 |    0.368 |   0.293 |




[1] Matrix factorization techniques for recommender systems

Reviewer comment (Evaluation):
Ambiguity in Metric Basis: The paper mentions "prediction accuracy" as the primary metric for comparative evaluation. Could you please clarify what specific metrics this "prediction accuracy" is based on? Is it linked to the previously mentioned Pearson correlation, or are there other relevant metrics involved? Providing this clarification will enhance the understanding of the evaluation process.

Author response:
**Q1.  Ambiguity in Metric Basis: clarify what specific metrics this "prediction accuracy" is based on. Is it linked to the previously mentioned Pearson correlation, or are there other relevant metrics involved?**

A1. The prediction accuracy is the accuracy of binary classification, which is calculated by $accu = \frac{\text{the number of true prediction}}{\text{the total example size}}$. We view the comparison of each aspect as a binary classification between the given two plots. We also clarify this point in the revised manuscript.

Reviewer comment (Experiments):
Limited Diversity in Baseline Models: The paper predominantly focuses on LLM-based approaches for score prediction. Are there any specific reasons for not including a wider range of commonly used methods in the field, such as matrix factorization or other score prediction techniques, as part of the baseline models for comparison? Including these alternatives could potentially offer a more comprehensive evaluation. Could you share your insights on this choice?

Author response:
**Q3. Limited Diversity in Baseline Models: For not including a wider range of commonly used methods in the field, such as matrix factorization or other score prediction techniques, as part of the baseline models for comparison?**

A2. We add matrix factorization baseline[1] on Per-MPST for the main Table 2 (previous Table 3) as below. However, on Per-Doc, both plot pairs and the annotators of the test set have no overlapping with the training set, so the matrix factorization cannot apply to this setting. MF baseline has a similar performance to the simple baseline, which averages the score in the preference. 

|             | Pearson | Spearman | Kendall |
|-------------|---------|----------|---------|
| simple baseline    |   0.301 |    0.302 |   0.230 |
| MF          |   0.308 |    0.313 |   0.269 |
| llama-2-7b  |   0.146 |    0.117 |   0.094 |
| llama-2-13b |   0.172 |    0.182 |   0.147 |
| llama-2-70b |   0.214 |    0.232 |   0.181 |
| GPT-4       |   0.315 |    0.312 |   0.253 |
| PerSE-7b    |   0.307 |    0.329 |   0.263 |
| PerSE-13b   |   0.345 |    0.368 |   0.293 |




[1] Matrix factorization techniques for recommender systems

Reviewer comment (Reproducibility):
There are missing technical details such as how exactly an anonymized plot is summarized.

Author response:
We use oasst-30b[1] to summarize the plots by prompting it to give a summary of the main idea of the anonymous plot. We add a flowchart for data construction in Figure 2 and provide the detailed prompts we use for anonymization and summarization in Appendix Figure 9. We also refine the description in Section 3 to make it clearer.

Reviewer comment (Experiments):
The datasets generated seem to be in a small scale. To show the proposed methods are scalable and widely applicable, it'd be great if much larger scale datasets could be used.

Author response:
One of our motivations for re-proposing Per-MPST and Per-DOC is the difficulty of collecting personalized evaluation data. Because of this difficulty, the fact that PerSE still works well with our current smaller dataset sizes is also a strength of our method - it doesn't need the larger datasets to work fine. Moreover, our Per-MPST has ~10k data for training and ~1k for testing, which is comparable to other personalized text classification and generation datasets[2].

Reviewer comment (Evaluation):
What is the significance of the comparison results between PERSEcomp-13b and PERSEcomp-7b in Table 4? Providing t-test results can be useful.

Author response:
We perform the t-test between PERSEcomp-13b and PERSEcomp-7b for Table 4 with *scipy.stats.ttest_ind* and report the p-value here. All p values are smaller than 0.05, indicating that these two models significantly differ from each other.

|    | p-value  |
|----|----------|
| Q1 | 2.78E-10 |
| Q3 | 1.50E-06 |
| Q4 | 3.32E-05 |
| Q5 | 3.12E-08 |
| Q6 | 8.65E-05 |

Reviewer comment (Evaluation):
The paper could benefit from a discussion of some factors about the practicalities of using an approach like PerSE for evaluation; for example, what happens if there isn't training data for PerSE in a domain? How biased is PerSE toward LLaMa generations? others asked in the questions below.

Author response:
**Q1. What happens if there isn't training data for PerSE in a domain?**

A1. In principle, PerSE is not specifically trained to evaluate stories because the main capability PerSE gains from finetuning is to analyze the reviewers’ preferences from the prior reviews and use these preferences to evaluate new cases. The story generation is a typical generation task where personalized evaluation is important. If there is no personalized training data for evaluation, we can directly prompt our personalized fine-tuned PerSE (as what we did on GPT-4 and LLaMA baselines) to evaluate. If there is a small amount of annotated personalized data in this domain, then we can further fine-tune PerSE on this domain to improve the performance.

---

**Q2. How biased is PerSE toward LLaMa generations?**

A2. Based on the description in the official repo [1], the story pairs in PerDOC are generated by a LLaMA-based fine-tuned model (oasst-30b). To investigate the potential bias in the story generation model, we randomly select one of the paired plots and ask GPT-4 to rewrite it. We randomly chose 50 examples for each aspect in Per-DOC. After rewriting one of the paired plots, we asked PerSE-7b and PerSE-13b to compare them again. We calculate the percentage that PerSE mistakenly selects the LLaMA’s output (the original plot) as the better one, and the total percentage that PerSE prefers LLaMA’s output. The results are shown below. PerSE-7b is a little biased on the LLaMA-based stories, while PerSE-13b has no obvious bias. It indicates that with finetuning our PerSE is less biased towards LLaMA-based generation.

|           | PerSE 7b |PerSE 13b |
| --------- | --------------------- | --------------------- |
| the percentage of mistakenly preferring LLaMA’s stories  | 54.74%            |   48.51%         |
| the total percentage of selecting LLaMA’s stories | 56.05%                | 49.60%                |

Reviewer comment (Experiments):
What is the rationale for expecting that movie review data of MPST is not a part of pretraining data for LLaMA? Is this likely to lead PerSE models to be biased similarly to GPT4 (albeit to a lesser extent due to personalization)?

Author response:
**Q3. What is the rationale for expecting that movie review data of MPST is not a part of pretraining data for LLaMA? Is this likely to lead PerSE models to be biased similarly to GPT4 (albeit to a lesser extent due to personalization)?**

A3. As we further discussed in previous Appendix A.1.2, LLaMA models have a similar bias as the GPT-4 (We moved this figure to the main body and it is Figure 3 now). This is why we use anonymization and summarization to alleviate the influence of memorization in our data construction. Therefore, our test set is less affected by the known bias and is a better benchmark to evaluate LLM’s capability of evaluation. Moreover, it ensures that our PerSE is tuned on a less biased training set.

Reviewer comment (Reproducibility):
Consider adding a citation/link for the MPST dataset.

Author response:
**Q4. Citation / link for the MPST dataset**

A4.We fixed this in the revision.

Reviewer comment (Writing):
I found the extended analysis of Sec 3.1 in A.1.2 to be significantly easier to understand than the writing of Sec 3.1. Please consider rewriting this section for clarity and to stand alone in the main body of the paper.

Author response:
**Q5. Rewrite Section 3**

A5. We have rewritten this section in our revision and see the general response for more details.

Reviewer comment (Evaluation):
Relatedly, model based evaluations have been found to contain self-biases (https://arxiv.org/abs/2212.10020) - consider discussing or experimentally demonstrating the extent to which PerSE is likely to prefer generations from a LLaMA based text generation model.

Author response:
**Q6. Discuss or experimentally demonstrate the extent to which PerSE is likely to prefer generations from a LLaMA-based text generation model.**

A6. See A2 above.

Reviewer comment (Experiments):
Sec 5.1: How are the k examples from a reviewer's historical reviews used for training and evaluation selected? Is this a random set of k examples for the reviewer or is something else done?

Author response:
**Q7. Sec 5.1: How are the k examples from a reviewer's historical reviews used for training and evaluation selected? Is this a random set of k examples for the reviewer or is something else done?**

A7. As described in Section 3, we randomly sample k reviews. It is used to simulate the inference time where the new reviewers randomly provide several reviews as their preference.

Reviewer comment (Experiments):
Do I understand correctly that the DOC dataset contains generations from an existing system? If so, please discuss the implications of training PerSE on generated stories and its ability to judge generations from other systems. In a future use of PerSE do you envision that researchers will train a personalized evaluation model on the ratings from a small set of annotators?

Author response:
**Q8. Discuss the implications of training PerSE on generated stories and its ability to judge generations from other systems.**  

A8. Here we would like to highlight that PerSE can evaluate stories generated by any system and the tuning phase will not introduce extra bias to the system it was trained on. There are several reasons:

1. As we show in A2, although PerSE is tuned on the stories generated by the LLaMA-based model, it does not show obvious preference over the LLaMA’s generations. 
2. In our training data, all stories are generated by the same system (DOC framework with oasst-30b). Some generated stories are preferred and some are rejected. It is less likely for PerSE to learn a bias towards this system (for example, always give a better score on this system).
3. Even if we have multiple systems in our training data, our PerSE will not learn the bias either. One reason is that during the finetuning there is no system label to indicate where the stories are from, so PerSE cannot get such information and learn the bias. The other reason is that in the personalized setting, for the same premise with the same story pair, the preferred one could change due to the different reviewers’ preferences. So it is less likely for PerSE to learn a clear bias of always preferring one system.
4. Training on human ratings of the generated system outcome is a commonly used method in learned evaluation metrics and has shown great success and generalization ability in many metrics, such as BLEURT[2] and COMET[3].

Reviewer comment (Evaluation):
Continuing from my previous question - it seems like the paper primarily proposes a method for rating prediction and posits that this may be used for personalized story evaluation. Please discuss how you envision this actually being used for text generation evaluation in much more depth. For example, what happens when there is no training data for training a model like PerSE in a specific domain? Why does it make sense to use a model pretrained on one set of users to evaluate text generated for a (hypothetical) other set of users? - it seems the most sensible for the text generation model to be trying to personalize text to the specific users on who the evaluation model is trained.

Author response:
**Q9. In a future use of PerSE do you envision that researchers will train a personalized evaluation model on the ratings from a small set of annotators?**

A9. As we clarify in General Response,  the small set of annotators and their preferences are organized as the instructions for instruction tuning [4]. The success of instruction-tuned LLMs (such as RLHF in GPT-4) shows that with a little data, LLMs can improve the capability of following the instructions. So in future use, researchers can use the training strategy of PerSE in other domains and improve the capability of evaluating from a personalized view.

---

**Q10. Why does it make sense to use a model pre-trained on one set of users to evaluate text generated for a (hypothetical) other set of users?**

A10. Two settings (predicting for existing reviewers / predicting for unseen reviewers) have their own use cases. For the first setting, the model already remembers the preferences of these reviewers and directly predicts for them. Under the second setting, the model can be easily adapted to new reviewers without further training. Our PerSE focuses on the second setting. It does not try to remember the preferences of these annotators. Instead, it predicts the preferences of new reviewers that are unseen during training. **In our test set, all reviewers are different from those in the training set.** The experimental results show that our PerSE can be used for new annotators with new preferences.

Reviewer comment (Novelty):
It is not clear to me that the the bias present in GPT4 for known plots immediately motivates a personalized evaluation - please consider rethinking/rewriting this motivation. For example, is there a simple bias correction one could apply to ratings produced from GPT4 so that known plots receive smaller scores?

Author response:
**Q11. The bias present in GPT4 for known plots immediately motivates a personalized evaluation - please consider rethinking/rewriting this motivation. For example, is there a simple bias correction one could apply to ratings produced from GPT4 so that known plots receive smaller scores?**

A11. To create a dataset for personalized story evaluation, we need to alleviate the influence of memorization to ensure the performance of different evaluation models is not affected by the known bias. We have revised this section to make it clearer and we refer you to the general response and the revision manuscript for detailed information. We propose two techniques: anonymization and summarization to reduce the known bias and use them during our dataset construction. Figure 3 shows the effectiveness of our strategies.