Paper: Eliciting Human Preferences with Language Models (ICLR-2024)

Reviewer comment (Experiments):
While the paper introduces three different domains, two of the three baselines were not available for two of the three tasks. Since GATE shows promising results for content recommendations, it is hard to assess the validity and generalizability of the results without similar comparisons to baseline elicitation approaches for the other two tasks.

Author response:
**Baselines**

Thanks for raising this point, as we could clarify better in the main text. The two primary baselines we consider are 1) No interaction (this is 0 on our graphs) and 2) Detailed prompts, where we have the participant write out their preferences. For the content preferences domain only, we also have access to a dataset of example news articles. We can use this to present two additional baselines: supervised learning and active learning. Unfortunately, in many domains, we do not have access to a large dataset of examples from the target domain; for example, there is no “natural distribution” over moral dilemmas or invalid email addresses. Thus, our experiments attempt to capture the performance of GATE in both settings. 

In fact we see that as an advantage of our method: lots of domains do not have large, pre-existing corpuses of data which allow for supervised learning and/or active learning. Our method opens the possibility of learning preferences in those domains without pre-collecting a labeled dataset.

Reviewer comment (Experiments):
It seems like elicitation using generative open-ended questions is the best of the three generative variants studied in this work (according to Figure 2), however, it also seems to be the most mentally demanding for the users (according to Figure 3) and seemed to be comparable to baselines. It would be nice to see a proposed modification to that approach that would maintain the *same* score in Figure 2 but would decrease the reported mental load, particularly given that the generative approaches that were tried in this work were all relatively simple (as stated by the authors themselves).

Author response:
**Mediating generative open-ended questions**

Broadly speaking, there is a tradeoff between interaction time/effort and the degree to which an approach can elicit human preferences. As the reviewer has noted, open-ended questions tend to be on one end of the tradeoff, requiring slightly higher mental demand (although equivalent mental demand to 2 of the 3 baselines), but being the best performing out of the three. Yes-or-no questions, while worse than open-ended questions at eliciting preferences, still beats out all baseline approaches, while being much equivalent or less mentally demanding to all three baselines. We agree that it would be great if there were some approach that could push the Pareto frontier, which we leave to future work.

Reviewer comment (Presentation):
How is the framework in Section 2 connected to Section 3? The two sections felt a bit disconnected.

Author response:
**How is the framework in Section 2 connected to Section 3?**

In short, the framework of Section 2 motivates a gap in the literature which we fill in Section 3. 

Section 2 describes how existing learning paradigms can be described as task elicitation methods along two axes. First, methods can elicit information passively (as in supervised learning) or actively (as in active learning). Next, they can elicit information with examples (again as in traditional supervised learning) or by rich free-form inputs (such as prompting). However, no present method actively elicits free-form inputs. Section 3 describes our proposal of GATE, a framework for actively eliciting free-form task specifications, and we describe how we operationalize this framework for investigation in our paper. Section 4 then describes the concrete experiments we do given this operationalization.

Reviewer comment (Presentation):
In Figure 2, what does the * signify?

Author response:
**In Figure 2, what does the * signify?**

The asterisk signifies statistical significance at the 0.05 level. We have added this information to the caption in our newest revision.

Reviewer comment (Experiments):
What do 6/10 and 7/10 settings refer to?

Author response:
**What do 6/10 and 7/10 settings refer to?**

Great catch – There were actually only 9 settings and GATE improves in 7 of them. Specifically, we examined the following 9 (domain, elicitation method) pairs from Figure 2 and counted in how many settings that GATE improves over user-written prompts, ignoring significance:
(Content domain, generative open-ended questions): Yes
(Content domain, generative yes-or-no questions): Yes
(Content domain, generative active learning): No
(Moral domain, generative open-ended questions): No
(Moral domain, generative yes-or-no questions): Yes
(Moral domain, generative active learning): Yes
(Email domain, generative open-ended questions): Yes
(Email domain, generative yes-or-no questions): Yes
(Email domain, generative active learning): Yes

6/10 referred to a metric we had in a previous version of the paper, apologies for the confusion. We have taken this out in our newest revision and fixed the denominator to 7/9.

Reviewer comment (Writing):
Does the term “time limit” refer to the 5-minute interaction period?

Author response:
**Does the term “time limit” refer to the 5-minute interaction period?**

Yes, we have edited this in our newest revision to clarify.

Reviewer comment (Novelty):
It is unclear where other known preference elicitation approaches such as pairwise comparisons / choice-based preference elicitation / ... fit into the stated related work. They might be situated in example-based/interactive, but especially there are other studied interaction mechanisms in this field which might have overlaps to free-form. It should be part of the related work coverage.

Author response:
**Other preference elicitation formats**

This is a great point! This is indeed relevant literature which we will discuss in related work. We believe that GATE is complementary to pairwise comparisons / choice-based elicitation, and we can definitely do them within the GATE framework (i.e. prompt the LM to generate pairwise queries, choices to select among), which would be a good topic for future work. We don't mean to claim that this paper has exhaustively characterized the space of possible GATE policies.

Reviewer comment (Experiments):
Part of the investigated methods (generative active learning and generating yes-or-no questions) might be too rigid, as interests/preferences might not be black and white. I understand that the user can give any response to any of the methods, thereby giving more refined answers than yes/no, but handling arbitrary relative answers might become difficult to handle. The paper does not analyse the availability of such a problem / rule it out based on the results of the conducted user study. Here, ablations of possible user answers or prompt changes might be insightful.

Author response:
**Some answer types are less flexible**

We agree—generative active learning and generative yes/no questions both assume a binary set of responses.  This might be limiting in real-world settings where the true answer might be “it depends,” or something more subtle. Interestingly, however, we do see some evidence that participants use the open-endedness of the chat interface to provide more details in these settings, rather than always replying with a single “yes” or “no.” And in general we believe that the advantage of our work is that it opens up the possibility for open-ended dialogue and answers that can more fully capture the space of nebulous human preferences, as we explore in the open-ended response settings we study.

Reviewer comment (Evaluation):
Did you test implications of answer diversity on the success of your methods? Taking yes/no questions, a user might still answer with any fuzzy statement which might be quite ambiguous. It would be interesting to know if limiting the answers to potentially some selected "in-between" categories would help or not. Along this, did you analyse how often a user answered with such an ambiguous statement?

Author response:
**Answer diversity vs. success of GATE**

This is a great question! We did a preliminary investigation of whether we could correlating the **% of answers that aren’t “yes” or “no”** in a transcript (for the “yes-or-no questions” elicitation setting) vs. the **delta p(correct)** after the transcript. Unfortunately, we could not find a strong correlation in our preliminary investigation (p-values $0.4$, $0.5$, $0.9$ respectively for the email, moral, and content domains). Note that writing an explanation takes additional time, so transcripts with more explanations tend to also have less turns overall. This preliminary result suggests that directly answering the questions asked by GATE may be more useful than whatever explanation a user may decide to provide. However, more fine-grained analyses may be necessary to break down the nature of the additional explanation / fuzzy answer, and the direct contribution of that answer to the accuracy.

Reviewer comment (Experiments):
Would it be possible to combine the proposed prompts to improve performance / reduce mental load? Did you experiment with such prompts? The open-ended version could, of course, do so by design - did you conduct analyses if this happens?

Author response:
**Combine proposed prompts to improve performance & reduce mental load**

This is a good question. We believe that combining question types would be an interesting direction for future work, and could be better able to navigate tradeoffs between performance vs. mental load. We will add this discussion to future versions of our paper. We also found that the open-ended case does indeed sometimes ask yes-or-no questions.

Reviewer comment (Theory):
the concept of morality is underdefined. What is the notion of morality employed here? There is quite some literature on the moral foundations theory and their detection from text which should be used as a benchmark for the model.

Author response:
**Morality**

We completely agree that there is no single objective framework for moral reasoning. The point of these experiments in our paper was to see whether GATE could enable a language model to elicit information about an individual’s subjective, personal moral preferences. Interestingly, we do find that many of the questions asked by the language model align with the axes explored by moral foundations theory (e.g. exploring the tension between care and fairness), and have added discussion and citations to that effect in the newest revision.

Reviewer comment (Experiments):
the testing is well-thought but quite limited. Such models require an extensive testing to ensure they are not overfitting, especially when for systems like this one which may eventually foster polarization as an effect of extreme self-centred recommendations.

Author response:
**Overfitting and polarization**

We agree it is crucial to prevent overfitting to people's preferences. Our work makes efforts to guard against two potential sources of overfitting. First, we guard against statistical overfitting of our classifiers by ensuring that models are always evaluated on how well they can predict preferences on held-out data from another distribution. The strong performance of our models in this setting indicates they are not overfitting to the training data. Second, we check that the model does not cause the humans to “overfit” by strengthening or modifying their own preferences (e.g. via echo chamber effects). Notably, Figure 3, right, demonstrates that people's preferences do not change after using GATE. That said, we agree that longer-term use of personalized recommender systems warrants caution and further study, and have included discussion of this in our ethical considerations section.

Reviewer comment (Experiments):
the concept of morality is underdefined. What is the notion of morality employed here? There is quite some literature on the moral foundations theory and their detection from text which should be used as a benchmark for the model.

Author response:
**Morality**

We completely agree that there is no single objective framework for moral reasoning. The point of these experiments in our paper was to see whether GATE could enable a language model to elicit information about an individual’s subjective, personal moral preferences. Interestingly, we do find that many of the questions asked by the language model align with the axes explored by moral foundations theory (e.g. exploring the tension between care and fairness), and have added discussion and citations to that effect in the newest revision.