Paper: Assessing Large Language Models on Climate Information (ICLR-2024)

Reviewer comment (Experiments):
The LLM prompt does not state the evaluation criteria on presentation and epistemology. If all the eight metrics were included in the prompt, it is likely the response quality will improve (my hypothesis). With the current prompt, the LLM is "unaware" that it is being evaluated this way. An ablation with and without such a prompt/instruction will also be interesting to see.

Author response:
1 - Thanks for the suggestion, we are trying to run an experiment where we augment the prompt with a description on the evaluation criteria. We hope to be able to update the paper with these results as soon as possible. Notice, however, that this is a somewhat of a theoretical factor. A generalist model (the object of this study) may be evaluated in many different ways on many different dimensions and topics and cannot be expected to be able to condition at answer generation time on the evaluation dimensions. In fact, some of the dimensions may have conflicting goals (say ‘creative tasks’ vs ‘scientific information’). At the same time, it is not realistic to expect that users will do that kind of ‘prompt tuning’ while asking questions.

Reviewer comment (Evaluation):
The LLM is not evaluated against a human expert output. It would be good to see an expert answer the same question, and evaluate them on the eight metrics. Currently, the 1-5 score grid is an arbitrary bar that the LLM is being held to without comparing with existing systems.

Author response:
2 - In preliminary experiments we found that evaluating actual experts' outputs is quite complex. This will be one of the main things we will focus on in the next steps. However, we are trying to see if we could get at least some evidence in that direction during the rebuttal period.

Reviewer comment (Experiments):
Claims in the sub-section "Resolution and Range" do not have sufficient evidence.

Author response:
4 - We wonder why the reviewer thinks this is not the case. We would welcome more details. In terms of range we, as a first approximation, consider the difference between the best model score and the worst model score. This turns out to be several times the magnitude of the standard deviation, and we interpret this as providing evidence for sufficient range. In terms of resolution, our results show that the model's performance, e.g. for the GPT models, is sufficiently aligned with expectations: newer models are better than older models, with the oldest (InstructGPT-002) performing the worst.

Reviewer comment (Writing):
Concerns about audience context is raised multiple times, but is not addressed in the paper

Author response:
5 - We are not exactly sure what the reviewer is asking about. For our evaluation, we assume that all 'audience context' is part of the question, i.e. we don't ask our raters to assume a specific context. However we do assess whether the answer is sufficiently specific to the question, e.g. if the question is about sea level rise around Iceland in the year 2100, the answer should address the relevant geographic region and time.

Reviewer comment (Evaluation):
Why did you use a 3-4 sentence restriction? Instead, it would have been better to ask the evaluators if the response was too long.

Author response:
1 - We add the 3–4 sentence length specification to provide a minimum of control on the maximum length of the answer, thus reducing the variance in answer form. This also helps to reduce the variance in raters' performance because longer answers may be more difficult to rate.

Reviewer comment (Reproducibility):
Is GPT-4 used for all of the AI assistance generated, or do you use respective LLMs to create the assistance paragraph?

Author response:
2 - Only GPT-4 is used to produce the auxiliary information, to avoid confounding factors due to the differing abilities of a specific model in performing them.

Reviewer comment (Experiments):
Do all the LLMs support the input length to supply the paragraphs in the Wikipedia articles as input? It may be that paragraphs in the lower sections get ignored given the input text limit.

Author response:
3 - We limit the Wikipedia article evidence to the top 3 passages, so capacity is not an issue. However, LLMs don’t need the Wikipedia articles to answer the question. The Wikipedia article is only used to generate the model assistance for the epistemological dimensions and we always use GPT-4 to generate the assistance, see above.

Reviewer comment (Experiments):
What happens if answers to the questions are not available in Wikipedia? Are there such questions in your dataset?

Author response:
4 - If there is no Wikipedia document relevant for the answer (which happens in about 10% of the answers) we simply do not generate AI assistance for the epistemological dimensions and such answers are not used for the AIS evaluation study. In any case, the raters don’t need the Wikipedia article for the presentational and epistemological rating task.

Reviewer comment (Novelty):
Do you include questions to which do not have answers to from the latest science?

Author response:
5 - Novel questions. We don’t have a dedicated set of this kind and we are not aware of questions of this kind in the current dataset. However, we believe this is a good suggestion and we will try to address this point in the next version of the data. This requires careful thought and validation.

Reviewer comment (Writing):
I didn't understand the concept of "previous exposure". Don't I have exposure as soon as I evaluate one question? From that respect, all but the first question have "previous exposure".

Author response:
6 - Previous exposure. Previous exposure means that a rater has seen and rated some answers where AI assistance was shown. The three categories work as follows:

1. *‘With AI Assistance’:* Answers in the evaluated set are accompanied by AI assistance and the raters are shown the assistance.
2. *‘Without AI Assistance but previous exposure’:* None of the answers in an evaluated set are associated with AI assistance. The raters assess the answers without being provided any AI assistance. However, in previous studies, the same raters have evaluated answers accompanied by AI assistance. What our results show is that this (previous exposure) seems to have a training effect on the raters and they are able to spot more issues compared to the ‘With AI Assistance’ setting.
3. *‘Without AI Assistance’:* A fresh set of raters assesses the evaluated set, for the first time, without ever having rated any answers that were provided with AI assistance. In this setting, (Figure 3a) raters recognize the least issues.

We will make this clearer in the paper.

Reviewer comment (Experiments):
A key weakness of the work is the lack of any prompt tuning. The prompt provided is a 2-sentence prompt that does not include any of the known techniques that improve LLMs' performance.

Author response:
We thank the reviewer for their comments and suggestions. Here we provide some initial answers. In the remaining rebuttal period, we will be trying to run some of the experiments suggested. Our goal is to integrate the results and discussion in an updated version of the paper as soon as possible.

To the specific point: We agree that prompt tuning, especially specific to each model, is likely to improve results. In this work our priority is mainly around defining the evaluation framework and protocol. For the first results we did not include prompt tuning in the experiments and instead compared several models using an identical prompt. Note that this is likely the most common way for people to get answers from LLMs. Prompt engineering, learning, reasoning etc. are in themselves important and current research topics to investigate and we are planning to do that in near-future work. In particular, we believe that it will be crucial, as we collect enough data, to evaluate learning methods across the full spectrum available from in-context learning, to supervised finetuning, RLHF etc. Again, this is beyond the scope of the current paper.

One unfortunate aspect of tuning individual prompts per LLM is that it is not an exact science and one could always claim that with a slightly different prompt, the output might be even better. To gauge the influence of better prompts we still argue that having a baseline number with a vanilla prompt is important for comparison.

Reviewer comment (Experiments):
In-context examples that can be random or retrieved based on query similarity should be included. This will help both grounding and tone—a factor that most LLMs were weak at. The likely cause is that the LLM does not understand the expected tone, and providing examples may make LLM outputs more similar to the desired tone.

Author response:
We are grateful to the reviewer for the idea of evaluating a prompt that summarizes the criteria (presentational and epistemological) used to evaluate the answers, possibly with few-shot examples. We believe they would provide a natural extension of the base case. We are trying to run an experiment in that direction. If we can have results in time, we will include them in an updated version of the pdf within the rebuttal period.

Reviewer comment (Writing):
At a minimum, the presentational and epistemic criteria can also be included in the prompt. For example, if there is a need for a certain style (e.g., text should not be informal), that can be explicitly included in the prompt.

Author response:
We are grateful to the reviewer for the idea of evaluating a prompt that summarizes the criteria (presentational and epistemological) used to evaluate the answers, possibly with few-shot examples. We believe they would provide a natural extension of the base case. We are trying to run an experiment in that direction. If we can have results in time, we will include them in an updated version of the pdf within the rebuttal period.

We sincerely appreciate your time and effort in reviewing our paper, and have now updated it.

To address your questions, we ran a preliminary experiment that is adding a description of the criteria to the prompt for GPT-4. The results are included in a new section of the Appendix, A.1.

Please let us know if you have any further questions.

Reviewer comment (Experiments):
Without basic prompt enhancements like these, the results may not be an accurate assessment of the models. For instance, the results on tone are likely to increase significantly after these changes.

Author response:
To the specific point: We agree that prompt tuning, especially specific to each model, is likely to improve results. In this work our priority is mainly around defining the evaluation framework and protocol. For the first results we did not include prompt tuning in the experiments and instead compared several models using an identical prompt. Note that this is likely the most common way for people to get answers from LLMs. Prompt engineering, learning, reasoning etc. are in themselves important and current research topics to investigate and we are planning to do that in near-future work. In particular, we believe that it will be crucial, as we collect enough data, to evaluate learning methods across the full spectrum available from in-context learning, to supervised finetuning, RLHF etc. Again, this is beyond the scope of the current paper.

One unfortunate aspect of tuning individual prompts per LLM is that it is not an exact science and one could always claim that with a slightly different prompt, the output might be even better. To gauge the influence of better prompts we still argue that having a baseline number with a vanilla prompt is important for comparison.

We are grateful to the reviewer for the idea of evaluating a prompt that summarizes the criteria (presentational and epistemological) used to evaluate the answers, possibly with few-shot examples. We believe they would provide a natural extension of the base case. We are trying to run an experiment in that direction. If we can have results in time, we will include them in an updated version of the pdf within the rebuttal period.

Reviewer comment (Experiments):
Lack of discussion of rationale for design of LLM answer structure: The authors present the protocol for getting answers, keypoints, evidence, and AI assistance from the LLM - but the rationale for using this protocol is not discussed. Is the goal to get as good a model as possible, a model that is representative of what's out there, to test out raw output vs. summarization vs. self-correction capabilities, etc.? Discussion of these design choices is critical to properly contextualize the evaluation of the outputs.

Author response:
**2- Design rationale**

The design of the LLM task was chosen so that it is a good match for the main use case we are addressing: The general public engaging with LLMs to satisfy climate change related information needs. We will highlight the assumed audience more explicitly. We’re trying to keep the setup as simple as possible and realistic at the same time. One reason for adding a soft limit to the answer length is to avoid (for the time being) adding confounding factors related to answer structure, length, and complexity. This helps to make answers comparable between models, which otherwise can produce much longer replies which could bias raters. 
We also didn’t want to make use of complex or model-specific prompts, e.g., reasoning-based, because they may be interpreted and executed in different ways by different models. The study of the effect of prompt engineering and reasoning is a very important topic, but would introduce too much complexity. We do however plan to investigate this extensively in future work. We believe our somewhat straightforward prompt is agnostic to the specific model and doesn't add relevant bias.
We found that keypoints and url generation provide a nimble first solution to adding evidence that can be used to 1) inform AI assistance, and 2) provide support for running the AIS evaluation (which needs references). Note that these steps are executed after the answer is generated, and always by the same model (GPT-4), so they don’t affect the answer quality, which is meant to represent the LLM’s ‘vanilla’ answer mode.
Overall, we iterated over multiple base versions to get to a simple and robust initial ‘baseline’ mode. We expect that models will perform better in more sophisticated modes, but it would introduce more complexity than appropriate at this stage. Overall, the high-level design choice is that model and evaluation sophistication must progress in tandem.

Reviewer comment (Evaluation):
Implicit weighting of criteria: Throughout the discussion of evaluative dimensions and their presentation in the numerical study, there is an implicit assumption that all criteria are weighted equally. However, different criteria can and often do have different weights - for instance, with respect to epistemological adequacy, if accuracy is not met, high performance on specificity/completeness/uncertainty is arguably irrelevant. This is not addressed in the submission at all, and scores are often averaged for the purposes of visualization and analysis. Similarly, error bars are presented to give a sense of uncertainty, but extremal measures (worst-case and best-case outcomes) are not reported - however, in settings where the answers may be "safety-critical" (e.g., the example of telling a farmer what practices to implement, where an incorrect answer has the potential to damage a farmer's entire crop and livelihood), worst-case outcomes in particular are key to have a handle on.

Author response:
**3- Weighting of Criteria**

We agree that the combination of scores is a critical problem for making decisions - which we don't. Averages in Fig. 1 are mainly given to illustrate the point that LLMs overall do much better in the presentational dimensions. For most of our analysis we report the individual dimensions. We agree that deciding which LLM is 'best' for the task might require some heuristic aggregation of the different dimensions. We deliberately do not propose an ad-hoc heuristic.

Reviewer comment (Experiments):
Choice of raters: The cohort of 32 raters chosen for the study is described as consisting of individuals with "at least an undergraduate degree in a climate-related field of study. This includes environmental disciplines (e.g. environmental science, earth science, atmospheric physics, ecology, environmental policy, climate economics), and also other disciplines (including the behavioral and social sciences)." There is some mismatch between the kinds of disciplines listed and the disciplines of the questions (see Figure 5) - topics like cities-settlements-infra, energy, and health-nutrition are covered in relatively different disciplines than those listed. Answers can be inaccurate in subtle but pernicious ways, which requires targeted expertise to catch - it is not clear that raters were necessarily matched to questions in their area of expertise, nor that they were given time to consult external literature (e.g. by performing their own Google Search or literature search) to fully evaluate the accuracy or broader epistemological quality of answers. For context, the reviewer notes that they meet the inclusion criteria for raters but would not be able to confidently answer some questions (e.g. about pest spread) without consulting external sources, since these can be wrong in subtle ways. This concern is substantiated by observations such as low inter-rater agreement on specific answers (Appendix H), and while the authors claim that system-level conclusions can still be drawn on the basis of the results, this may not be categorically true and warrants more nuanced discussion. While it would be unfair to ask for a redo of the entire study, these issues of rater expertise should be foregrounded in the main body of the paper, and be taken into account via appropriate caveats on results (including the results on the effectiveness of AI assistance).

Author response:
**4- Choice of raters.**

In our current approach we struck a compromise between availability / cost of ratings and specific domain expertise. If we require too specific domain expertise, recruitment problems will lead to issues regarding scalability and replicability. In future work, we plan to explore the furthest end of this by working with domain experts and matching them to questions based on topic, also adding a reconciliation step where the domain experts can resolve divergences. We hope it is clear that this goes far beyond what we are currently doing here.

Notice that we don’t forbid or discourage consulting external sources, we only specify that this should be done quickly to clarify specific points rather than to perform an in depth research of the topic, because this may not only make the task less scalable, but also introduce other confounding factors. However, this is an aspect of the research which we are actively reviewing and will evolve as we identify better solutions. Again, we don’t feel this choice invalidates the findings of the current work. 
At the same time, we fully agree with the reviewer that appropriate caveats and a more nuanced presentation of take homes from the results should be presented more clearly. We will address this in the revised version of the paper.

Reviewer comment (Evaluation):
Analysis of AI assistance: The effects the authors describe with respect to the helpfulness of AI assistance may actually be more of an indication of the lack of matched expertise of the raters, rather than being a clear indication of whether AI assistance is actually helpful in reality or not (especially given lack of access to ground truth in the context of this study). In addition, results indicating that AI assistance helped raters find more mistakes may also be a result indicating phenomena along the lines of "anchoring and adjustment" rather than the true presence of more mistakes in the answers (especially given that, as shown in Figure 3a, the biggest boost actually came in specificity, completeness, and uncertainty - which are more subjective - rather than accuracy).

Author response:
**5- AI Assistance.**

We agree that our results about AI Assistance only apply to the specific setting we're using it in, and can’t be generalized to a setting where domain experts are used as raters on matched questions. We do however believe that the setting we’re studying is of interest, because a group of raters with relevant background is much more accessible than a group of domain experts.

We do in fact want to explore the question whether domain experts would also benefit from AI Assistance, but it's out of scope for this paper and something we have to leave for future work.

Reviewer comment (Presentation):
Next steps and takeaways: While the goal of the paper is presumably to motivate potential next steps and/or cautions regarding the development and use of LLMs for answering questions about climate change, it is notable that discussion of critical recommendations, next steps, or hopes for usage of the presented framework are missing. Explicit discussion is necessary to concretize the "point" of this paper, and provide further context for the analysis/design choices therein.

Author response:
**6- Next steps and takeaways.**

We will add a paragraph on this.

Section 2.2
This is a good point. It ties back to the way such evaluation should be used and how the individual dimensions are combined and weighted. Consider the following example: “is planting trees good for climate change?” Maybe a baseline weight should be ‘accuracy trumps everything else’. But it is not so easy. For instance, it is probably accurate to say that planting trees can be good for climate change, because they are natural carbon capture and storage. However, research also suggests that if done badly (e.g., due to differential albedo effects in equatorial vs boreal regions) it can do more harm than good. Thus, completeness must also be taken into account. But, then, uncertainty may need to be factored in, leading to different conclusions… and so on and so forth. There are no shortcuts and easy answers, an expert would take all these dimensions into account to make a comprehensive judgment.
These are key observations and limitations of our research which we are not in a position to resolve, the reviewer is absolutely right, but we feel they go beyond the scope of the current work.
These are observations that must be articulated clearly at the beginning and at the end, whenever takeaways are presented.

Reviewer comment (Experiments):
Prompt wording: The prompt used for the LLMs specifies the LLM is an "expert on climate change communication" - notably, not, e.g., an "expert on climate change who is good at general public communication." The current choice of prompt implies a particular weighting between presentational vs. epistemological criteria, and a different prompt would imply a different rating. While it would be unfair to ask the authors to redo the entire study by rewording the prompt, the potential implications of the current prompt should at least be highlighted explicitly.

Author response:
**7- Prompt.**

We evaluated many variants of prompts and targeted the simplest most reasonable configuration. However, the effect of prompting is complex and deserves a thorough investigation that goes beyond this paper.