Question Answering in Climate Adaptation for Agriculture: Model Development and Evaluation with Expert Feedback (2025)

Abstract: The generative capabilities of the large language models (LLMs) are deployed for domain-specific question answering systems. However, their ability to answer climate adaptation questions remains unclear. In particular, can they be used by agronomists and climate scientists to answer questions on the best climate adaptation strategies? Answering questions in this domain requires knowledge of climate data and its uncertainties, and the ability to link them to the broader climate literature while accommodating the unique constraints of users and experts. We investigate the generative and evaluative capabilities of several state-of-the-art LLMs, open-source and proprietary, on climate adaptation for agriculture questions posed by domain experts using evaluation criteria designed by the experts.We propose an iterative exploration framework that enables LLMs to dynamically aggregate information from heterogeneous sources, such as text from climate literature and structured tabular climate data from climate model projections and historical observations. Our experiments demonstrate that LLMs can aggregate heterogeneous data to (1) answer questions, but at a trade-off between presentation quality and epistemological accuracy; and, (2) evaluate answers, but are not as competent at identifying high-quality answers and erroneous information compared to domain experts.

## introduction domain-specific question answering (qa) has been studied for decades @xcite . with the increase in generative capabilities of large language models, generative qa has advanced considerably in various domains such as the medical @xcite , finance @xcite , education @xcite and climate mitigation @xcite . yet a crucial domain left under-explored is climate adaptationthe process of adjusting to current and projected climate change impacts to safeguard people and ecosystems @xcite . climate adaptation, however, poses unique challenges when developing question answering systems. it requires integrating three different categories of information to help shape decisionmaking and adaptation strategies: (1) climate projections @xcite ; (2) scientific literature @xcite and climate adaptation reports @xcite ; and, (3) location-specific historical data @xcite . for instance, consider the question "how do i ensure wheat production remains profitable in southeast australia in 2050?" answering this requires not only expert knowledge of the relevant climate factors (e.g., rainfall variability and average temperature) but also contemporary knowledge of viable adaptation strategies specific to wheat production drawn from specialised literature. to advance the climate adaptation qa, we pro-pose a novel dual-purpose framework that enables exploring and aggregating large quantities of climate information from various sources, allowing us to probe the generative and evaluative capabilities of current llms. this framework can be adapted to other domains that require domain expertise and different information types and modalities. our contributions are: ## methodology domain-specific qa often requires specialised knowledge and climate adaptation qa is no exception. climate adaptation qa requires integrating heterogeneous information sources-including large-scale tabular climate projections, nationspecific historical data, and diverse climate literature such as scientific literature, adaptation reports, and books-to help shape decision-making and adaptation strategies @xcite . we propose a two-step approach to measure the capabilities of llms in climate adaptation qa by: (1) constructing the datasets and tools to access heterogeneous climate resources; and, (2) creating a framework for probing the climate adaptation qa and automatic evaluation capabilities of llms. climate literature # documents # chunks climate science 2.36m 27.1m environmental science 144k 163k regional literature 28 1293 ## experimental setup embeddings to encode document and query embeddings, we used stella 1.5b v5 foot_1 , which was observed to perform best on a validation set of human judgments of query-document pairs (more details in appendix a.3). ## results and analysis rq1: how well-presented and epistemologically accurate are llm-generated answers? our results indicate that the experimented llms were better at presentational aspects (context, structure, language) than epistemological ones (citations, specificity, scientific accuracy) ( rq2: how does proficiency in using climate data and literature affect generated answer quality? we find that different models have varying capabilities in utilising external resources ( sonnet 3.5 was the only model that saw overall significant (@xmath0.05) performance gains, as it used both cd and cl with high correctness (0.96). we observe that cl use was higher than other models, leading to improvements in the context and comprehensiveness criteria because relevant domain knowledge could be incorporated into the response. additionally, correct cd use led to significant improvements in accuracy and specificity. all models had significantly (@xmath1.05; sign test) lowered structure scores when using gef. from expert feedback, we find that this primarily results from (1) the difficulty of representing climate data textually, which often makes responses challenging to read and follow (see appendix figure 7 ); and, (2) the position of summaries varied within the response, even from the same model, and often they were after the references or within the references. rq3: what are common errors that llms make in climate adaptation qa? from expert feedback (figure 2 ), we find that the baseline methods tend to (1) be too generic to be useful; for example, sonnet 3.5 stated the temperature would be warmer for a region but did not say by how much or how it would affect specific commodities; ## conclusions we systematically analyse the climate adaptation question answering capabilities of llms and find that their baseline answer generation capabilities fall below the expectations of climate experts. although grounding llms with external resources, such as climate data and climate literature, improves epistemological accuracy, it degrades answer presentation and introduces new sources of errors, such as intrinsic hallucination and misinterpreting data. additionally, our study shows that llms cannot yet serve as reasonable evaluators of climate adaptation responses, achieving moderate agreement with experts. even when equipped with external data, llms lacked the ability to properly validate erroneous answers and identify higherquality examples while being too lenient in scoring compared to experts. overall, our work measures the capabilities of current llms in answering climate adaptation questions and the challenges of working with external resources in domain-specific question answering. it also provides a foundation for the field of climate adaptation question answering, which could be expanded to other similar scientific areas. limitations our study is not without limitations, such as (1) the use of single-turn/single-response question answering; (2) the evaluation of answers and tool parameters and not the climate data/scientific literature; (3) having two or fewer annotations per sample rather than having three experts deciding on majority labels; (4) a limited number of questions and answers; (5) the use of three llms for generating and evaluation; (6) study design; (7) lack of more advanced prompting strategies; ( 8 ) unable to measure the language evaluation capability of llms; and, (9) the metrics were not weighted. while we could not explore these limitations further in this paper, we highlight these as a potential starting point for researchers who want to explore this domain further and also provide the rationale below. a major limiting factor was that the climate experts were volunteers, so asking for more of their time remained impractical. further, many volunteers withdrew from the study due to the lack of time, lowering the annotation workload that could be reasonably given to the remaining experts. therefore, a multi-turn qa setup or climate data/scientific literature evaluation would exceed this workload (limitations 1 and 2). the limited annotation budget also lowered the number of questions that could be made, as these require careful consideration from experts and also the number of answers that could be evaluated for each question (limitations 3 and 4). although stronger llms existed, we opted to use three llms (limitation 5) for generation and evaluation for the following reasons: (1) many of the stronger llms were not capable of function calling and structured generation (e.g., openai o1); (2) the cost of running these models would be prohibitively expensive in terms of monetary costs for proprietary models and computational costs for open-source models, lowering the number of tested models; and, (3) hardware limitations, we did not have resources to run very large open-source models (e.g., mistral-large 123b and llama3 405b). the study design introduced distrust from some climate scientists as the responses became highly specific. the study was designed to be singleblind; participants were not told which method or model they annotated. however, scepticism increased with gef-based approaches because annotators could not see the data that models used to generate the answer (limitation 6). although the overall findings seem unaffected, as experts could still determine low-quality and high-quality responses, this could lower expert agreement. applying more advanced prompting techniques (limitation 7) would likely improve the quality of the responses and lower scepticism at the risk of making it easy to distinguish our system from the baselines. we find that automatic llm evaluators predicted perfect or close to perfect scores for the language criterion in responses, irrespective of method (limitation 8). this limited the ability of our agreement metrics to capture the llm's evaluation performance (the responses could not be ranked as they were all tied) to the language criterion, so we chose to exclude these from our automatic evaluation analysis. we did try to rectify this in appendix 5, but to no avail. we hypothesise that this problem is likely a result of the annotation criteria for language being simplistic, as baseline results for language are almost a perfect score for most methods. although this is a negative result, we highlight it as an interesting insight for researchers working in automatic llm evaluation. finally, we did not investigate the importance weightings of the metrics and instead opted for looking at trends based on the category of metric (presentation or epistemological) or individual metrics (limitation 9). this is because the importance of the metrics is dependent on the user. for example, a scientist may value presentation less than a layperson. this importance weighting and the inclusion of an uncertainty metric @xcite , which could be an important dimension, given the overconfidence of llms providing responses, may serve as a foundation for future work. instructions this activity has been designed to gather feedback on each of the responses provided by 3 llms (we refer to all as systems). this allows us to understand how the responses/outputs compare to other systems. it should be noted that all systems selected for testing will be anonymised to avoid bias. the system's response is comprehensive and does not just give a partial, incomplete answer • rating: yes/no 6.b. shows depth of knowledge/understanding regarding topic • rating: yes/no 6.c. answers beyond the question's scope to provide context • rating: yes/no 7. scientific accuracy 7.a. is the information scientifically robust? answer to the best of your knowledge • rating: yes/no/n/a • explanation: 7.b. does the response meet scientific expectations? (consider own knowledge or through supported literature) • rating: yes/no/n/a 7.c. does the response have any errors? answer to the best of your knowledge • rating: yes/no/n/a a.5 questions we provide the question set that was used for climate science qa generation below: 1. what are the ideal pollination conditions for growing almonds? 2. what can i do to prevent sunburn risk in apples? 3. what varieties of apples are more tolerant to sunburn? 4. what regions will support growing cotton in 2070? 5. how does the climate in south west western australia compare from 1970 to now? 6. what will be the greatest climate risk for growing wheat in the wheatbelt in 2050? 7. will my rainfall continue to increase in variability in northern nsw? 8. in north-east sa, how many days will i likely experience over 45 degrees? 9. how accurate are climate projections? 10. what is the difference between a heatwave and a hot day? 11. will we likely see less cold risk days over the lambing season in central tasmania? 12. how will climate change impact cherry production in young? 13. what is the production cycle of potatoes? 14. are there regions in australia where agriculture will not be viable in 2050? 15. will commodity distribution in australia change under a future climate? 16. what are the 6 main climatic drivers of chickpea growth in australia? 17. what happens to chickpea crops when there is a deficit of soil moisture at the time of sowing? 18. what is the likelihood of the incidence of extreme heat increasing in the future during the reproductive phase of my chickpea crops (grown in biloela in central qld)? 19. what is the extreme heat threshold of kabuli chickpeas and does it differ from desi? 20. what is the extreme heat threshold of kabuli chickpeas and does it differ from desi? which type of chickpea is more heat-resistant? 21. what is the critical temperature below which chickpea crops are non-viable? 22. where in australia might chickpeas become viable in 2040 where they are not viable now? 23. how cool does the temperature have to be, and for how long, to induce flowering in avocados? are these periods of cool weather going to become rarer in the northern rivers region by 2030? 24. how cool does the temperature have to be, and for how long, to induce flowering in avocados? are these periods of cool weather going to become rarer in the northern rivers region by 2040? 25. what is the most heat resistent cultivar of avocado available? 26. what are the main climate change risk factors for chickpeas in australia? 27. what are the main climate change risk factors for mangos in australia? 28. what are the main climate change risk factors for apples in australia? 29. what temperature is considered the threshold for increased risk of sunburn in apples? 30. what are the main ways climate change can impact farmers trying to grow avocados? 31. when growing avocados, what temperature is the threshold for a 'cool day' during flower induction? 32. what are the main ways climate change will impact farmers trying to grow field tomatoes? 33. what are the main ways climate change will impact farmers trying to grow chickpeas? 34. what are the main ways climate change will impact farmers trying to grow cherries? 35. what is the threshold for a hot day during the flowering season when growing avocados? 36. in the context of growing avocados in shepparton vic, what is meant by a pollination event? 37. are there existing australian agricultural regions that could become unviable in the future? 38. what is the date range within which flowering occurs in the northern avocado growing region? what are the main ways climate change can impact farmers trying to grow avocados?