My Climate Advisor: An Application of NLP in Climate Adaptation for Agriculture (2024)

Abstract: Climate adaptation in the agricultural sector necessitates tools that equip farmers and farm advisors with relevant and trustworthy information to help increase their resiliency to climate change. We introduce My Climate Advisor , a question-answering (QA) prototype that synthesises information from different data sources, such as peer-reviewed scientific literature and high-quality, industry-relevant grey literature to generate answers, with references, to a given user’s question. Our prototype uses open-source generative models for data privacy and intellectual property protection, and retrieval augmented generation for answer generation, grounding and provenance. While there are standard evaluation metrics for QA systems, no existing evaluation framework suits our LLM-based QA application in the climate adaptation domain. We design an evaluation framework with seven metrics based on the requirements of the domain experts to judge the generated answers from 12 different LLM-based models. Our initial evaluations through a user study via domain experts show promising usability results.

## introduction climate change impacts are seen across the globe in many different ways, from an increase in annual mean temperatures to an increase in the frequency of natural disasters. according to the united nations framework convention on climate change @xcite , climate change adaptations are increasingly necessary to adjust and respond to the impacts of climate change. these can include technological developments @xcite , behavioral changes @xcite , early warning systems for extreme events @xcite , and improved risk management @xcite . in the agricultural sector, climate adaptation means improving farmers' ca-pacity to deal with climate change. this adaptation can include the development and use of tools to increase their knowledge of and resilience to climate change. @xcite . our study contributes to the goal of making such knowledge accessible. specifically, our contributions are two-fold: (1) to make the evolving knowledge of climate change and adaptation practices accessible, we have developed a question-answering tool called my climate advisor (mca). it is a prototype online service for farmers and farm advisors to gain easier access to information from scientific literature, grey literature and reports, as well as future climate projection data. given a farmer or farm advisor's question, it responds with information synthesised from the literature alongside references for further reading; and, (2) we propose a novel framework for evaluating such a system, with seven different evaluation criteria, which we share through an annotation guideline together with our initial experimental results. note that the domain experts carefully designed these criteria. 1 the tool will integrate with my climate view's api, allowing access to both historical and projected climate data within a 100-year window for a breadth of representative concentration pathway (rcp) emission scenarios @xcite . climate adaptation climate adaptation is described as an adjustment in a social, economic or ecological setting in response to actual or expected climate change @xcite . in agriculture, farmers may need to adjust their practices to improve resilience to variations in temperature, precipitation patterns and extreme weather events @xcite . farmers may need to implement new technologies, crop cultivars and management techniques to ensure food security or economic security in a sustainable manner @xcite . to help farmers adapt to climate change, a goal of my climate advisor is to produce regionally-and commodity-relevant, upto-date management insights from the literature. ## methods my climate advisor is currently designed as a question-answering tool foot_2 with several components and data sources. we detail our data collection method and corpora used for the retrieval augmented generation (rag) and the retrieval algorithm to search over the corpora. for generation, we detail the large language model (llm) used in the study and the decoding algorithms and hyperparameters used for answer generation. ## experiments to understand how our tool performs, we benchmark it against other existing and proprietary methods. with consultation of climate risk and adaptation experts, we created 15 questions about australian climate change impacts and adaptation (appendix ??), which we used to generate responses. these questions range from general climate change and adaptation questions to more difficult commodity and region-specific questions. ## results and analysis in the literature, we often see that proprietary generalist models perform better than open-source models @xcite . however, we found no clear distinction between proprietary and open-source models ( agreement inner-annotator agreement using kendalls's tau @xcite led to 0.319 (moderate) agreement and an overlap of 41.5%. although the annotators mutually drafted the evaluation criteria, scientific accuracy was a source of significant disagreement ( another source of disagreement was with specificity however, upon inspection, many of these disagreements were within one point and can be attributed to human error or bias. we can further back this claim by looking at the sentiment of scores. when the labels are binarised, scores higher than 2 become positive, and scores 2 or less become negative. in this binary setting, kendall's tau agreement is 0.488 (moderate), with an overlap of 76.6%, which can be interpreted as the annotator's overall sentiments of responses being closely aligned. when removing accuracy annotations from this calculation, strong agreement is reached at 0.635 with an overlap of 85.4%, highlighting that the annotator's sentiments are closely aligned. system preference both annotators preferred gpt-4 with llama-3 70b faring well also. the initial results indicated that the most scientifically accurate model is claude opus (one annotator). both annotators agreed that chatgpt (gpt-3.5 turbo) was the worst model. this is noteworthy given that it is currently the most popular public-facing chat model. when analyzing the combined raw distribution of scores (figure 3 ), we note that the highest performing question-response pair was from the llama-variants, llama 3 8b + rag and mistral 7b + rag, to questions 6 and 15 respectively from each annotator (see appendix b). these responses were not only scientifically accurate but were stylistically similar to the responses from gpt-4, where a list of dot points is given, a summary and references at the end. therefore, we find that there is potential for our tool to outperform gpt-4 once aligned with this style of response. both annotators agreed on the worst performing question-response pair, where gemini 1.0-pro responded to question 3 with a hallucinated apples do not get sunburned response. an initial hypothesis could be that the model was trained with incorrect data. however, this did not occur with gemini 1.5-pro, assumed to be trained with similar data, where the model responded with the correct strategies to prevent sunburn risk. regarding individual scores, the first annotator ( question difficulty a hypothesis that can be reasonably drawn is that llms should struggle with questions that are more specific to locations, commodities and time periods. however, we did not see this trend within our annotation. instead, from figure 2 , we see that questions requiring more reasoning tended to be more difficult (questions 3, 8, 11) for the llms over questions more knowledgerecalled oriented (questions 5, 9, 15). in particular, question 8 was difficult as many models responded by telling the user to check the weather forecasts rather than a concrete response. the gpt-4 fared the worst for question 13; although the response was stylistically well-received, it uses generic terminology that is not in line with the industry standard, opting for the term growth over the more accurate vegetative growth or tuber bulking. gpt-4 also had a problem with question 8, where it explained what climate projections were but did not elaborate on their accuracy. some questions were underspecified to test the applicability to the australian climate, such as question 12. surprisingly, only four models failed to recognise that young was a town in new south wales, australia. claude's opus model performed the worst on this question, providing a generic response about its inability to access climate projection data and, therefore, unable to answer the question. a similar answer was provided by claude haiku, but the model still provided an answer after its generic response. mistral 7b and claude haiku had a similar issue but with question 7 and question 11, respectively, where they provided a generic response about being unable to predict weather patterns. the rag models underperformed for specific questions for which the counterpart model did not. a detailed results table for each question and model pair can be found in the appendix: the most surprising observation was that the llama3 70b rag variant under-performed. in particular, the questions that the retriever failed to find relevant impacted the models the most. in particular, as llama3 70b is more aligned with instructionfollowing, it suffered the most performance drop as it refused to answer questions where the answer cannot be found in the documents. this was seen in question 3, where the documents referred to sunburn as sunscald and did not contain relevant information related to sunburn risk. a similar occurrence happened with question 8, where the retriever found information about the number of days over 40 degrees in adelaide (south australia), but the models were either too aligned with instructionfollowing (llama3 70b) or misinterpreted the locations (mistral 7b + rag). overall, we observe that the relevance of retrieved documents impacted the rag models. however, smaller models were less inclined to follow instructions and answered using their internal knowledge rather than our documents and scored higher. ## conclusions my climate advisor is a question-answering tool designed to provide trustworthy climate change risk and adaptation information for farmers and their advisors. our tool is created on an in-house llama 3 with rag, which synthesises information from peer-reviewed scientific literature and trustworthy grey literature. an evaluation framework that outlines criteria designed to differentiate llmgenerated answers to a set of questions was created by our domain experts. while our initial evaluations show a gap between our tool and the leading proprietary systems, the outcome is still encouraging. our analysis shows that our tool is on par for scientific accuracy while providing provenance for explainability. our system can be fine-tuned for further improvements in the near future. note that due to privacy concerns and the financial and environmental costs of proprietary llms, we are limited to open-source models. we will refine the prompting strategy to synthesise climate adaptation information better without sacrificing readability. finally, we plan to expand the input to multimodal data, including numerical data and graphs, for more accurate representations of climate data including climate projections.