A rabic Mini- C limate GPT : A Climate Change and Sustainability Tailored A rabic LLM (2023)

Abstract: Climate change is one of the most significant challenges we face together as a society. Creating awareness and educating policy makers the wide-ranging impact of climate change is an essential step towards a sustainable future. Recently, Large Language Models (LLMs) like ChatGPT and Bard have shown impressive conversational abilities and excel in a wide variety of NLP tasks. While these models are close-source, recently alternative open-source LLMs such as Stanford Alpaca and Vicuna have shown promising results. However, these open-source models are not specifically tailored for climate related domain specific information and also struggle to generate meaningful responses in other languages such as, Arabic. To this end, we propose a light-weight Arabic Mini-ClimateGPT that is built on an open-source LLM and is specifically fine-tuned on a conversational-style instruction tuning curated Arabic dataset Clima500-Instruct with over 500k instructions about climate change and sustainability. Further, our model also utilizes a vector embedding based retrieval mechanism during inference. We validate our proposed model through quantitative and qualitative evaluations on climate-related queries. Our model surpasses the baseline LLM in 88.3% of cases during ChatGPT-based evaluation. Furthermore, our human expert evaluation reveals an 81.6% preference for our model’s responses over multiple popular open-source models. Our open-source demos, models and curated instruction sets are available here : https://github.com/mbzuai-oryx/ClimateGPT

## introduction climate change poses a significant and urgent challenge in our modern era, as it has far-reaching and profound consequences for the environment, ecosystems, economies, and human well-being. figure 1 : arabic mini-climategpt conversation: user: how does climate change affect food security and freshwater? response: climate change has a significant impact on food security and freshwater. for example, high temperatures can lead to crop loss, water scarcity, the spread of pests and diseases, resulting in reduced agricultural production and increased food prices. additionally, high temperatures and changes in precipitation patterns can lead to a decline in the availability of freshwater, affecting agricultural production, energy, and other industries that rely on irrigation methods.user:in this case, how can we preserve the crops? response:crops can be preserved through various methods, which include using sustainable agricultural practices such as efficient irrigation techniques, crop diversification, and crop rotation. these efforts can help reduce reliance on traditional crops and enhance the resilience of agricultural systems to climate change. (best viewed in zoom) ## related work climateq&a @xcite is a chatgpt based tool that can distil climate science knowledge based on ipcc reports and environment articles to user friendly responses. although this model is available on huggingface, it is not open-sourced as it is dependent on chatgpt. moreover, its responses are limited to english. climatebot (rony et al., 2022) is another machine reading comprehension bot for question answering about climate change documents. it is limited to localizing the answer from a given context as it is based on a clustering approach and lacks the generative capabilities of modern llms. the model responses and document embeddings are limited to english. to the best of our knowledge, we are the first to introduce an open-sourced, climate-specialized conversational agent in arabic. climabench: this is a benchmark dataset @xcite for climate change text understand-ing, which is built upon the publicly available surveys released by cdp (formerly the carbon disclosure project @xcite ) and the naic climate risk disclosure survey. climabench consists of clima-insurance and clima-cdp for text classification and clima-qa for question answering tasks based on the cdp data. we use this clima-qa data compiled from the cdp-cities, cdp-states and cdp-corporation subsets and preprocess into 485k question answer (qa) pairs. the proposed arabic clima500-instruct dataset is built upon ccmrc and climabench datasets. ## arabic mini-climategpt our arabic mini-climategpt is an open-sourced conversational agent instruction-tuned for climate change and sustainability in arabic. we first create a conversational style instruction-tuning dataset named clima500-instruct in arabic language based on ccmrc and climabench (see sec. 3.1). our arabic mini-climategpt is built upon the vicuna-7b model by fine-tuning on the clima500-instruct. we also integrate a dedicated inference module to incorporate region-specific documents in arabic to extend the knowledge base of our model. (see sec. 3.2) ## experiments as llms become more sophisticated, the existing open benchmarks may no longer suffice to adequately assess their capabilities. these benchmarks are typically designed to test chatbots on a limited set of tasks, such as question-answering or context-based localization. to address this, we compare the model responses with ground truth responses on a held-out clima500-instruct arabic test set using chatgpt. we compare the generated responses of our model with other open-source llms like vicuna @xcite , stanford alpaca @xcite , dolly v2 @xcite and arabic alpaca @xcite on the test set. for each question within the test set, we provided chatgpt with the following inputs: (i) the ground-truth answer text, (ii) the response texts generated by our model, and (iii) the text generated by the designated competitor model. in this context, chatgpt's role was to assess and contrast the text outputs of our model and the competitor model, subsequently selecting the response that best semantically aligns with the ground truth. if neither of the generated texts semantically matches the ground truth, chatgpt outputs "neither" as the response. the row headers in our arabic mini-climategpt achieves an overall win rate of 81.6% on human expert evaluations. ## conclusion in this paper, we present arabic mini-climategpt, a climate-specialized conversational agent in arabic. our framework incorporates a vector embedding based retrieval mechanism to extend the knowledge-base of our model during inference. our proposed dataset clima500-instruct is an ex-