Retrieval Augmented Generation of Subjective Explanations for Socioeconomic Scenarios (2024)

Abstract: We introduce a novel retrieval augmented generation approach that explicitly models causality and subjectivity. We use it to generate explanations for socioeconomic scenarios that capture beliefs of local populations. Through intrinsic and extrinsic evaluation, we show that our explanations, contextualized using causal and subjective information retrieved from local news sources, are rated higher than those produced by other large language models both in terms of mimicking the real population and the explanations quality. We also provide a discussion of the role subjectivity plays in evaluation of this natural language generation task.

## introduction retrieval augmented generation (rag) has emerged as a powerful technique to mitigate the limited and static knowledge horizon of large language models (llms) @xcite . however, rag methods struggle with tasks that cannot easily be captured through search @xcite . for example, darpa's habitus program foot_0 , which aims to ingest subjective information from local populations into scientific models, recently organized an evaluation, descriptively called predict what the locals would predict (pwlwp), in which natural language generation (nlg) systems had to predict the responses of a local population to several hypothetical socioeconomic scenarios. in particular, the population of interest consisted of adults from the ashanti region of ghana; all scenarios focused on mining in the region. an example of the such a scenario and an explanation by our approach is shown in to properly address this task, this work is the first to propose a rag approach that explicitly models causality (so we can generate causal explanations) and subjectivity (so we can capture the beliefs of a local population). 2 the key contributions of our work are: ## approach in this section, we provide a brief overview of the system, with the pipeline described in detail in section 3 and the experiment in section 4. ## system overview in this section, we provide the technical details of the approach used for the task. as mentioned above, our approach is an instance of retrieval augmented generation, expanded to rely on context that is relevant for the task, i.e., context sentences that are likely to describe beliefs held by a local population for a given hypothetical scenario. the overview of the proposed architecture is shown in figure 1 . we discuss the key contributions below. ## experiment for the experiment, we produce answers to the multiple choice questions and generate explanations for seven scenarios. for each scenario, we produce generations using five different types of context provided to the model: no context and context in the form of thematically related sentences containing either beliefs, causal relations, causal beliefs, or just information related to the topic. see for the evaluation of our response rankings in relation to the local population survey gold data (see appendix b for survey methodology), we compare the probability distribution for all choices produced by our system with the distributions produced by three baselines using mean absolute error (mae). the baselines are llm-based with some additional features for information retrieval, chain of thought prompting, and inclusion of population identity context. two of the baselines use gpt-4 and differ in how they produce distributions to compare against the survey data: the 'gpt-4 topvote' baseline does that by keeping track of how many times each multiple choice answer was selected as the top choice over a number of samples; and the 'gpt-4 calibrated' baseline assigns weights to each response based on the answer choice ranking produced for each sample. the third baseline uses an offline llm (mistral 7b), simplified to approximate a real world scenario, where resources may be limited and privacy concerns may preclude the use of online llm apis. for benchmark implementation and metrics details, see appendix d. for the explanations evaluation, we assemble the scenarios and the generated explanations into a spreadsheet, with one scenario per row followed by the explanations presented in random order. the generated explanations come in a standard essay format, with an introduction, between two and six body paragraphs, and a conclusion. we ask the annotators to provide a score for each explanation without knowing the context type setting on a scale from 0 to 10 (ten being the highest) and provide a rationale for their score. the evaluation was done by two annotators: a domain expert and a linguist. additionally, an annotation supervisor (another linguist) supervised the annotation process and provided meta analysis of the rationale and additional comments on the quality of the outputs. while some authors point out various issues with nlg evaluation criterion inconsistency @xcite , we chose not to provide any specific evaluation guidelines to the evaluators. with the evaluation being closely connected to a project in a real world setting, it was crucial to see what criteria the domain expert views as relevant for their field in this evaluation, and we did not expect previously defined, non-domainspecific criteria to be necessarily relevant. for the linguist evaluator, we expected the criteria to be similar to those described in previous work, but we chose to not provide detailed guidelines beyond a few example criteria to keep the evaluation procedure consistent between the two annotators. ## conclusion in this paper, we make predictions about and generate explanations for hypothetical socioeconomic scenarios by leveraging contextual information retrieved from the web. we show that our retrieval augmented generation approach results in outputs that are both comparable in accuracy to other llm baselines and high in quality, as evidenced by the evaluation conducted by a domain expert and a linguist. using subjective and causal information further improves the quality of the explanations. moreover, by analyzing the evaluations from the two experts, we show how this is another instance of a task where differences in annotations are to be expected and encouraged thanks to its subjectivity. 7b, which was used for one of the baselines, the results could have been different. second, while english is an official language of ghana, it is not the only widely used language in the country. by sticking to one language, we may be missing out on information that could have provided important background knowledge to the model. moreover, we do not account for possible local variations in the use of english, except for focusing our data selection using the location-specific word galamsey. used to refer to illegal, small scale mining, it has been used for information retrieval within the project since it is directly related to our use case. however, based on our analysis of the extracted causes (see section 3.3), our nlp tools are able to extract the intended information from the retrieved data with no obvious issues. third, when providing background information to the model, we operate over individual sentences. by not using broader context (e.g., the full paragraph) for each sentence, we may be eliminating longer reasoning chains described in text. finally, our system is only marginally better than the baselines on capturing the answer distributions from local surveys. there could be multiple explanations for that. for instance, we only provide the model with a small snippet of extracted background information with each api call because of the limited token window allowable with each prompt. larger amount of context could have resulted in better performance. people's decisions are also not necessarily only influenced by information directly related to the question, which is what we have since we use similarity for context sentence selection: general views, e.g., people's attitudes to the importance of legality or money in general could impact their opinions on questions about involvement in illegal mining activities. that said, we believe that providing more relevant, higher quality explanations, which our system does based on qualitative evaluation, is the main benefit of using location-specific context to prompt the model.