Investigating LLM s as Voting Assistants via Contextual Augmentation: A Case Study on the E uropean Parliament Elections 2024 (2024)

Abstract: In light of the recent 2024 European Parliament elections, we are investigating if LLMs can be used as Voting Advice Applications (VAAs). We audit MISTRAL and MIXTRAL models and evaluate their accuracy in predicting the stance of political parties based on the latest “EU and I” voting assistance questionnaire. Furthermore, we explore alternatives to improve models’ performance by augmenting the input context via Retrieval-Augmented Generation (RAG) relying on web search, and Self-Reflection using staged conversations that aim to re-collect relevant content from the model’s internal memory. We find that MIXTRAL is highly accurate with an 82% accuracy on average with a significant performance disparity across different political groups (50-95%). Augmenting the input context with expert-curated information can lead to a significant boost of approx. 9%, which remains an open challenge for automated RAG approaches, even considering curated content.

## introduction while large language models (llms) exhibit unprecedented natural language understanding capabilities, recent work has been exploring political biases @xcite and political reasoning capabilities @xcite in llms. 1 llms prove to have a considerable level of political knowledge and be capable of reasoning in this context. hence, an open question is what is the potential of newer and bigger llms to disseminate political information and inform citizens, as an alternative, or in addition, to traditional media. in pre-election periods, e.g., as in may 2024 upon the 2024 european parliament elections in june, voting advice applications (vaas) have been used to inform the public about the political stance of candidate parties and help citizens find out which parties align the most with. vaas rely on fixed (pre-defined) curated questionnaires, where citizens have to answer several multiple-choice questions, which are then parsed by a rule-based system to compute affinity to the candidate parties. while vaas are good predictors relying on curated content by experts, they are static 'one-size-fits-all' solutions covering a few topics with pre-defined questions that do not necessarily cover the interests of all individual users in detail. in this regard, conversational ai agents (llms) could potentially pose an interesting alternative for political information dissemination that better satisfies the interests of individual users with different priorities in a personalized contextualized responsive manner. users could have open-ended conversations that help them cover more topics of interest in greater detail compared to traditional vaas or other forms of media (e.g., news articles). 2 in this work, we ask the question: could stateof-the-art llms potentially be used as vaas? and how, can we enhance their abilities with contextual augmentation? to answer this question, we employ two of the most recent powerful open-access llms, mistral @xcite and mix-tral @xcite , and audit their abilities as vaas using a newly released dataset based on the "eu and i 2024" (euandi-2024) vaa questionnaire. 3 furthermore, we explore alternatives to improve models' performance by augmenting the input context via retrieval-augmented generation (rag) relying on web search, and "self-reflection" using staged conversations that aim to re-collect relevant content from the model's internal memory. we find that the larger llm, mixtral, is highly accurate with an 82% accuracy on average while augmenting the input context with expertcurated "gold-standard" information can lead to a significant boost of approx. 9%, which remains figure 1 : depiction of the experimental framework. in setting (0), there is no context augmentation. in setting (a) the context is augmented using web search to retrieve relevant content. in setting (b), the context is self-augmented by asking the model preliminary questions to generate a summary for the party and its expected opinion related to the question. in setting (c), the input context is augmented with the party's position related to the question. an open challenge for automated augmentation approaches, even considering curated content. interestingly, rag leads to a substantial performance boost in the case of mistral (+8%). similar to @xcite , we find a significant disparity in how accurately the models represent different political groups (50-95%). ## conclusion and future work in conclusion, we find that recent larger llms have a great potential to serve as voting advice applications (vaas) with an accuracy of 80%+ out-of-the-box. nonetheless, there is a substantial performance disparity among different political groups. the overall performance is improved when we consider contextual augmentation with expertcurated information (90%+). unfortunately, we find that automatically augmenting the context via web-based rag is still an open challenge leading to minor improvements (+3%) for mixtral; contrary web-based rag substantially boosts the performance of the smaller mistral (+8%). in future work, we will further investigate the use of custom-curated rag, which has the potential to further improve performance by grounding models' generation on useful up-to-date credible sources. we will also investigate the potential benefit of continuous instruction-tuning of llms with partisan data, such as the "eu debates" corpus of @xcite . similarly, a usability study assessing the experience of human actors when interacting with llms in the context of politics is a crucial next step.