Order Up! Micromanaging Inconsistencies in C hat GPT -4o Text Analyses (2024)

Abstract: Large language model (LLM) applications have taken the world by storm in the past two years, and the academic sphere has not been an exception. One common, cumbersome task for researchers to attempt to automatise has been text annotation and, to an extent, analysis. Popular LLMs such as ChatGPT have been examined as a research assistant and as an analysis tool, and several discrepancies regarding both transparency and the generative content have been uncovered. Our research approaches the usability and trustworthiness of ChatGPT for text analysis from the point of view of an “out-of-the-box” zero-shot or few-shot setting, focusing on how the context window and mixed text types affect the analyses generated. Results from our testing indicate that both the types of the texts and the ordering of different kinds of texts do affect the ChatGPT analysis, but also that the context-building is less likely to cause analysis deterioration when analysing similar texts. Though some of these issues are at the core of how LLMs function, many of these caveats can be addressed by transparent research planning.

## introduction the potential for utilising large language models for data analysis has generated widespread interest among researchers across various fields @xcite , and one of the most extensively studied is the field's market-leader openai's chatgpt @xcite . as a freely the specific case under examination will focus on the discussion regarding the green transition, which in political rhetoric generally refers to a shift towards an ecologically sustainable and low-carbon economy @xcite . green transition is an interesting term in the political debate as it evokes very polarising reactions from highly supportive and positive to strongly negative and dismissive. the reactions also vary a lot by country: for example, in norway, the term "green transition" has very positive connotations and was even voted as the word of the year in 2015 @xcite . in finland, the government in power since 2023 led by the national coalition party and its support party the finns has made efforts to replace the term with other expressions such as "clean transition" ("puhdas siirtymä") or "the blue-white transition" ("sinivalkoinen siirtymä") @xcite . to study the debate from the perspectives of both politicians and the public, we analysed communication on twitter and within the finnish parliament during the conference of parties climate change conference (cop27) held in egypt in late 2022. we chose the highly publicised international event as green transition was one of its core focus points @xcite and because it stirred conversation both online and in the parliament so close to the finnish parliamentary elections of 2023. the data comprises parliamentary speeches and twitter comments collected from october 19, 2022, to december 1, 2022. each dataset contains 20 texts, resulting in a total of 20 tweets (t) and 20 parliamentary speeches @xcite . the study expects to determine if and how the order and context in which texts are presented to the llm influence the outcomes it should also be pointed out that the crosschat "memory" feature for plus tier chatgpt subscribers was not available in the eu area during the testing and so had no impact on it. the feature that allows chatgpt to "remember details between chats, allowing it to provide more relevant responses" has was made available in eu in september 2024 (coombes 2024, openai 2024c).