Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a C hat GPT and Bard Newspaper (2023)

Abstract: Neutrality is difficult to achieve and, in politics, subjective. Traditional media typically adopt an editorial line that can be used by their potential readers as an indicator of the media bias. Several platforms currently rate news outlets according to their political bias. The editorial line and the ratings help readers in gathering a balanced view of news. But in the advent of instruction-following language models, tasks such as writing a newspaper article can be delegated to computers. Without imposing a biased persona, where would an AI-based news outlet lie within the bias ratings? In this work, we use the ratings of authentic news outlets to create a multilingual corpus of news with coarse stance annotations (Left and Right) along with automatically extracted topic annotations. We show that classifiers trained on this data are able to identify the editorial line of most unseen newspapers in English, German, Spanish and Catalan. We then apply the classifiers to 101 newspaper-like articles written by ChatGPT and Bard in the 4 languages at different time periods. We observe that, similarly to traditional newspapers, ChatGPT editorial line evolves with time and, being a data-driven system, the stance of the generated articles differs among languages.

## introduction instruction-following language models (ilms) are omnipresent. their use is not so extended as that of search engines yet, but due to the availability and high quality of systems and models such as alpaca @xcite , bard @xcite , bloomz and mt0 @xcite , @xcite , llama 2-chat @xcite , or @xcite , their use is expected to be more common in the near future. these models face several problems being the most relevant the lack of trustworthiness @xcite @xcite . they are not ready to be used as a source of reliable information if their outputs are not fact checked. a second big issue with systems based on language models (lm) is the fact that they might reproduce the biases present in the training data @xcite . biases range from cultural missrepresentation due to data imbalance to offensive behaviour reproduced from written texts. lms are finetuned into ilms either in a supervised way using input-output pairs and an instruction @xcite @xcite or with reinforcement learning from human feedback @xcite . in both cases, the finetuning should help removing bias. but neutrality is something very difficult to achieve, also for the humans that generate the supervisory data. the finetuning phase might therefore over correct the original biases or introduce new ones. for methods that generate the supervision data with the lm itself, the original biases might be inherited. we focus on a specific use of ilms: the writing of newspaper articles. journals and newspapers follow an editorial line which is in general know to the reader. besides, sites such allsides, foot_0 media bias fact check foot_1 (mb/fc), or ad fontes media foot_2 provide ratings about the political bias of (mostly usa) media sources and their quality with respect to factual information. with these ratings, conscientious readers can make informed decisions about which media outlets to choose in order to get a balanced perspective. but what happens when journalists use systems such as chatgpt or bard to aid in their writing? as said above, humans also have biases, the danger lies in being unaware of them, as they might affect the user's/reader's perspective @xcite . chat-gpt already warns its users about misinformation. however, the political bias, if any, is not known apart from the subjective perception that a user has. we address the question above for articles generated by chatgpt and bard in four languages: english, german, spanish and catalan. we do this in an automatic and systematic way with almost no human intervention so that the method can be easily extended to new languages and other ilms with few effort. we do not aim at classifying individual articles with their specific bias, but to classify the media source (an ilm in this case) as left or right-oriented in a similar way as the media bias sites do for newspapers and other media outlets. ## corpora compilation we approach our task as a classification problem with two classes: left (l) and right (r) political orientations. this is a simplification of the real problem, where articles can also be neutral and there might be different degrees of biases. previous work relied on 3 or 5 classes, always including the neutral option @xcite . in these works, data was manually annotated creating high quality training data but also limiting a lot the scope of the work in terms of languages and countries covered. when using the fine-grained classification scale, the authors acknowledge a bad generalisation of the classifiers to new sources. on the other hand, @xcite and @xcite exclude the neutral class and work with a binary or multiclass left-right classifications of tweets from spanish and italian politicians respectively, but their work does not include longer texts. the binary classification might be justified as they worked with tweets, a genre where people tend to be more visceral and therefore probably more polarised. in our case, we need to be sure that the classifier generalises well to unseen sources and we stick to the 2-class task while minimising the number of neutral articles in training (see below). distant supervision. as far as we know, only a manually annotated newspaper corpus in english @xcite and another one in german @xcite are available. we follow a different approach in the spirit of @xcite and @xcite . we do not manually annotate any article, but we trust all-sides, mb/fc, political watch and wikipedia (the latter only in cases where the information is not available in the previous sites) with their classification of a newspaper bias. we extract this information for newspapers from usa, germany, spain and catalonia. with the list of newspapers, their url, foot_3 and their stance, we use oscar, a multilingual corpus obtained by filtering the common crawl @xcite , to retrieve the articles. appendix a lists the sources used in this work: 47 usa newspapers with 742,691 articles, 12 german with 143,200, 38 spanish with 301,825 and 19 catalan with 70,496. topic modelling. not all articles have a bias, some topics are more prone than others. while the sports section of a newspaper is usually less prone to reflect political biases, the opposite happens with the international section. we therefore use topics to select a subset of relevant training data for our binary classification. we do topic modelling on the articles extracted from oscar using mallet @xcite which applies lda with gibbs sampling. we cluster the data in both 10 and 15 groups per language, roughly corresponding to the number of sections a newspaper has. the keywords extracted for each topic are listed in appendix b. we choose articles that fall under the topics we label as international, government, law & justice, economy, live science/ecology, and specific language-dependent topics such as immigration and violence for english, nazism for german, and social for spanish. the selection is done after the inspection of the keywords. for the final dataset, we do the union of the selected articles clustered to 10 and 15 topics. the process filters out 49% of the spanish articles, 39% of the german and 31% of the english ones. preprocessing and cleaning. we discard articles with more than 2000 or less than 20 words before cleaning. afterwards, we remove headers, footers and any boilerplate text detected. this text has the potential to mislead a neural classifier, as it might encourage the classifier to learn to distinguish among newspapers rather than focusing on their political stance. we select a newspaper per language and stance for testing and clean manually their articles. to create a balanced training corpus for each language, we randomly select a similar number of left and right-oriented articles from the remaining collection. this balanced dataset is divided into training and validation as shown in chatgpt/bard corpus. we create a multilingual dataset with 101 articles. for this, we define 101 subjects including housing prices, abortion, tobacco, barak obama, etc. and translate them manually into the 4 languages (see appendix d). the subjects consider topics prone to have a political stance such as those related to feminism, capitalism, ecologism, technology, etc. we also include proper names of people in the 4 countries being considered, whose biography may differ depending on the political stance of the writer. these subjects are inserted into the template prompt (and its translations into german, spanish and catalan): 5 ## political stance classification the network. we finetune xlm-roberta large @xcite , a multilingual transformer-based masked lm trained on 100 languages including the 4 we consider. the details of the network and the hyperparameter exploration per model are reported in appendix f. the models. we train 4 models: 3 monolingual finetunings with the english, german and spanish data, plus a multilingual one with the shuffled concatenation of the data. all models are based on multilingual embeddings (roberta) finetuned either monolingually or multilingually. notice that we do not train any model for catalan. with this, we want to compare the performance of mono-and multilingual finetunings and explore the possibility of using multilingual models for zero-shot language transfer. coarse classification with newspaper articles. this is the behaviour we obtain for all the test newspapers but for the german right-oriented newspaper: die preußische allgemeine zeitung (paz). the german model is trained only on 12 when we use the english system, the language does not play a role any more and only the stance features are considered. when we apply the english model to the catalan newspapers we do not obtain satisfactory results though (95±1% for the left but 16±3% for the right newspaper) showing that the relatedness across languages is important. the multilingual model however properly detects the stance of the catalan newspapers probably because it has been trained with an heterogeneous corpus that includes a related language (spanish). we are able to perform zero-shot language transfer classification when we deal with close related languages. coarse classification with ilm-generated articles. the bottom part of we first focus on the english and spanish models as the german one did not properly classify our test newspapers. the most relevant aspect to notice in chatgpt is the strong change in political stance between february (v02) and may (v05) followed by a movement towards neutrality in august (v08). we checked that this polarity change is not an effect of the length of the outputs -the major shallow change in the generated articles. the training data in english has 5,730 l-6,988 r articles with 584<length(words)<624 (similar to chatg-ptv05 length) and 4,563 l-7,127 r articles with 331<@xmath0. in both cases the number of articles is larger for the right stance, but the prediction for chatgptv02 clearly points towards the left, rejecting the hypothesis that length plays a role in the classification. a similar thing happens for spanish. according to our models, the may 24th version of chatgpt in english and spanish would have an editorial line close to the right ideology, which differs from the ideology of the previous versions. notably, this period corresponds to the time when chatgpt experienced a performance drop in several tasks according to @xcite . the german and catalan outputs would still show an imprint from the left ideology also in v05 but more diverse training data would be needed to confirm this with our monolingual models. it is interesting to notice that if we use the english monolingual model for german and catalan, we still get the left imprint (60±10% for german and 87±7% for catalan). so we have indications that the political stance of chatgpt depends on the language, which is not surprising in a data-driven system. the last version, chatgptv08, produces the most neutral texts, with only german clearly leaning towards the left. the two generations, v08a and v08b, show that results are robust and are not tied to a particular generation. hartmann et al. (2023) used pc, 38 political statements from the voting advice application wahl-o-mat (germany) and 30 from stemwijzer (the netherlands) to conclude that chatgpt's ideology in its version of dec 15 2022 was pro-environmental and left-libertarian. a study conducted by the manhattan institute for policy research foot_10 reported that chatgpt tended to give responses typical of left-of-center political viewpoints for english (rozado, 2023). the authors administered 15 political orientation tests to the chatgpt version of jan 9. their results are consistent with our evaluation of the feb 13 model. finally, motoki et al. (2023) performed a battery of tests based on pc to show that chatgpt is strongly biased towards the left. the authors do not state the version they use, but the work was submitted on march 2023. all these results are therefore before the move to the right we detected in may. ## summary and conclusions media sources have an editorial line and an associated bias. getting rid of political biases is difficult for humans, but being aware of them helps us getting a global view of news. biases are sometimes clear and/or appear in form of harmful text, but sometimes are subtle and difficult to detect. these subtle hidden biases are potentially dangerous and lead to manipulation whenever we are not aware of them. in this work, we systematically studied the subtle political biases behind chatgpt and bard, those that appear without assigning any persona role @xcite . we showed that chatgpt's orientation changes with time and it is different across languages. between feb and aug 2023, chatgpt transitioned from a left to neutral political orientation, with a right-leaning period in the middle for english and spanish. the evolution for bard cannot be studied yet. its current version as of aug 2023 consistently shows left-leaning for the 4 languages under study. this bias is independent on the factual mistakes that the model generates, and should also be considered by its users. we provide models to regularly check the bias in text generations for usa, germany and spain, as well as in closely related political contexts and languages using a zero-shot approach. as a by-product of our analysis, we created a multilingual corpus of 1.2m newspaper articles with coarse annotations of political stance and topic. we show that distant supervision allows us to build meaningful models for coarse political stance classification as long as the corpus is diverse. we make available this data together with the lms generations and our code through zenodo (españa-bonet, 2023) and github. 12