Measuring Alignment to Authoritarian State Media as Framing Bias (2020)

Abstract: We introduce what is to the best of our knowledge a new task in natural language processing: measuring alignment to authoritarian state media. We operationalize alignment in terms of sociological definitions of media bias. We take as a case study the alignment of four Taiwanese media outlets to the Chinese Communist Party state media. We present the results of an initial investigation using the frequency of words in psychologically meaningful categories. Our findings suggest that the chosen word categories correlate with framing choices. We develop a calculation method that yields reasonable results for measuring alignment, agreeing well with the known labels. We confirm that our method does capture event selection bias, but whether it captures framing bias requires further investigation.

## introduction information warfare is of increasing concern in the current era, persecuted through various channels. recently, social media has become a key focus, however traditional media continues to play an important role. as @xcite notes, the economically powerful authoritarian chinese communist party (ccp) is rapidly expanding its influence over media production and distribution channels globally, and is applying tactics developed for co-opting the chinese diaspora in different countries to local mainstream media. the scale of ccp investment may also signify the importance they attach to the role of traditional media @xcite . there have been some apparent successes. for example, a recent study significantly correlated access to ccp state media in africa with favourable views of the ccp-state @xcite . on the other hand, ccp media is not considered to be nearly as successful in the west. as an authoritarian party-state, ccp-branded media suffers from a large credibility deficit. as varral (2020) notes, at china global television network (cgtn) it is generally accepted internally that cgtn's influence is not what is could be given the amount of funding they receive. nevertheless, given the scale of investment, it is possible that ccp state media are learning from their mistakes, refining their messaging, and iteratively improving their ability to influence foreign nations. of equal concern are efforts to overcome the key branding issue. as detailed by @xcite , the ccp's media strategy has long included the policy of "borrowing a boat to go on the ocean": strategic partnerships with foreign media organizations to spread their messaging. a recent example is advertorial agreements with western media such as the washington post to run a china daily supplement in their own newspapers (albeit with a small warning label). such agreements allow the ccp to shake off their own brand and borrow from the credibility carefully developed by such newspapers. the "borrow a boat" policy has also been augmented with the "buy a boat" policy: strategic mergers and acquisitions of foreign media assets @xcite . for example, auckland's only chinese-language 24-hour radio station, fm 90.6, was taken over by a subsidiary of china radio international (cri), and now sources all its news from cri and its australian subsidiary @xcite . with a bought and paid for boat, audiences do not even have the luxury of a small warning label on the content. related to this last category are media outlets that are owned by agents loyal to the ccp, either ideologically or financially motivated. a prime example is taiwan's china times, a company of the want-want group who are proudly aligned to the ccp. 1 advertorial supplements from china daily, subsidiaries of ccp state media, and media outlets run by loyal agents of the ccp are examples of alignment to authoritarian state media, the focus of our work. we define such alignment as effectively acting as an agent of that state: spreading their messaging, and working towards their goals. alignment in this sense can therefore pertain to agents (media outlets or journalists), and also to text (news articles). in our work with civil society organizations and journalists around the world, we have identified the usefulness of an nlp tool to detect and track media that are aligned to the ccp. the primacy of investigative journalism is not in question. in many cases the alignment of agents can be established by investigating the people and companies involved. however, such an nlp tool could be helpful in a number of ways. it could flag new and emerging media for investigation. it could detect the point in time when a media organization undergoes change in the bias of its content (which may be coincidental, for example, with when it was purchased by another company, flagging the company for investigation). it could be used to compare ccp influence across countries and time. it could hopefully also give insight into how the alignment is expressed -e.g., what issues and framings are important -which may help to understand the evolving nature of ccp information operations. finally, it would ideally be able to provide hard statistical evidence to support claims of bias. from an nlp perspective, this research problem is novel (as far as we are aware). yet by operationalizing alignment in terms of media bias we are able to draw upon a wealth of prior research. our conceptualization of media bias follows the sociological model reported in @xcite . the types of bias we intend to address concern what is reported, and how it is reported. what is (and is not) reported is covered by event selection bias. how it is reported is covered by word choice, labeling, and framing bias. in this work we aim to measure framing bias, and find that our methodology also captures event selection bias. we ideally want an unsupervised solution that is applicable to as many languages (and therefore national contexts) as possible. in the abstract, our methodology is to treat the ccp state media data as a "propaganda distribution." as news outlets in other countries will follow their own news cycle and political context, we need a way to factor out this part of their observed distribution. we then check the remainder for alignment in the direction of the ccp propaganda distribution. in the present work, we concretely implement this methodology by using the frequencies of words in psychologically meaningful lexical categories drawn from the linguistic inquiry and word count (liwc) dictionary @xcite . we identify 15 categories that appear to relate to framing. for each chosen category, we calculate an indicator variable representing whether a media outlet differs from the taiwanese mean in the direction of the ccp, and take the average of these indicators as a score. our results are a strong match for the expected labels, and also capture the ideological leaning of the four taiwanese media outlets studied. we therefore have reasonable confidence that our methodology captures the known bias in the data. however, an initial qualitative study is inconclusive regarding whether we have captured framing bias. we do, however, confirm that our methodology unexpectedly captures event selection bias. therefore, a careful analysis of framing in our dataset should be conducted to confirm the nature of the bias our method is capturing. field (2018) develop a multilingual method for detecting frames based on the coding in the media frames corpus @xcite . their methodology could ostensibly be used to study international framing bias. they collected frame-specific lexicons in english using the pmi of words with the annotated frames, then translated the lexicons into the target language, using word embeddings to refine the lexicon through query expansion so as to more accurately capture the local linguistic and political context. our experiments with this methodology did not prove as successful as our reported methodology, yielding more variable and noisy results. it is possible that the lexicon construction process is too noisy for our purpose. we also suspect that using a pre-defined set of high-level frames is not sensitive enough. it is possible that our method of using word categories that relate to framing as opposed to the high-level frames themselves is more flexible, allowing unannotated framings to be expressed in the relative frequencies of lower-level word categories. @xcite follow an approach most similar to ours, using the moral foundations dictionary @xcite to analyze framing bias in an unsupervised manner. their results appear reasonable, and are supported by other studies that suggest the usefulness of moral foundations in understanding framing bias @xcite . one advantage of liwc is that, to the best of our knowledge, it has been translated into more languages than the moral foundations dictionary. our experience is that lexicon quality is important. nevertheless, comparing our method to a chinese version of the moral foundations dictionary is necessary future work. more recent work has considered multi-label and multi-lingual frame detection in a few-shot setting with minimal labeled data @xcite . however, their data comes solely from articles about gun violence in the u.s. and also uses a pre-defined set of framings. a fairly comprehensive plan for research into automated framing bias detection is offered by @xcite . however, this methodology still involves upstream inaccuracies and components of supervised learning -e.g., aligning entity mentions through coreference resolution. although we support this research direction as well principled and promising, our initial investigation aims to be fully unsupervised for two reasons. the first is to avoid propagation of upstream errors as much as possible. 2 second, due to our desire to work with as many languages (and therefore national contexts) as possible, we want to reduce our reliance on language-specific resources such as supervised coreference resolution classifiers. @xcite released a dataset for analyzing bias in coverage of the ukranian crisis including a range of media outlets from 43 countries. including russian state media also makes this work closer to our task of measuring alignment to authoritarian state media, although it only deals with a single issue. application of our methodology to this dataset remains as future work. the present work complements our existing work on ccp information operations @xcite . the development of our tool could assist this research by identifying local agents (e.g. journalists, influencers) working on behalf of the ccp, which have been found to be an important component of ccp information warfare directed against taiwan. media outlet label num. documents num. tokens china internet information center ccp state media 65,804 45,203,219 china daily ccp state media 6,190 4,877,458 china development gateway ccp state media 544 487,252 china times (ccp) ccp state media 543 632,425 global times ccp state media 24,585 15,263,884 national party media information public platform ccp state media 39,556 23,431,048 xinhua ccp state media 806 751,059 people's daily ccp state media 102,158 64,067,371 qiushi ccp state media 3,210 3,063,152 china times (taiwan) aligned to ccp 17,143 5,790,180 united daily bias towards ccp 17,275 6,137,210 liberty times bias against ccp 6,704 2,015,226 apple daily oppose ccp 6,076 1,864,850 ## data we scraped news articles from the media outlets listed in ## methodology in this section we first motivate our use of liwc categories, and our method of selecting which categories reliably relate to framing. we then outline our calculation method and procedure for statistical inference. ## results for each domain to be scored, we perform 1,000 bootstrap samplings of the whole dataset, and report the distribution of results. the 95% confidence interval is calculated in terms of percentiles of the bootstrap distribution. we are interested to know how stable the results are with varying amounts of data. we therefore repeat the bootstrap scoring procedure with varying time periods. all scores start at the january 1st, 2020, and extend an integer number achievement anger anxiety death friend health leisure money negative emotion number positive emotion religion sexual time we 0.05 0.00 0.05 0.10 0.15 0.20 npmi media outlet ccp state media china times united daily liberty times apple daily figure 3: npmi of liwc categories with sentences including the word "protest." it stands out that the ccp prefers the death and religion categories. randomly sampling 20 sentences with "protest" and these categories reveals that when the ccp report on protest in our dataset, they discuss murder, crime, and damage to property. however, they also talk about protests in the us, suggesting that this statistical signal is also picking up event selection bias. limitations: although the results appear to be stabilizing, we believe this length of time is still short enough to leave uncertainty about the reliability of our methodology. the results should also be reproduced in different national and linguistic contexts for more thorough verification. ## analysis our results suggest we have measured bias in our dataset in accordance with the expected labels. however, our case that we have captured framing bias rests on the findings of section 4.1: that the liwc categories we chose correlate well with the annotations in the media frames corpus. this case needs to be strengthened by a careful look at frames in our data. we conducted a cursory analysis that is suggestive yet inconclusive about framing bias. however, it does confirm that our methodology captures event selection bias. we identified sentences that include at least one instance of the word "protest." we then calculated the npmi of liwc categories with these sentences following equation (1), however at the sentence not document level. 8 the distribution over liwc categories is given in figure 3 . we observe that the ccp prefers the categories "death" and "religion" compared to taiwanese media (with china times biasing in their direction). given how these categories correlate with frames as observed in figure 1 , these would appear to indicate the ccp frames protest in terms of "crime and punishment" and "morality" in our dataset. to check this hypothesis, we we also observed that the majority of these sentences were about protests in the united states after the murder of george floyd. we therefore hypothesized that we are also measuring event selection bias. ## conclusion we are surprised by how well our calculation method matches the known bias labels. lexical category approaches have some well known difficulties, particularly that they do not consider syntax -e.g., a negative emotion word following a negation may not express negative emotion. furthermore, it may be that following ccp framing does not necessarily represent the ccp's interests in different national contexts. we observe an example of this: the china times uses "we" in sentences containing "china" far more than any other taiwanese newspaper -but the ccp state media does not. despite these complications, with enough data our methodology appears to work qualitatively well and yield stable results. drilling down to consider framing around specific issues also appears to yield qualitatively reasonable insights consistent with our expectations. our hypothesis is therefore that the aforementioned complications are effectively smoothed over by enough data. our assumption that the liwc categories capture framing, and therefore that our methodology captures framing bias, is based on the observation of meaningful correlations of these categories with the annotations in the media frames corpus. however, we also saw in our qualitative analysis that we are picking up event selection bias. a closer examination will be necessary to establish the extent to which our calculation method is picking up framing and event selection bias. it will be necessary to observe data over a longer period of time in order to validate the observed stability of our conclusions. we must also be apply it to different linguistic and national contexts, over many more media organizations. we should also compare our methodology to another apparent correlate of framing: the moral foundations dictionary. the liwc resource is also a constraint on the number of languages we can handle. currently there are 15 languages covered by liwc. therefore, future work is to find a way to cover more languages, ideally in an unsupervised manner that does not rely on such resources. our calculation method is very simple, and can almost certainly be improved. for example, we have found that using vectors of our chosen liwc category frequencies and taking the first principal component also yields a surprisingly coherent ideological ordering of media, including the different ccp state media outlets. vector-based calculation methods are likely to be more sensitive to the statistics than our binary indicator variables, and may also facilitate more complicated calculations, such as comparing to multiple, different propaganda distributions. it also makes more sense to consider frames as occupying regions in the 15-dimensional space of frequencies induced by our liwc category selections (see figure 1 ), as opposed to considering the categories separately. how might our methodology be applied to other authoritarian states? our results suggest that collecting two months of data from the authoritarian state media, and from media outlets of interest with unknown labels, should be enough to apply our scoring calculation. it will be interesting to explore other potential applications of our methodology, such as detection of content farms @xcite , or networks of twitter accounts operated by the ccp. we make our code publicly available to facilitate subsequent research. 9 in the introduction we outlined a number of goals for the nlp tool we have set out to build. how does our methodology help to those ends? with two months of data we could tag an emerging or unknown media outlet as potentially part of ccp information operations. we have not modeled the time dimension so far, so we could not detect when a media outlet undergoes a change in alignment. we could compare the media outlets within and across countries, but not across time. drilling down around specific concepts, events, or entities does facilitate analysis of how alignment is expressed, as our analysis of protest suggests. our calculation method can be considered a form of statistical evidence, however should be further validated as outlined above. we note an important weakness in our methodology. taking the difference from the average national media will fail where the national media is almost entirely aligned to the ccp. for example, it is estimated that in australia 95% of the chinese language media is under ccp influence @xcite . a similar situation exists in new zealand @xcite . in such contexts our methodology would require modification. it would be most desirable to seek a more absolute measure of alignment, but it may be difficult to develop one that is automatic and works in every national political context. for future work, we intend to explore the use of word embeddings with techniques developed in the gender bias literature to try and identify directions in the ccp state media embedding space that capture the propaganda distribution. initial experiments are promising. for example, the distance between "taiwan" and "country" is much larger in the ccp embedding space then in the taiwan (minus china times) embedding space, with china times falling quite far outside the taiwanese distribution in the direction of the ccp. 10 it may be possible to build a list from official ccp framings of particular issues such as taiwan @xcite . however, such a static approach would be less ideal than one that could be automatically learned and updated daily. we leave these challenges for future work.