Press Freedom Monitor: Detection of Reported Press and Media Freedom Violations in T witter and News Articles (2021)

Abstract: Freedom of the press and media is of vital importance for democratically organised states and open societies. We introduce the Press Freedom Monitor, a tool that aims to detect reported press and media freedom violations in news articles and tweets. It is used by press and media freedom organisations to support their daily monitoring and to trigger rapid response actions. The Press Freedom Monitor enables the monitoring experts to get a fast overview over recently reported incidents and it has shown an impressive performance in this regard. This paper presents our work on the tool, starting with the training phase, which comprises defining the topic-related keywords to be used for querying APIs for news and Twitter content and evaluating different machine learning models based on a training dataset specifically created for our use case. Then, we describe the components of the production pipeline, including data gathering, duplicates removal, country mapping, case mapping and the user interface. We also conducted a usability study to evaluate the effectiveness of the user interface, and describe improvement plans for future work.

## introduction press freedom is under constant and increasing attack, even in europe. therefore, now more than ever, it is important to monitor developments and advocate for measures to protect press and media freedom. mapping media freedom foot_0 (mmf) is a project and platform which identifies and documents threats, violations and restrictions faced by media workers across europe and beyond. the documented incidents include physical attacks, threats of violence made online and offline, legal actions aimed at silencing critical coverage and moves to block access to information or reporting on incidents or denying access to inde-pendent and government-critical media platforms. these incidents are published as alerts on mmf and combined with analysis reports they provide an overview of the current state and development of press and media freedom in europe. this project is run by the media freedom rapid response (mfrr foot_1 ), a rapid response mechanism against press and media freedom violations in the european union member states and candidate countries 3 . it provides legal support, shelter, public advocacy and information to protect journalists and media workers. the alliance is led by the european centre for press and media freedom (ecpmf) in conjunction with article 19, the european federation of journalists (efj), free press unlimited (fpu), the institute for applied informatics at the university of leipzig (infai), international press institute (ipi) and cci/osservatorio balcani e caucaso transeuropa (obct). the project commenced in 2020. it is funded by the european commission. the mmf alerts guide mfrr to directly engage with and help at-risk journalists and media workers the alerts are submitted mainly by the mfrr monitoring experts, as well as an international network of local partners. however, mmf is also a crowdsourced platform that enables anyone to upload an alert, which is then verified by the expert network before publication to guarantee the reliability of the cases and the comprehensiveness of the published details. in order to support the labour-intensive manual monitoring of incidents, we developed the press freedom monitor which regularly monitors and automatically detects reports about press and media freedom violations in vast amounts of online published text sources, namely news articles and tweets. the automatic detection is based on a trained deep learning model. these detected incident reports are then verified by the monitoring experts who create and publish an mmf alert in-cluding comprehensive details and trigger further response actions. the advantage of integrating automatic extraction processes is that they can use a wider range of sources and provide a faster alert mechanism. this in turn means that more violations can be found and more sources can be provided for each case, allowing a more realistic and reliable assessment of the press freedom situation. ## related work monitoring content published in social networks to detect abuse, harassment, or freedom violations has been the subject of several research projects. @xcite , @xcite , @xcite worked on the detection of misogynistic language and hate speech towards women on twitter. @xcite developed an approach to automatically detect suicide-related tweets. @xcite and @xcite aimed to detect and analyse tweets that contain racism, sexism and abusive language. the starting point was always to define a list of keywords, hashtags, and accounts related to the topic and to collect the relevant tweets using twitter api. however, the absence of domain-specific labelled corpora drove many projects to manually create their own training dataset to meet their needs. all the works listed above consider historic or archived data, whereas our crawling process runs continuously and collects almost live tweets and news multiple times a day. furthermore, our project is distinguished from others by not aiming to detect direct abuses, harassment or freedom violations but reported attacks and violations directed to the specific group of media actors, which include journalists, media workers and media companies. ## production phase the aim of the press freedom monitor as an application in production is threefold: first, and most importantly, it should constantly monitor news and tweets and present automatically-detected reports about press freedom violations to the monitoring experts in a convenient format. second, the tool should provide support for the monitoring experts when searching for further items that report about the same incident during the verification process. third, it should help to improve the model by collecting manual feedback about the classified items, which can be used as additional training data. figure 1 shows the process pipeline of the press freedom monitor in production. the data collection is performed continuously as described above. the trained model is used to detect reported violations within the gathered data. in parallel, duplicate removal, case mapping, and a country mapping based on geocoding is performed to increase the usability of the tool. ## usability study to evaluate the usability of the press freedom monitor, we conducted a usability study with 7 participants. for this purpose, we used the computer system usability questionnaire (csuq) version3 @xcite . the study showed a 79% overall satisfaction among the monitoring experts; similarly, the system usefulness achieved 79%, and information quality 76%. moreover, the user interface quality achieved the highest score with 84%. ## language extension: hungarian by monitoring english language content, we already have broad geographical coverage. extending the monitoring with additional languages can detect incidents not detected via the monitoring in the english language. we selected hungarian as the test language, based on the ongoing deterioration in the field of press and media freedom in hungary. the creation of specific training data for additional languages and the adaptation to language-specific processes (such as ner or geocoding) was not feasible in our project. instead, we use google translate foot_11 to translate the texts to english and classify them with the model trained for english. to cope with the challenge that the monitoring experts might not speak hungarian and therefore can not understand the news articles or tweets that are presented, the frontend also shows the translated text for each item first. this allows the experts to assess the content even if they do not speak the language. again, each item can be clicked to show it in its original context and language for further inspection. though the hungarian version detects less content than the english one based on the smaller amount of news and tweets in hungarian, it has already proved to be a valuable source of information during initial tests and usage. it detected an incident in hungary and an incident in germany that were not already known to the monitoring team and also not detected by the english monitoring. the first evaluation of 237 items resulted in a recall of 0.8 and precision of 0.95 when setting the confidence threshold to 0.5. ## future work future work includes the extension of the manual feedback as well as the retraining of the models including this feedback data. a common improvement request by the experts was to further invest in displaying texts about the same incident together. thus, future work will involve clustering based on incidents to enhance the case mapping performance. however, we need to evaluate how well clustering can perform in this already narrow use case with high similarities between different incidents. as the first step, we plan to use the country mapping to present the incidents based on mapped countries. this is already expected to provide a better distinction between different incidents and might result in a first improvement with a good cost-benefit ratio. furthermore, we want to extend the press freedom monitor with additional languages.