Monitio - Large Scale MT for Multilingual Media Monitoring (2022)

Abstract: Monitio is a real-time crosslingual global media monitoring platform which delivers actionable insights beyond human scale and capabilities. Our system continuously ingests a massive number of multilingual data sources that are automatically translated, filtered and categorized to generate intelligence reports specially geared towards media monitoring professionals’ needs.

## origin the starting point of monitio was a multilingual media monitoring prototype developed between 2016-2019 in tight collaboration with the british broadcast corporation (bbc) and deutsche welle (dw). @xcite both broadcasters monitor a growing number of video streams in different languages, by assigning teams of human analysts that are grouped by languages. this approach is not scalable hence the need to reinvent media monitoring, tackling it globally and in a scalable fashion to break the current internationalization and scalability barriers. the emergence of mature natural language processing (nlp) and artificial intelligence technologies gives european companies an opportunity to push europe to the leadership of the media monitoring market, where multilinguality is a major issue and, simultaneously, a major opportunity. ## challenges a major challenge for monitio is the huge volume of data ingested daily into the platform. one of monitio's goals is to process 10 million new multilingual items per day. this has to be accomplished with a minimum delay to enable near real-time monitoring. the automatic translation of this amount of documents is just one of the challenges for the platform's enrichment pipeline. all content entering the platform is subject to a series of nlp steps, namely its classification according to a standard topics taxonomy, the recognition of named entities like people, organizations, brands and places and linking them to external knowledge bases like wikipedia or wikidata, the production of a summary, and the clustering of all articles related to the same event in storylines. adding to the complexity, the large-scale dissemination of content on social media platforms, while ensuring a broad coverage of multiple connected viewpoints on the same subject of interest, stresses the problem of information verification which is a daunting task without the support of automation. failing to address this problem leads to biased views on the subject and insufficient (or even wrong) insights. ## solution the scalability goals but also the cost implications led to the integration of free/open source mt models like easynmt from hugging face, deployed in a dedicated gpu infrastructure. by employing mt in the beginning of the pipeline, monitio enables indexing of the documents in all languages translated to a language users understand (e.g., english), thus allowing the users to search documents in other languages. on the other hand, monitio does not employ mt before the nlp steps, which are executed in the source language of the documents to minimize error propagation. to enable processing and organization of multilingual documents in different stories and topics, monitio employs transfer learning through contextual multilingual distilbert sentence transformers foot_0 for crosslingual document clustering, topic detection and entity linking. one nlp task which still relies heavily on language specific annotated corpora is namedentity recognition, which is one of the most difficult tasks to train generalized multilingual models. ## future when a translation of better quality is needed, for instance, to be included in a report or to clear any doubt that may arise from the default mt, the user will be able to invoke third-party services on demand for a specific document. monitio will also integrate automatic speech recognition combined with mt to transcribe and translate video and audio content using wav2vec, an end-to-end deep learning model. another objective of the monitio project is the creation of innovative tools for assisted fact checking. we are developing tools that help the users to verify a claim using the multilingual information available in the platform.