GAIA : A Fine-grained Multimedia Knowledge Extraction System (2020)

Abstract: We present the first comprehensive, open source multimedia knowledge extraction system that takes a massive stream of unstructured, heterogeneous multimedia data from various sources and languages as input, and creates a coherent, structured knowledge base, indexing entities, relations, and events, following a rich, fine-grained ontology. Our system, GAIA, enables seamless search of complex graph queries, and retrieves multimedia evidence including text, images and videos. GAIA achieves top performance at the recent NIST TAC SM-KBP2019 evaluation. The system is publicly available at GitHub and DockerHub, with a narrated video that documents the system.

## introduction knowledge extraction (ke) aims to find entities, relations and events involving those entities from unstructured data, and link them to existing knowledge bases. open source ke tools are useful for many real-world applications including disaster monitoring @xcite , intelligence analysis @xcite and scientific knowledge mining @xcite . recent years have witnessed the great success and wide usage of open source natural language processing tools @xcite @xcite @xcite , but there is no comprehensive open source system for ke. we release a new comprehensive ke system, gaia, that advances the state of the art in two aspects: (1) it extracts and integrates knowledge across multiple languages and modalities, and (2) it classifies knowledge elements into fine-grained types, as shown in gaia has been inherently designed for multimedia, which is rapidly replacing text-only data in many domains. we extract complementary knowledge from text as well as related images or video frames, and integrate the knowledge across modalities. taking figure 1 as an example, the text entity extraction system extracts the nominal mention troops, but is unable to link or relate that due to a vague textual context. from the image, the entity linking system recognizes the flag as ukrainian and represents it as a nationalitycitizen relation in the knowledge base. it can be deduced, although not for sure, that the detected people are ukrainian. meanwhile, our cross-media fusion system grounds the troops to the people detected in the image. this establishes a connection between the knowledge figure 2 : user-facing views of knowledge networks constructed with events automatically extracted from multimedia multilingual news reports. we display the event arguments, type, summary, similar events, as well as visual knowledge extracted from the corresponding image and video. compared to coarse-grained event types of previous work @xcite , we follow a richer ontology to extract fine-grained types, which are crucial to scenario understanding and event prediction. for example, an event of type movement.transportperson involving an entity of type per.politician.headofgovernment differs in implications from the same event type involving a per.combatant.sniper entity (i.e., a political trip versus a military deployment). similarly, it is far more likely that an event of type conflict.attack.invade will lead to a contact.negotiate.meet event, while a conflict.attack.hanging event is more likely to be followed by an event of type contact.funeralvigil.meet. coarse-grained types fine-grained types entity 7 187 relation 23 61 event 47 144 the knowledge base extracted by gaia can support various applications, such as multimedia news event understanding and recommendation. we use russia-ukraine conflicts of 2014-2015 as a case study, and develop a knowledge exploration interface that recommends events related to the user's ongoing search based on previously-selected attribute values and dimensions of events being viewed foot_1 , as shown in figure 2 . thus, this system automatically provides the user with a more comprehensive exposure to collected events, their importance, and their interconnections. extensions of this system to real-time applications would be particularly useful for tracking current events, providing alerts, and predicting possible changes, as well as topics related to ongoing incidents. ## overview the architecture of our multimedia knowledge extraction system is illustrated in figure 3 . the system pipeline consists of a text knowledge extraction (tke) branch and a visual knowledge extraction (vke) branch (sections 3 and 4 respectively). each branch takes the same set of documents as input, and initially creates a separate knowledge base (kb) that encodes the information from its respec- textual relation extraction textual mention extraction textual event extraction elmo-lstm crf entity extractor attentive fine-grained entity typing textual entity coreference collective entity linking and nil clustering contextual nominal coreference dependency based fine-grained relation typing assembled cnn extractor textual event coreference coarse-grained event extraction bi-lstm crfs trigger extractor cnn argument extractor graph based coreference resolution textual kb background kb fine-grained event typing framenet & dependency based fine-grained event typing rule based fine-grained event typing visual entity coreference visual entity extraction faster r-cnn ensemble visual entity linking facenet landmark matching dbscan clustering generic features classactivation map model flag recognition face features heuristics rules visual kb cross-media fusion visual grounding multimedia kb cross-modal entity linking applications news recommendation multimedia news multi-lingual text content images and video key frames english russian ukrainian fusion and pruning mtcnn face detector tive modality. both output knowledge bases make use of the same types from the darpa aida ontology 8 , as referred to in ## text knowledge extraction as shown in figure 3 , the text knowledge extraction (tke) system extracts entities, relations, and events from input documents. then it clusters identical entities through entity linking and coreference, and clusters identical events using event coreference. ## visual knowledge extraction the visual knowledge extraction (vke) branch of gaia takes images and video key frames as input and creates a single, coherent (visual) knowledge base, relying on the same ontology as gaia's text knowledge extraction (tke) branch. similar to tke, the vke consists of entity extraction, linking, and coreference modules. our vke system also extracts some events and relations. ## related work existing knowledge extraction systems mainly focus on text @xcite @xcite @xcite @xcite , and do not readily support fine-grained knowledge extraction. visual knowledge extraction is typically limited to atomic concepts that have distinctive visual features of daily life @xcite @xcite @xcite , and so lacks more complex concepts, making extracted elements challenging to integrate with text. existing multimedia systems overlook the connections and distinctions between modalities @xcite . our system makes use of a multi-modal ontology with concepts from real-world, newsworthy topics, resulting in a rich cross-modal, as well as intra-modal connectivity. for our demo, the distinction between beneficial use and harmful use depends, in part, on the data. proper use of the technology requires that input documents/images are legally and ethically obtained. regulation and standards (e.g. gdpr foot_9 ) provide a legal framework for ensuring that such data is properly used and that any individual whose data is used has the right to request its removal. in the absence of such regulation, society relies on those who apply technology to ensure that data is used in an ethical way. even if the data itself is obtained legally and ethically, the technology when use for unintended purposes can result in harmful outcomes. this demo organizes multimedia information to aid in navigating and understanding international events described in multiple sources. we have also applied the underlying technology on data that would aid natural disaster relief efforts @xcite 19 and we are currently exploring the application of the models (with different data) to scientific literature and drug discovery @xcite 20 . one potential for harm could come if the technology were used for surveillance, especially in the context of targeting private citizens. advances in technology require increased care when balancing potential benefits that come from preventing harmful activities (e.g. preventing human trafficking, preventing terrorism) against the potential for harm, such as when surveillance is applied too broadly (e.g. limiting speech, targeting vulnerable groups) or when system error could lead to false accusations. an additional potential harm could come from the output of the system being used in ways that magnify the system errors or bias in its training data. our demo is intended for human interpretation. incorporating the system's output into an automatic decision-making system (forecasting, profiling, etc.) could be harmful. training and assessment data is often biased in ways that limit system accuracy on less well represented populations and in new domains, for example causing disparity of performance for different subpopulations based on ethnic, racial, gender, and other attributes @xcite . furthermore, trained systems degrade when used on new data that is distant from their training data. the performance of our system components as reported in to understand, organize, and recommend information, our system aggregates information about people as reported in its input sources. for example, in addition to external kb linking, the system performs coreference on named people and uses text-visual grounding to link images to the surrounding context. privacy concerns thus merit at-tention @xcite . the demo relies on publicly available, online sources released by the ldc 21 . when applying our system to other sources, care should be paid to privacy with respect to the intended application and the data that it uses. more generally, end-to-end algorithmic auditing should be conducted before the deployment of our software @xcite . a general approach to ensure proper, rather than malicious, application of dual-use technology should: incorporate ethics considerations as the first-order principles in every step of the system design, maintain a high degree of transparency and interpretability of data, algorithms, models, and functionality throughout the system, make software available as open source for public verification and auditing, and explore countermeasures to protect vulnerable groups. ## conclusion we demonstrate a state-of-the-art multimedia multilingual knowledge extraction and event recommendation system. this system enables the user to readily search a knowledge network of extracted, linked, and summarized complex events from multimedia, multilingual sources (e.g., text, images, videos, speech and ocr).