MMEKG : Multi-modal Event Knowledge Graph towards Universal Representation across Modalities (2022)

Abstract: Events are fundamental building blocks of real-world happenings. In this paper, we present a large-scale, multi-modal event knowledge graph named MMEKG. MMEKG unifies different modalities of knowledge via events, which complement and disambiguate each other. Specifically, MMEKG incorporates (i) over 990 thousand concept events with 644 relation types to cover most types of happenings, and (ii) over 863 million instance events connected through 934 million relations, which provide rich contextual information in texts and/or images. To collect billion-scale instance events and relations among them, we additionally develop an efficient yet effective pipeline for textual/visual knowledge extraction system. We also develop an induction strategy to create million-scale concept events and a schema organizing all events and relations in MMEKG. To this end, we also provide a pipeline enabling our system to seamlessly parse texts/images to event graphs and to retrieve multi-modal knowledge at both concept- and instance-levels.

## introduction recently, many knowledge graphs (kgs) have been curated (e.g., wikidata @xcite ) and successfully applied to various applications, ranging from information extraction @xcite to information retrieval @xcite . kgs typically store billions of world facts in a directed graph, where nodes denote entities and edges denote their relations. although simple yet effective, the expression ability of such entity-centric kgs is limited @xcite . how we can represent more complex knowledge, such as events, situations, or different modalities, becomes a key question for broader applications. in this paper, we present a large-scale multi-modal event knowledge graph (mmekg) that bridges, complements, and disambiguates different modalities of knowledge, for better understanding or reasoning. similar to real-world happenings, mmekg takes events as its basic building blocks. each event is defined by a concept, several arguments, and corresponding roles. among events are various types of relations, such as causal, temporal, or sub-event relations. thus eneities can be arguments in kgs. figure 1 shows two example events: a visual sleep event with arguments cat (sleeper) and sofa (place), and a textual dressing event with arguments cat (wearer) and scarf (clothing), where argument roles are in brackets. the two events not only bridge the text and image with complementary arguments but also offer underlying commonsense knowledge -covering with a scarf usually happens when sleeping. compared with existing event kgs @xcite @xcite , mmekg advances this field in the following three aspects: (1) a large-scale ontology contains 990 thousand concept events and 644 relation types, which covers most types of real-world happenings. there are mainly two steps to build mmekg. (1) to construct a schema and acquire concept events, we first manually combine framenet @xcite and wordnet @xcite to initialize a high-quality event ontology; we then expand it automatically via ontology induction from instance events. for flexibility and exchangeability, we extend the simple event model (sem) @xcite to define our ontology in resource description framework (rdf). (2) to extract instance events from either texts or images, we developed a knowledge extraction system to support fast and massive extraction under the practical scenario. this system consists of event extraction and event relation extraction in both modalities, as well as the alignment between them. in addition, this system can parse any input texts/images to event graphs and seamlessly retrieve multi-modal knowledge from mmekg. to cover a variety of events, we apply our extraction system into multiple sources, including c4 news foot_0 , wikipedia foot_1 , bookcorpus foot_2 , and cc3m&12m @xcite . these data sources result in 863 million instance events and 934 million relations. to ensure its quality, we evaluate both our extraction system and the constructed mmekg. compared with state-of-the-art models of each sub-tasks, our methods achieve comparable or better performance on standard benchmarks. the adaptation to practical corpus led to no significant degradation. we sample thousands of events and relations from mmekg for manual evaluation. the precision is acceptable at both concept and instance levels. ## cross-modal event ontology ontology is critical because it not only confines what types of knowledge are concerned but also offers a reasoning ability -only the induction from instances to concepts brings new knowledge, i.e., from the special to the general. the deduction from concepts to instances has no uncertainty but provides additional information. in this section, we introduce our rdf schema to model ontology data (section 3.1), an initial ontology by combining external resources (section 3.2), and ontology induction for continuous expansion (section 3.3). ## knowledge extraction system this section briefly introduces our knowledge extraction system collecting large-scale instance events and relations for mmekg, which is shown in figure 5 . we follow the overall framework of previous knowledge extraction systems like gaia @xcite and resin @xcite , but extends and optimizes event-related components to enable it extracting billion-scale, high-quality events efficiently. with more advanced models, tuning strategy and component architectures, our system achieves comparable if not better performance on each component using a common benchmark. we also substitute all crossencoder in the system to bi-encoder if possible and conduct a joint model of multi-task training during event relation extraction for efficiency. ## related work event knowledge graph existing event knowledge graphs @xcite @xcite ) usually face a dilemma about quality and quantity. atomic @xcite annotates manually and constructs high-quality taxonomy induction: positive: eat.v.01__ingestibles:apple negative: eat.v.01__ingestibles:table relation induction: positive: ( destroy.v.02__patient:building, include, kill.v.01___victim:people ) negative: ( destroy.v.02__patient:building, include, invalidate.v.01__phenomenon:subpoena ) figure 6: examples of pair constructed for taxonomy (top) and relation induction (bottom). each pair includes one positive and one negative sample. positive ones are sampled from induced concept events or relations. negative ones are generated by substituting the arguments (taxonomy) or tail events (relation) in positive samples. knowledge extraction system previous multimodal knowledge extraction systems, such as gaia @xcite and resin @xcite , jointly extract information of a small domain from relatively small-scale resource. our system inherits their overall framework but is applied for extracting billion-scale and universal events. therefore we optimize event-related modules targetedly for both efficiency and effectiveness. cross-media event argument alignment some previous works @xcite also bridge texts and images through fine-grained alignments of event arguments for various tasks, such as multi-modal event extraction and fake news detection. instead, we fuse knowledge from different modalities to construct such a large-scale kg. ## conclusion we present the first multi-modal event kg (mmekg) with a large-scale event ontology. it not only bridges and complements different modalities of knowledge via more expressive events but also benefits comprehensive reasoning with rich cross-modal contexts. additionally, we provide a demo system that can seamlessly parse and link any texts/images via our knowledge extraction system.