Employing Glyphic Information for C hinese Event Extraction with Vision-Language Model (2024)

Abstract: As a complex task that requires rich information input, features from various aspects have been utilized in event extraction. However, most of the previous works ignored the value of glyph, which could contain enriched semantic information and can not be fully expressed by the pre-trained embedding in hieroglyphic languages like Chinese. We argue that, compared with combining the sophisticated textual features, glyphic information from visual modality could provide us with extra and straight semantic information in extracting events. Motivated by this, we propose a glyphic multi-modal Chinese event extraction model with hieroglyphic images to capture the intra- and inter-character morphological structure from the sequence. Extensive experiments build a new state-of-the-art performance in the ACE2005 Chinese and KBP Eval 2017 dataset, which underscores the effectiveness of our proposed glyphic event extraction model, and more importantly, the glyphic feature can be obtained at nearly zero cost.

## introduction event extraction aims to extract events from the sentence, each of which consists of four types of elements: a trigger and multiple arguments are exist as the raw spans in the input text, an event type or role type are assigned to corresponding trigger and argument as a result of classification. the example in figure 1 contains an event record: an meet event triggered by "讲话"(speech), the corresponding person argument is "总理"(prime minister) and "民众"(people), and place argument is "受灾山区"(disaster-stricken mountainous area). recent studies on event extraction have incorporated a variety of features, such as textual elements @xcite , extra annotations @xcite , and multi-modal components @xcite . nevertheless, research on chinese event extraction remains sparse. the majority of these studies tend to directly implement english event extraction techniques on chinese datasets @xcite , while only a limited number of works have tailored their methodologies based on the inherent traits of the chinese language @xcite @xcite . despite their effectiveness, previous works with sophisticated feature have encountered high annotation costs and narrow application scopes, making them less than optimal for chinese event extraction. in this study, we shift our attention to a long-existing yet often neglected feature: glyphs. chinese, as a hieroglyphic language, embeds substantial information within the glyphs of its characters, which is pivotal for chinese event extraction. to illustrate, the radical glyph plays a critical role in communicating the semantic essence of the trigger phrase "讲话" (speech). the radical "讠", signifying speech, is present in both characters, emphasizing its connection to the act of speaking. furthermore, the shape of the first character "山" in "受灾山区" directly evolved from the actual silhouette of a mountain (a shape glyph). this intuitive glyphic representation facilitates the straightforward extraction and classification of "受灾山 区" as a place argument. however, it is challenging to incorporate glyphic information into chinese event extraction tasks. this difficulty arises because it is unclear how glyphs impact event triggers or arguments along with their connections, and we also lack effective methods for incorporating glyphs into downstream tasks such as event extraction. the straightforward adoption of the efforts in pretraining from previous works @xcite are not applicable since their ways of splitting sequence into characters and radicals to align with the tokenized sequence are hard to capture the semantic connection across words in the sentence, which could be the crucial for downstream tasks. in this study, we utilize glyphic images at sentence-level as an alternative to radical or character information for capturing glyph details. as illustrated in figure 2 , we transform the character sequence of a sentence directly into a glyphic image with active visual emphasises and leverage this image for chinese event extraction. this approach is distinct from splitting into radicals or characters, providing a comprehensive representation of the sentence, enabling the model to perceive glyphic features through a high-level visual perspective, enhancing the extraction process. furthermore, we adopt a vision-language model (vlm) integrated with two modality alignment methods to decipher the interplay between the input sentence and the glyphic image. this integration enables the model to bridge the gap between character sequence and glyphic image, and learn the interaction between them. the detailed evaluation shows that our proposed model significantly advances the state-of-the-art performance on several benchmarks, indicating that the glyphic information can be obtained to enhance chinese event extraction at nearly zero cost. ## related works in this section, we introduce two related topics: event extraction and applications of glyphic information. ## chinese event extraction via glyphic vision-language model in this study, we utilize a glyphic vision-language model specifically designed for chinese event extraction. as shown in figure 2 , our approach involves several key steps. firstly, we convert the input sentence into a glyphic image using a visual emphasis construction method. secondly, we employ a vision-language model to learn the interactions between the input sentence and the glyphic image. finally, we generate the event structure based on two fusion strategies. in the below of these section, we will discuss these issues one by one. ## experiment in this section, we introduce the datasets used for evaluation and the baseline methods employed for comparison. we then report the experimental results conducted from different perspectives, and analyze the effectiveness of the proposed model with different factors. ## analysis and discussion in this section, we give some analysis and discussion to show the effectiveness of proposed glyphic vision-language model.