Event Extraction in Video Transcripts (2022)

Abstract: Event extraction (EE) is one of the fundamental tasks for information extraction whose goal is to identify mentions of events and their participants in text. Due to its importance, different methods and datasets have been introduced for EE. However, existing EE datasets are limited to formally written documents such as news articles or scientific papers. As such, the challenges of EE in informal and noisy texts are not adequately studied. In particular, video transcripts constitute an important domain that can benefit tremendously from EE systems (e.g., video retrieval), but has not been studied in EE literature due to the lack of necessary datasets. To address this limitation, we propose the first large-scale EE dataset obtained for transcripts of streamed videos on the video hosting platform Behance to promote future research in this area. In addition, we extensively evaluate existing state-of-the-art EE methods on our new dataset. We demonstrate that such systems cannot achieve adequate performance on the proposed dataset, revealing challenges and opportunities for further research effort.

## introduction event extraction is an important task in the full pipeline of information extraction. in ee, the goal is to identify mentions/trigger words of events, and their participants and attributes of interest. for instance, in the sentence "joe biden was born on november 20, 1942", an event of birth is mentioned. an event mention consists of two important components: (1) trigger: the word(s) that most clearly refer to the occurrence of the event (e.g, "born" in the above example); and (2) argument: the entity mentions involved in the event with some role (e.g., "joe biden" with the role of entity) or attributes of the event (e.g., time and location). due to its importance, various methods and annotated datasets have been proposed for ee @xcite @xcite . also, with the proliferation of ee methods, datasets for ee have been diversified to cover different domains and settings, e.g., multiple domains @xcite , multiple languages, @xcite , literary texts @xcite , cybersecurity texts @xcite , and events in long documents @xcite . however, despite all progress thus far, most of the available datasets for ee are restricted to the domains of formally written texts, e.g., news, reports, scientific papers, or books. as such, the challenges for ee in other domains with informal and noisy texts are less explored. one of such domains that has not been studied before for ee involves video transcripts obtained by automatic speech recognition (asr) tools. since the such transcripts might be noisy, e.g., incomplete sentences, incorrect words selected by the asr tool, lack of correct punctuation and segmentation, repeated words or sentences, etc., existing ee models are not well evaluated and might not perform well in this domain. this is unfortunate as an effective ee model can be extremely helpful for downstream applications that utilize video transcripts. for instance, search engines can employ events detected in a transcript to locate relevant portion of a video to a query. it can also benefit video summarization, knowledge base construction, and script generation from videos. as such, it is necessary to study the challenges and potential directions for ee improvement in the video transcript domain. due to the lack of ee datasets for video transcript domain, we propose the first large-scale ee dataset annotated for transcripts of streamed videos on the popular video hosting platform behance. videos in this platform are streamed by artists who would like to share their creative projects using adobe creative cloud products (e.g., photoshop, illustrator, etc). videos have been first transcribed using the microsoft automatic speech recognition tool. in order to annotate the events mentioned in the video transcripts, we first define a taxonomy of event types and their arguments for the domain of creative projects (e.g., add a shape, modify the color of an object, etc.). using the pre-processed transcripts and the provided event ontology, we hire annotators with domain expertise to provide high-quality annotation for event triggers and their arguments. in addition, we employ the annotated dataset to evaluate the performance of the stateof-the-art (sota) ee models. compared to the domains with formally written texts, our analysis shows that the current sota ee models fail to achieve comparable performance on video transcripts. this performance drop indicates the challenging nature of video transcripts and call for more research effort for ee in this domain. we will publicly release our dataset, called transcriptee, to foster future research in this area. we employ human annotation to find event mentions in the paragraphs of video transcripts. to annotate event triggers, we follow prior work on ed @xcite to ask the annotators to select the word or phrases (e.g., a phrasal verb) that most clearly mention the occurrences of events. also, for event arguments, we require the annotators to select the head words of the noun phrases that refer to the arguments of events. event triggers and arguments can belong to different sentences in the paragraphs in our dataset. in this work, we leverage upwork, a freelancer platform, to hire expert annotators. the hired annotators have experience in both creative projects (e.g., using photo editing programs such as photoshop) and data annotation. we train the annotators with the event ontology and designed examples. based on the performance of the annotators in a pilot study, we select the final pool of five annotators. to compute the inter-annotation agreement (iaa) scores, 20% of the paragraphs are shared by the annotators for co-annotation while the remaining 80% is distributed evenly for the annotators. as such, annotators first independently identify event triggers in the shared paragraphs, achieving an agreement score of 0.812 for the krippendorff's alpha @xcite with masi distance metric @xcite . afterward, the annotators discuss to resolve conflict cases for the co-annotated data, and then perform annotation individually on the remaining data to produce the final version of event triggers in our dataset. in the next step, given the annotated triggers, annotators also independently annotate event arguments for the shared paragraphs, achieving the krippendorff's alpha of 0.783. finally, conflict argument examples in the co-annotated data are resolved and individual annotation on the rest of the data is done by the annotators to generate the final version of our dataset transcriptee. as such, we achieve strong agreement scores for both event trigger and argument annotation, showing the high quality of transcriptee. to facilitate future research, we randomly split the dataset into separate training, development, and test sets with the ratio of 80/10/10, respectively. the statistics for each data split along the entire dataset are presented in ## experiments we study the performance of existing state-ofthe-art ee systems on the proposed dataset tran-scriptee. since ee consists of two sub-tasks, i.e., event detection (ed) and event argument extraction (eae), we consider two types of baselines: performance of the pipeline and joint models is presented in tables 2 and 3 , respectively. the joint models outperform their counterpart pipeline systems. specifically, bert and bert+crf enjoy 2.73 and 2.99 f1 point improvement for ed, ## related works previous ee systems can be classified according to the representation construction methods, i.e., feature engineering @xcite @xcite vs. deep learning for representation learning @xcite @xcite @xcite , or the formulation approaches, i.e., pipeline @xcite @xcite vs. joint models @xcite @xcite @xcite . a majority of prior ee work utilize the ace 2005 @xcite and tac kbp datasets @xcite that focus on newswire article domains. recently, there have more efforts to create ee datasets for more diverse domains, including biomedical @xcite , literary @xcite , cybersecurity @xcite , wikipedia @xcite , multilingual @xcite , history @xcite , and suicide understanding @xcite . however, none of such prior work and datasets explores ee for video transcripts. ## conclusion we present transcriptee, the first manually annotated dataset for ee for video transcripts. the videos are obtained from the behance platform which is dedicated to sharing creative projects. transcriptee contains more than 2,000 labeled paragraphs for various edit events in creative projects with high quality. our analysis with stateof-the-art ee models reveals challenging nature of the dataset for future research.