VKIE : The Application of Key Information Extraction on Video Text (2023)

Abstract: Extracting structured information from videos is critical for numerous downstream applications in the industry. In this paper, we define a significant task of extracting hierarchical key information from visual texts on videos. To fulfill this task, we decouple it into four subtasks and introduce two implementation solutions called PipVKIE and UniVKIE. PipVKIE sequentially completes the four subtasks in continuous stages, while UniVKIE is improved by unifying all the subtasks into one backbone. Both PipVKIE and UniVKIE leverage multimodal information from vision, text, and coordinates for feature representation. Extensive experiments on one well-defined dataset demonstrate that our solutions can achieve remarkable performance and efficient inference speed.

## introduction extracting information from video text is an essential task for many industrial video applications, i.e., video retrieval @xcite , video recommendation @xcite , video indexing @xcite , etc. visual text embedded in videos usually carries rich semantic descriptions about the video contents, and this information gives a highlevel index for content-based video indexing and browsing. conventional methods utilize ocr @xcite @xcite to extract visual texts from videos frames and employ text classification techniques @xcite to categorize the extracted content. however, these methods suffer from two significant shortcomings: 1) visual texts are typically coarse-grained at the segment level, and are unable to capture finegrained information at the entity level, which is critical for downstream tasks. 2) traditional methods have not fully utilized the fusion of features from different modalities. therefore, in our work, we introduce a novel industrial task for extracting key information from video text and exploring the relationship between entities, which we refer to as vkie. the task aims to extract valuable hierarchical information from visual texts, explore their relationships, and organize them in structured forms. this approach enables effective management and organization of videos through the use of rich hierarchical tags, which can be utilized to index, organize, and search videos at different levels. figure 1 provides an example of the hierarchical key information extracted by vkie, where subtitles are captured at the segment level, and personal information is organized with names and identities at the entity level. to enhance clarity, we decompose vkie into four subtasks: text detection and recognition (tdr), box text classification (btc), entity recognition (er), and entity linking (el). while the first subtask, tdr, is typically accomplished using offthe-shelf ocr tools, our work concentrates on the remaining three subtasks of btc, er, and el. since tdr outputs all boxes with text content and coordinates information, there are massive useless texts, such as scrolling texts and blurred background texts, which could have side effects on downstream tasks. btc aims to eliminate these useless texts and find valuable categories, such as title, subtitle, and personal information. although the btc method can obtain segment level information, the results are relatively coarsegrained and will limit its deployment to many downstream applications. for example, in videotext retrieval, the query is usually in different forms, such as keywords, phrases, or sentences. in video indexing, a video is required to be stored with hierarchical tags. to address these issues, we designed er to extract entities from text segments and el to explore the relations among the entities. with this structured information, videos can be well managed with rich hierarchical information at the entity and segment levels. in this paper, we present two solutions that have been deployed in our industry system. the first approach, called pipvkie, involves performing the tasks sequentially, which serves as our baseline method. the second approach, called univkie, achieves better performance and efficiency by more effectively integrating multimodal features. in summary, our contributions are as follows: ## conclusion this paper introduces a novel task in the industry, referred to as vkie, which aims to extract crucial information from visual texts in videos. to address the task, we decouple vkie into four subtasks: text detection and recognition, text classification, entity recognition, and relation extraction. furthermore, we propose two complete solutions utilizing multimodal information: pipvkie and univkie. pipvkie performs these three subtasks in different stages, while univkie unifies all of them in one model with higher efficiency and lower resource cost. experimental results on one well-defined dataset demonstrate that our solutions can achieve remarkable performance and satisfactory resource cost. with vkie, structured information could be effectively extracted and well organized with rich semantic information. vkie has been deployed on an industrial ai platform.