M 2 C : Towards Automatic Multimodal Manga Complement (2023)

Abstract: Multimodal manga analysis focuses on enhancing manga understanding with visual and textual features, which has attracted considerable attention from both natural language processing and computer vision communities. Currently, most comics are hand-drawn and prone to problems such as missing pages, text contamination, and text aging, resulting in missing comic text content and seriously hindering human comprehension. In other words, the Multimodal Manga Complement ( M2C ) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for vision and language understanding. To this end, we first propose the Multimodal Manga Complement task by establishing a new M2C benchmark dataset covering two languages. First, we design a manga argumentation method called MCoT to mine event knowledge in comics with large language models. Then, an effective baseline FVP-M ^{2} using fine-grained visual prompts is proposed to support manga complement. Extensive experimental results show the effectiveness of FVP-M ^{2} method for Multimodal Mange Complement.

## introduction comics are enjoyed globally, often featuring a combination of text and illustrations. however, due to their hand-drawn nature, they are susceptible to damage during circulation, such as missing pages, text contamination, and aging. to address these issues, we propose a task called multimodal manga complement (m2c), illustrated in fig. 1 . this task extends conventional text-based content complement by incorporating corresponding images as additional inputs to mitigate data sparsity and ambiguity @xcite . similar to other multimodal manga tasks (e.g., multimodal manga text why are you just so beautiful? vision generation figure 1: motivation of our m2c task. the left part describes both visual and language modalities as the input (vision part depicts a sequence of images from a page of comics, and the language part is the corresponding text dialogues). the right part displays a page of comics and the text that we want to complete. translation @xcite and comic image inpainting @xcite ), m2c aims to exploit the effectiveness of visual features for manga complement. furthermore, m2c can serve as a foundation for other comic-related tasks. however, as we can see in fig. 1 , compared to image caption @xcite or other multimodal tasks, the comic data we use has its own characteristics. here we conclude the differences and corresponding challenges: i). comics have shorter conversational text lengths than other natural language forms, and contain many inflections or auxiliaries. this leads to a greater randomness in content generation that requires complement. ii). the text and images of comics are complementary, often in pairs. this requires us to design better ways to exploit image features. iii). comic images and text contain timelines and implicit logic information, which shows us a challenge to leverage the logical knowledge to help improve the stability of generating complementary content. therefore, in our work, we first propose the multimodal manga complement (m2c) task to achieve the complement for with both vision and language information. to eliminate the above limitations, inspired by recent cots @xcite , we first design the three-hop manga chain-of-thought (mcot) argumentation method to exploit timelines and logic knowledge in comics with large language models @xcite . it contains a threehop inferring stage (theme, opinion, and future). then we propose a simple and effective fvp-m 2 method, including feature encoding, fine-grained visual prompt generation, and dialogue complement. specifically, in the token encoding stage, we use the pre-trained vision encoder to extract the visual tokens. then, we follow @xcite to utilize the transformer to encode the textual tokens. in fvpg, inspired by @xcite and @xcite , we adopt a new way to aggregate global and local visual information. specifically, the local visual features are aggregated in a self-attention way. while a mapping network in fig. 4 is leveraged to generate the global visual feature, which will serve as a bias term added into the calculation of local visual features. further, the outputs are the fine-grained visual prompts. after that, during the dialogue complement, following the works (e.g., vilbert @xcite )), we utilize co-transformer to generate the vision-guided language tokens. then the transformer decoder is adopted to predict the results. the contributions are summarized as follows: vision-language models. the success of visionlanguage models can be attributed to the transformer architecture @xcite @xcite @xcite @xcite en: ## datasets we introduce the multimodal manga complement (m2c) benchmark dataset. here, we described the details of the m2c dataset. ## conclusion we first propose the multimodal manga complement (m2c) task to handle issues such as missing pages, text contamination, and text aging in hand-drawn comics by providing a shared semantic space. we establish a new m2c benchmark dataset covering two languages. then, we propose an effective fvp-m 2 method for the m2c task. experimental results on m2c benchmark datasets show the effect of fvp-m 2 method.