Exploring the Capability of Multimodal LLM s with Yonkoma Manga: The YM anga Dataset and Its Challenging Tasks (2024)

Abstract: Yonkoma Manga, characterized by its four-panel structure, presents unique challenges due to its rich contextual information and strong sequential features. To address the limitations of current multimodal large language models (MLLMs) in understanding this type of data, we create a novel dataset named YManga from the Internet. After filtering out low-quality content, we collect a dataset of 1,015 yonkoma strips, containing 10,150 human annotations. We then define three challenging tasks for this dataset: panel sequence detection, generation of the author’s creative intention, and description generation for masked panels. These tasks progressively introduce the complexity of understanding and utilizing such image-text data. To the best of our knowledge, YManga is the first dataset specifically designed for yonkoma manga strips understanding. Extensive experiments conducted on this dataset reveal significant challenges faced by current multimodal large language models. Our results show a substantial performance gap between models and humans across all three tasks.

## introduction yonkoma manga, it will be referred to as yonkoma below for the sake of simplicity, also known as 4koma manga, which is a very regular comic data, originated in japan and consists of four panels of equal size. each panel typically follows this structure: task2: intention generation task3: description generation human-authored: by personifying the meteoroid, the author praises those who are able to make great sacrifices for their ideals. ## dataset and task setups we collect yonkoma strips from several influential comic websites, such as pinterest, gocomics, and from the personal pages of various comic artists. through a rigorous process of both machine and manual filtering, we retain 1,015 high-quality yonkoma strips. all of these have been authorized by their respective creators. for detailed information on our criteria for filtering data, please refer to appendix a. ## experiments llava-next(with_description): character b's son is crying. llava-next(without_description): the father asks if the child has taken their medication, to which the child responds that they haven't. true description: character b's son is taking the medicine. llava-next(with_description): character b's son happily eats the medicine. llava-next(without_description): the child reluctantly eats the ice cream. trained to align images and captions in the we-bimagetext corpus (400m pairs) @xcite , as a baseline model for psd, considering only the image information of yonkoma. specifically, we take the pooling output of clip-vit and subsequently connect an multilayer perceptron @xcite to map it to the binary classification of whether the sequence of yonkoma panels is correct. instructblip. instructblip @xcite consists of a vision transformer @xcite , a q-former module, and a large language model. it has undergone vision-language instruction fine-tuning on the pre-trained blip-2 @xcite model. we choose the instructblip-flan-t5-xxl(12b parameters) model which is followed by a flan-t5-xxl @xcite . for task psd, we use its vision transformer and q-former module to encode the image-text data of yonkoma, obtain its pooling output, and then connect an multilayer perceptron @xcite to map it to the binary classification of whether the sequence of the yonkoma panels is correct. for tasks 2 and 3, we fine-tune instructblip-flan-t5-xxl using low-rank adaptation @xcite . llava-next. llava-next @xcite is a multimodal large language model improved upon llava @xcite . it has enhanced reasoning abilities, optical character recognition (ocr), and world knowledge, outperforming many multimodal llms on various general visual language tasks, including gemini pro @xcite . we choose llava-v1.6-mistral-7b-hf(7b parameters), an open-source model available on hugging face, as the baseline for our last two generation tasks. we also use low-rank adaptation @xcite to fine-tune it. gpt-4o. we also test our three tasks on the gpt-4o (openai, 2024a,b). we design detailed prompts for these three tasks. for task psd, we set the model to output "yes" or "no" to achieve the effect of binary classification. for the next two generative tasks, we design prompts for the presence or absence of panel descriptions. see appendix c for more prompt details. ## related work research on comics analysis. comics, as a highly comprehensive form of multimodal data, have attracted increasing attention from researchers in recent years. however, existing research on comics mainly focuses on specific aspects. for instance, @xcite explore dialogue recognition in images through comics and introduced a dataset containing 1,438 annotated panels. @xcite propose a hierarchical method to study the reading order of comic text bubbles. hinami et al. (2021 2017 ) presented a comic dataset called comics and introduced three cloze-style tasks, requiring the model to predict the narrative of a panel given the context of the previous n panels. @xcite construct a new japanese comic onomatopoeia dataset called coo, which challenges the recognition of irregular text. although there is a variety of work on comic data, these studies focus on specific features of comics and fail to fully leverage the comprehensive nature of comic data. moreover, these works are not suitable for testing mllms, which possess strong multimodal capabilities. the development of multimodal llms. the successful application of large language models has promoted the development of research in the multimodal field and paved the way for the construction of multimodal large language models. there are three main methods for building multimodal large-scale language-based models, each aiming to achieve strong zero-shot generalization capabilities in the field of visual language. for example, flamingo @xcite is a pioneer in this field, using frozen visual encoders and large language models equipped with gated cross-attention for cross-modal alignment. this method has been widely adopted by models such as llava @xcite and shikra @xcite . however, a significant limitation of this approach is the creation of lengthy visual sequences. to address this issue, blip-2 @xcite drew inspiration from detr @xcite and developed a q-former to effectively reduce the sequence length of visual features. kosmos-1 @xcite , mplug-owl2 @xcite , and minigpt-4 @xcite all adopt this design to reduce the visual sequences. despite the rapid development of mllms, our experiments show that these models have limitations in handling complex image-text modal reasoning problems. ## conclusion and future work to explore the limitations of mllms, we introduce the ymanga dataset, which consists of 1,015 high-quality yonkoma manga samples with corresponding human annotations. we designed three tasks based on ymanga: panel sequence detection (psd), intent generation (ig), and description generation for masked panels (dg). through rigorous baseline experiments and in-depth analysis of the results, we identified three main limitations of mllms: insufficient ability to capture fine-grained image details, a tendency to hallucinate when handling complex image data, and inadequate generative capabilities in open-domain tasks. our future work will build on the ymanga dataset to further explore ways to mitigate the three limitations of mllms. this will primarily focus on two areas: first, designing more efficient methods for modal alignment to capture more image details, and second, improving the generative quality of the models by incorporating external knowledge bases or leveraging techniques such as reinforcement learning.