Towards Artwork Explanation in Large-scale Vision Language Models (2024)

Abstract: Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension. However, it has not been clarified to what extent LVLMs understand the knowledge necessary for explaining images, the complex relationships between various pieces of knowledge, and how they integrate these understandings into their explanations. To address this issue, we propose a new task: the artwork explanation generation task, along with its evaluation dataset and metric for quantitatively assessing the understanding and utilization of knowledge about artworks. This task is apt for image description based on the premise that LVLMs are expected to have pre-existing knowledge of artworks, which are often subjects of wide recognition and documented information.It consists of two parts: generating explanations from both images and titles of artworks, and generating explanations using only images, thus evaluating the LVLMs’ language-based and vision-based knowledge.Alongside, we release a training dataset for LVLMs to learn explanations that incorporate knowledge about artworks.Our findings indicate that LVLMs not only struggle with integrating language and visual information but also exhibit a more pronounced limitation in acquiring knowledge from images alone. The datasets ExpArt=Explain Artworks are available at https://huggingface.co/datasets/naist-nlp/ExpArt

## introduction in the field of vision & language (v&l), large language models (llms) @xcite @xcite have been combined with visual encoders to create large scale vision language models (lvlms) @xcite @xcite . these models have achieved success in various v&l benchmarks (li lvlm mona lisa sfumato leonardo da vinci renaissance portrait 16 th century perspective van gogh ## lvlms lvlms @xcite @xcite integrate a vision encoder @xcite trained through contrastive learning to process visual information with large language models (llms) @xcite @xcite . this integration requires further training to effectively combine vision and language capabilities. as a result, these lvlms outperform conventional pretrained models, even those with over ten times more parameters @xcite . however, it is unclear whether the knowledge from the llm and the vision encoder are appropriately aligned by the additional network layers in lvlms @xcite . generating explanations that involve knowledge about art especially requires careful and systematic alignment and utilization of the information from both the vision encoder and the llm. this challenge motivates us to design a new task for lvlms. with title in the context of creative assistance, the title often contains the author's intent for the artwork, and it is desirable to generate explanations considering this intent. in this setting, both the image and its title are inputs, testing whether lvlms can generate appropriate explanations based on both language and visual information. without title as shown in figure 1 , there are cases where a title does not exist potentially because the artwork is in the process of creation. this setting tests whether lvlms can generate appropriate explanations using only visual information from images. additionally, analyzing the performance changes with and without titles allows us to verify the lvlms' pure vision-based knowledge. furthermore, to thoroughly assess the generalization capabilities of lvlms, we compare two cases: 1) a seen case in which images are observed during finetuning, and 2) an unseen case in which images are not observed during finetuning. ## dataset creation the process of dataset creation, illustrated in figure 2 , involved the following steps: step 1: we collected all the artwork articles from the english wikipedia that have an infobox (about 10,000), divided them into sections, and created descriptive texts. additionally, hyperlinked texts within the articles were extracted as entities related to the artwork. each descriptive text is accompanied by four pieces of information: the title, the hierarchy of sections (i.e., section, subsection, subsubsection), the image, and the aforementioned entities. step 2: we filtered out sections that did not contribute directly to the understanding of artwork, articles without images, and texts not specific to individual art pieces to ensure the relevance and quality of the content. step 3: to prevent biases that may arise due to the notoriety of the artworks included in the lvlm's training data, we shuffled the data. first, we ranked the data using six metrics: page views, number of links, number of edits, number of references, number of language versions, and article length. we then evenly split the data into test, development, and training sets at a ratio of 1:1:8 to maintain the average ranking across these sets ( step 4: the sorted data for each set were then formatted into instructions using the templates described in section 3.1. to diversify the training data, we prepared seven different templates inspired by @xcite (see appendix e.3). ## conclusion we introduced a new task, artwork explanation generation, and its dataset and metrics to quantitatively evaluate the artistic knowledge comprehension and application. using lvlms, we assessed their retention and utilization of artworks knowledge from base llms, with or without artwork titles. our findings indicate that while lvlms maintain much of the artistic knowledge from their llm counterparts, they do slightly lose some in practice. furthermore, the challenges in generating text solely based on visual inputs clearly show a significant dependency on text-based cues.