Caption Generation in Cultural Heritage: Crowdsourced Data and Tuning Multimodal Large Language Models (2025)

Abstract: Automated caption generation for paintings enables enhanced access and understanding of visual artworks. This work introduces a novel caption dataset, obtained by manual annotation of about 7500 images from the publicly available DEArt dataset for object detection and pose estimation. Our focus is on describing the visual scenes rather than the context or style of the artwork - more common in other existing captioning datasets. The dataset is the result of a crowdsourcing initiative spanning 13 months, with volunteers adhering to explicit captioning guidelines reflecting our requirements. We provide each artwork in the dataset with five captions, created independently by volunteers to ensure diversity of interpretation and increase the robustness of the captioning model. In addition, we explore using the crowdsourced dataset for fine-tuning Large Language Models with vision encoders for domain-specific caption generation. The goal is to improve the performance of multimodal LLMs in the context of cultural heritage, a domain with “small data” which often struggles with the nuanced visual analysis and interpretation required for cultural objects such as paintings. The use of crowdsourced data in the domain adaptation process enables us to incorporate the collective perceptual insights of diverse annotators, resulting in an exploration of visual narratives and observing a reduction in hallucinations otherwise produced by these large language models.

## introduction to offer innovative methods for engaging with and understanding visual artefacts at scale, many systems rely on rich metadata -for instance, in the form of captions or descriptions. having access to good captions of artworks not only facilitates broader public access to these artifacts but also fosters a deeper appreciation for their cultural significance. however, the automatic generation of captions is not without challenges. artworks often present scenes with intricate symbolism and complex narratives, where the most important elements can be hard to identify and demand nuanced caption beyond simple object recognition. in this paper we introduce a novel dataset of captions of the visual content of artworks and showcase how it can help in the domain adaptation of state-of-art approaches such as multimodal large language models (mllms) @xcite for the task of caption generation. the image dataset was sourced from the publicly available deart object detection and pose estimation dataset @xcite , a curated assemblage of paintings spanning diverse european cultures, centuries and artistic movements. our motivation for collecting this new dataset was twofold. first, good models rely on the existence of large amounts of quality data. for reasons that are (1) technical -small data with a large variety between the representation of objects -real or imaginary, depiction of actions usually not captured in photographs, etc -but also (2) the relatively low interest in cultural heritage -which results in limited effort and financing, there is still a considerable gap between how precise multimodal llms perform for photographs and artworks. this gap could be narrowed by new quality datasets. secondly, we chose to focus on the visual scene because we believe that it is necessary to identify all/most of the elements to be able to assign cultural meaning to a work; additionally, in those cases where a visual setup can consistently derive further meaning could be inferred more reliably top-down (from domain knowledge) rather than being generated based on a limited dataset. the model adaptation work was motivated by the experiments we ran that use mmmls to generate captions for cultural heritage (ch) artifacts; the results underlined some apparent shortcomings: on the one hand, content unrelated to the visual scene (i.e., mistaken identity for both objects and relationships between objects), and on the other, missing elements. our hypothesis was that these models could effectively leverage domain knowledge from datasets like ours, to overcome some of their apparent limitations. our crowdsourcing campaign was hosted on the zooniverse platform and involved volunteers from various backgrounds and expertise levels, who created detailed caption annotations for about 7500 deart paintings during a year-long period. given that a high percentage of the images are noniconic @xcite , gathering 5 different annotations per image allows for a diversity of perspectives and interpretations, which can make the trained model more robust. based on this data, we use parameter-efficient fine-tuning techniques @xcite and demonstrate the possibility of mitigating hallucinations in llm-generated captions. ## related work early efforts in image captioning, such as @xcite , laid the groundwork for later advancements. the distinctive challenges posed by cultural heritage artworks demand specialized solutions due to several important features not present in everyday pictures: anachronic objects, imaginary beings, actions not present in photographs -eg decapitations, etc. several works have made significant contributions in the area of captioning for cultural heritage, of which we briefly present those directly related to our task -visual content captioning. more recently, the artcap dataset @xcite provides 3,606 paintings, each annotated with five captions, showcasing high-quality annotations and effectiveness in benchmarking painting captioning models. the semart dataset @xcite , designed for semantic art understanding, includes fine-art paintings with attributes and textual artistic comments. it also introduces the text2art challenge, a multi-modal retrieval task linking artistic texts and paintings. the deart dataset @xcite focuses on object detection and pose estimation for 15k images of european artwork between the 12th and the 18th centuries. it includes 69 object classes, many of which are specific to cultural heritage, but does not include caption annotations. recognizing this gap and considering the rich variety of noniconic images in deart, we decided to leverage a subset to create a caption generation dataset. recent advances in the field of large language models (llms) (openai, 2023) have seen the successful integration of visual information into these models, giving rise to a new generation of mllms. notable among these is llava @xcite , which, along with other models such as mini-gpt4 (openai, 2023) and instruct-blip @xcite , have shown impressive image captioning and question-answering capabilities. like llms and unlike most of the artcap and semiart works, our approach relies on crowdsourcing data. this has the advantage of training the model with a variety of interpretations of paintings, coming from volunteers with different levels of expertise in cultural heritage. we believe that this can make the trained model more flexible and accurate. other works in metadata generation for cultural heritage exist, but they at least partly focus on the generation of style and context information (artwork's history, author's biography etc.), which introduces noise in the captions. to create effective guidelines 1 , we drew inspiration from established practices such as @xcite , and we discussed our proposal with several cultural heritage experts. after deciding to use zooniverse as a platform, we received expert advice from one of their shepherds. our guidelines emphasize the requirements of clarity, simplicity, and objectivity. we encourage annotators to start captions with the most crucial elements, progressing from foreground to background. we recommend avoiding assumptions, e.g., the identity of characters, events or places, assumptions about time periods (which, e.g. may bias the choice of object names), or professional jargon. the focus should always be on what is visually present in the image, avoiding implications or intentions. named entities should be identified, but only if they are clearly recognizable or convey important information. guidelines provide specific instructions for spatial orientation, using absolute positioning and limiting the use of "background/foreground" to essential details. they also advocate for concise annotations, restricting captions to 250 characters, while encouraging multiple sentences for clarity and simplicity. the language should be straightforward and avoid comparative constructs (e.g. larger, smallest), pronouns, and unnecessary punctuation. the annotation interface included examples to illustrate the preferred style and promote clear and informative annotations in a standardized manner. ## crowdsourcing process and preprocessing of the annotated data after a thorough assessment of various platforms, which included short tests for quality of annotations, we decided to use the zooniverse platform. zooniverse's established reputation in supporting citizen science projects with quality metadata, the possibility of hosting caption annotation tasks, and the reality that volunteers make for better annotators (possibly due to the inherent interest in the project), were decisive factors in our decision. due to gdpr and other law restrictions, zooniverse platform doesn't allow the collecting of data about volunteers. however, in march 2015, the zooniverse team conducted a survey to better understand their volunteer community. the survey, part of a 1 link to guidelines master's thesis by victoria homsy at oxford university, gathered responses from approximately 300 active participants. key findings revealed a gender distribution of 60% male and 40% female volunteers. age-wise, the community was diverse, with a slight underrepresentation of older individuals. geographically, the user base was primarily from english-speaking countries, notably the uk and the us, each contributing about a third of the participants, while only 2% hailed from developing nations. employment data indicated that around half of the volunteers were employed, 15% were retired, 10% unemployed, and 4% unable to work due to disability. the survey also highlighted a wide range of occupations among volunteers, including roles such as professor, administrator, guard, and various technical positions. the crowdsourcing campaign was initiated with the design of a user-friendly interface (ui) to facilitate efficient interaction between volunteers and the paintings. we ran the campaign in batches to try to get 4-5 good annotations per image for increasingly larger subsets of deart, while at the same time keeping a balance between diversity and thematic consistency. concretely, we included images with different styles and from different time periods, while excluding most portraits and other iconic images with limited interest from a captioning perspective (e.g.images that weren't iconographic or that had low expected variability for the captions). this process was iterative and involved: (1) the gradual decrease of the size of the batches to increase the motivation of the volunteers to complete the work, and (2) the adaptation of the image selection process to propose paintings of complexity that had lead to good captions in previous batches. the multiple captions generated for each painting by the different volunteers reflect diverse artistic interpretations and visual insights and thus help us train a more robust captioning model. at the end of each batch annotation process, we ran a data health check; rigorous quality control mechanisms were applied to manually verify the adherence of captions to guidelines and to maintain thematic alignment. corrections and clarifications were incorporated into our guidelines and user interface to enhance annotation accuracy. this iterative batch approach enabled us to capitalize on the collective contributions of volunteers while preserving dataset integrity. the total number of uploaded images was 7543. the dataset health check was based on several rules: following the dataset health check, 34535 captions were retained. our crowdsourced dataset stands out for its richness (i.e. number of annotations per images and total annotations) and diversity (i.e. different annotator views, given by the number of annotations per image) in comparison to the datasets that are the largest and most relevant in cultural heritage, as indicated by the metrics in while it contains fewer images than semart, our dataset offers an average of 4.57 captions per image. the higher caption diversity is crucial to train more nuanced models, as it reflects how a visual scene can be described differently -which increases the power of generalization. in contrast, semart provides only one caption per image (although of museum-expert quality), which may limit the range of insights available for each artwork. although artpedia offers a moderate number of captions per image (3.1 on average), its total image count is significantly lower, leading to a smaller pool of captions overall (9173). this comparative analysis highlights the balance achieved in our dataset between the quantity of images and the variety of annotations. the emphasis on obtaining multiple captions per image enriches the dataset by incorporating a variety of descriptive styles, subjective interpretations, and visual details, thus providing a comprehensive base for fine-tuning models. the iterative process of data collection and quality checks ensures that our dataset maintains both breadth and depth, allowing the generation of high-quality, diverse painting captions. to measure diversity, we calculated three metrics and compared them with the artcap dataset. our choice is due to the similarity in dataset structure (e.g. multiply captions per image, focus on visual content). diversity was measured using the following metrics: results are shown in ## model architecture and training process for model training and caption generation experiments, we chose an open-sourced model, the llava (large language and vision assistant) llava-v1.5-7b. llava is a novel end-to-end large multimodal model that combines a vision encoder and an llm for general-purpose visual and language understanding. it represents a significant metric our dataset artcap observation lexical diversity 0.5831 0.4765 our dataset has higher lexical diversity, meaning it uses a larger variety of unique words relative to its total word count. this suggests that captions in our dataset are more varied in vocabulary compared to artcap data. semantic diversity 0.8094 0.8236 both datasets exhibit high semantic diversity, but artcap dataset is slightly more diverse. captions in artcap likely describe the images using different structure of sentences more often than in our dataset. edit distance 174.73 44.07 our dataset has a much higher edit distance, indicating its captions are structurally more distinct. captions in artcap dataset are more similar in word arrangement and structure. retraining or even fine-tuning llms typically demands extensive datasets and significant gpu hours. this process not only consumes considerable computational resources but also carries the risk of catastrophic forgetting, where the model loses the knowledge it previously acquired when too many layers of the network update their weights. to address these challenges, one of the parameter-efficient fine-tuning (peft) approaches @xcite has been used for domain adaptation of the llava model. peft methods are designed to adjust only a small subset of the model's parameters while keeping the majority of them fixed. this makes the fine-tuning process more efficient and less resource-intensive. by focusing on a limited number of parameters, peft techniques significantly reduce the computational load and the amount of data required, enabling quicker and more cost-effective adaptation to new domains. this can lead to a more agile and scalable deployment of llms for specialized application domains, ensuring that the model remains both accurate and efficient. lora (low-rank adaptation of large language models) @xcite is one of the peft techniques to train llms on specific tasks or domains. this technique introduces trainable rank decomposition matrices into each layer of transformer architecture and also reduces the number of trainable parameters for downstream tasks while keeping the pre-trained weights frozen. to further optimize resource usage and finetuning efficiency, we employed qlora (quantized low-rank adaptation) instead of traditional lora. qlora was the most optimal choice because it reduces the memory footprint even further by leveraging 4-bit quantization, allowing for the fine-tuning of llms on consumer-grade hardware without sacrificing model performance. the use of qlora enables efficient memory utilization, allowing us to fine-tune larger models with fewer hardware resources, significantly lowering both the cost and time required for adaptation @xcite . our qlora( ## evaluation however, the results on semart were more modest. fine-tuning improved rouge1 from 0.19 to 0.21 and rouge2 from 0.027 to 0.11, while the gains in rougel and rougelsum were similarly limited (0.14 to 0.16). the lower clipscore of 0.315 for the fine-tuned llava on semart, compared to 0.42 on our dataset, indicates that the captions generated for semart images were less contextually aligned with the visual content. this disparity suggests that the model's ability to generate highly relevant captions is influenced by the characteristics of the dataset used for training, with our dataset providing a better foundation for capturing the nuanced relationship between text and imagery. overall, the evaluation demonstrates that finetuning using qlora can significantly improve the performance of mmmls when training for specific domains, especially when these domains do not (or cannot) have extensive datasets. moreover, the richer and more diverse the manual annotations, the higher the quality of the generated captions, as reflected by the lower clipscore. ## limitations and discussion given the widespread excitement surrounding llm capabilities and despite the improvements our fine-tuned model brings, we questioned whether these quantitative results also reflect a better quality of the generated captions from a human viewpoint. we thus embarked on an empirical exploration; our experiments with the baseline llava model and the improvements that the fine-tuned llava model achieved point to limitations in terms of the effectiveness of general-purpose mllms in the absence of domain-specific adaptations. as a dog and a bird, which do not exist in the painting. similarly, for "jupiter and bellerophon" (figure 2a ), it inaccurately describes a scene involving angels when the painting actually features a man and a winged horse. this may also be interpreted to some extent as a case of mistaken identity in the case of the horse, whose wings made the baseline model believe it is an angel. on the other hand, the man on the left does not have wings, and the baseline model hallucinates angel instead. finally, in "annunciation" (figure 2b ), the basic model hallucinates a baby and a potted plant; this last could also be interpreted as mistaken identity since we assume the wings are interpreted as greenery. both models hallucinate a man in white. these limitations suggest that foundation models like llava struggle with both accuracy and completeness in complex visual art contexts. developing a domain-specific dataset and fine-tuning mllm models like llava for art interpretation is a more reliable approach. ## conclusion this work introduces a novel dataset for caption generation of paintings sourced from a publicly available dataset. our caption dataset was created through a year-long crowdsourcing campaign on zooniverse, with volunteers adhering to explicit captioning guidelines. we have shown that this is a valuable resource for research and development in the field of automated caption generation for visual art. it can be used to fine-tune mllms -such as llava -for domain adaptation, resulting in significant improvements in the quality of generated captions compared to the baseline models. the lora approach to fine-tuning mllms appears to be particularly effective in enhancing the model's ability to generate captions that are more coherent, contextually aligned, and better capture the nuances of (a) "jupiter and bellerophon", unknown. baseline llava: the image depicts a painting of a man being lifted into the air by two angels. the man is positioned in the center of the scene, with one angel on his left side and the other on his right side. the angels are holding the man, and he appears to be flying through the sky. in the background, there are two people standing, possibly observing the scene or waiting for their turn to be lifted. the painting captures a moment of wonder and awe. fine-tuned llava: the image features a painting of a man and a winged horse flying through the sky. the man falls from the horse, which is positioned towards the right side of the scene. the painting captures the dynamic and dramatic moment of the man and the horse soaring through the air. in the background, there is another man visible, on the left side of the painting. the scene appears to be set in a mountainous landscape, with a mountain visible in the background. the overall atmosphere of the painting is one of adventure and freedom. baseline llava: the image depicts a scene from a painting, featuring a woman and a child. the woman is kneeling down, holding a baby, while the child is standing nearby. the woman appears to be dressed in a yellow gown, and the child is wearing a white gown. the painting is set in a room with a window in the background. in the room, there is a chair positioned towards the right side of the scene, and a potted plant is located on the left side. additionally, there is a bowl placed on the floor, close to the woman and the child. fine-tuned llava: the image features a painting of a woman sitting in front of an angel, who is kneeling and holding a white flower. they appear to be engaged in a conversation or a religious ceremony. the woman is wearing a yellow dress, while the man is dressed in white. in the background, there is a window with a curtain, which adds to the atmosphere of the scene. the painting is likely a religious or historical depiction, as it captures the interaction between the two individuals. baseline llava: the image features a woman dressed in a warrior costume, standing on a horse and holding a shield. she appears to be engaged in a battle or a mock fight. the horse is positioned in the center of the scene, with the woman standing on its back. there are two other people in the background, one on the left side and the other on the right side of the image. the scene also includes a dog, which is located on the left side of the image, and a bird, which is flying in the sky above the woman and the horse. fine-tuned llava: the image depicts a painting of a woman killing a centaur. the woman is wearing a helmet and is holding a shield, while the man appears to be holding a sword. the scene takes place in a lush green field, with a body of water in the background. there are several other centaurs in the painting, some of whom are closer to the water and others further away. the overall atmosphere of the painting suggests a battle or a moment of intense action. artistic interpretation. we believe that this research will contribute to further advancements in automated caption generation for paintings and other forms of visual art, ultimately enhancing accessibility and understanding of these cultural artifacts.