Context or Retrieval? Evaluating RAG Methods for Art and Museum QA System (2025)

Abstract: Recent studies suggest that increasing the context window of language models could outperform retrieval-augmented generation (RAG) methods in certain tasks. However, in domains such as art and museums, where information is inherently multimodal, combining images and detailed textual descriptions, this assumption needs closer examination. To explore this, we compare RAG techniques with direct large-context input approaches for answering questions about artworks. Using a dataset of painting images paired with textual information, we develop a synthetic database of question-answer (QA) pairs for evaluating these methods. The focus is on assessing the efficiency and accuracy of RAG in retrieving and using relevant information compared to passing the entire textual context to a language model. Additionally, we experiment with various strategies for segmenting and retrieving text to optimise the RAG pipeline. The results aim to clarify the trade-offs between these approaches and provide valuable insights for interactive systems designed for art and museum contexts.

## introduction the role of interactive question-answering (qa) systems in cultural activities, specifically art and museum contexts has grown significantly in recent years with the widespread of conversational chatbots @xcite . these systems must navigate the complexities of multimodal data, combining visual and textual elements to provide accurate and contextually relevant responses. recent advances in large language models (llms) with extended context windows @xcite suggest new possibilities for such applications, potentially reducing reliance on classical retrieval mechanisms, but a series of problems arise due to the nature of these systems. along these advancements, retrieval-augmented generation (rag) remains a promising approach, especially for domains where information density exceeds the limits of even the largest context windows @xcite . inspired by @xcite , the present work investigates the trade-offs between rag methods and direct large-context approaches for qa tasks in art and museum contexts, where the information is inherently multimodal and domain-specific. by translating this visual context into detailed captions, and combining it with the original data, we will create a synthetic dataset for the evaluation different rag pipelines. comparing their outputs to a baseline generated using a large-context model, we aim to provide actionable insights for developing interactive systems in this domain. section 2 will perform an analyses of largecontext models, rag and their place within the landscape, as well as hot topics within the research community. we will lay down the foundations of the methods in section 4, outlining the methodology and introducing the dataset we used for the experimentation. section 5 covers the experiments performed with the different configurations and parameters, and finally a conclusion is given in section 6, along with a few lines of research that might be interesting to explore. ## related retrieval-augmented generation (rag) has emerged as a key paradigm for improving the performance of language models in tasks requiring external knowledge @xcite and to reduce hallucinations (false or nonsensical outputs). prior works have acknowledged the efficacy of rag in both open and domain-specific applications, ranging from technical documentation to healthcare qa systems @xcite , @xcite . multimodal data comes as well with its own set of challenges @xcite . however, research on its applicability in domain-specific tasks, such as art and museum contexts interaction, remains sparse. parallel to these developments, advancements in large-context models (even in the range of millions of tokens) have enabled direct ingestion of extensive input, offering an alternative to rag for tasks involving dense information. multimodal large language models (mllms) research has also been a hot topic lately @xcite . with this in mind, the need for a representation of the visual context in the rag systems arises, specially in visual domain applications @xcite . recent research builds around the use of rag in conjunction with llms @xcite . specifically in the rag landscape, the chunking strategies of the indexed documents remains a research topic @xcite , as well as the embedding techniques for retrieval tasks @xcite . this paper builds on these foundations by comparing rag with large-context approaches in a multimodal setting where the visual context is inferred to textual data, focusing on the art and museum domain. ## data preparation when considering the interaction that visitors in a museum will have with a corporate chatbot, we can consider mainly two types of questions: those related with direct visual elements in the artwork, and those that transcend the artwork (i.e., name of artist, year, movement, technique, history, etc.). while designing our methodology, we consider both types of questions and the information we can have available: the images of the artworks and text describing the artwork (information usually available near the artwork or provided by the museum). therefore, the dataset we used initially consists of painting images paired with detailed information sourced from museum databases and archives. the information that comes along the paintings at times references the author of the painting, and its historical context, but it rarely contains detailed descriptions of the painting's visual characteristics. for this reason, each image is processed using a vision language model to generate a descriptive caption, representing its visual context, divided in subject (depicted object(s)), form (visual organization: color, style, shape) and content fields (impact or meaning) @xcite . this caption is then concatenated with the original textual description to create a comprehensive context. using a language model, we developed a synthetic database of question-answer (qa) pairs based on this multimodal context. the dataset includes 1020 images and 12240 qa pairs, covering diverse artistic styles, historical periods, and cultural themes. this means that 12 questions are generated for each work of art. we wanted the questions and answers to be diverse, so we ask the large language model to formulate and answer 6 questions that a beginner user will ask, and other 6 as an expert in art. furthermore, for each level, 3 questions are generated based on the original textual data, and the other 3 are based on the descriptive caption generated. this multimodal dataset serves as the foundation for evaluating both rag and large-context approaches. ## methods this section outlines the methodology developed and applied in this study to evaluate retrievalaugmented generation (rag) approaches in contrast with large-context models in a multimodal art synthetic question-answer (qa) dataset. first, a detailed caption of the image is generated with the "qwen2-vl-7b-instruct" model, a vision model with a size of 7b parameters @xcite . the rag pipeline begins by segmenting the text provided by the museum related to each artwork. using several strategies. recursive splitting splits the text based on the structure of its paragraphs, first splitting by double newline, then by newline and finally by period @xcite . in the semantic chunking approach we divide documents based on semantic similarity, using the "all-minilm-l6-v2" (transformers, 2021) embedding model we compute the similarity between sentences and group them together @xcite . sentence-based division chunks texts into accurate and contextually meaningful chunks with the spacy tokenizer @xcite , maintaining the integrity of sentences and their relationships within the text. the resulting chunks will then be embedded using three different embedding types (dense, sparse and late interaction). nomic's "modernbert-embedbase" @xcite model (embedding length of 768) is used as the dense embedding with cosine distance for comparisons. contextualized late interaction over bert (colbert) @xcite introduces a late interaction architecture that encodes the query and the document using bert and then employs an interaction step that models the similarity. finally for the sparse representation, we will use "bm42" (qdr, 2024), which builds upon the foundations of tf-idf (term frequency-inverse document frequency). the alternative approach involves feeding the full artwork context, i.e., comprising the original text information and the qwen generated description captions, directly into a llama 3.1 model of size 8b parameters @xcite . we chose this model for its performance in "needle in a haystack" benchmarks, for its usage of context length and relatively small number of parameters @xcite . this model generates answers based on the entire input without relying on intermediate retrieval steps. the model is instructed to base its answers solely on the input text, to prevent hallucination, or answers already learned (see annex a.3 for the prompt). synthetic qa pairs were created using "gpt-4turbo" model with 128k of context window @xcite . with the original text plus the generated captions as input, we generate pairs of synthetic questions along their corresponding answers by prompting the gpt-4-turbo model both the question and the answer (see annex a.1 for the prompt). a random sample of around 1200 qa (aprox. 10% of the dataset) pairs were manually inspected to verify the quality of the generated dataset. these generated qa pairs will be then taken as ground truth, providing a benchmark baseline for evaluating the similarity of the answers generated by both rag and large-context approaches. we will then, for each question in the synthetic dataset, test two approaches: (a) first, retrieve the most relevant documents based on the cosine similarity score between the question embedding and different types of chunk embeddings., and (b) compare the performance against the responses of llama3.1-8b with the full-context approach. in both cases, we will embed the original answer from the qa pairs, the full-context (llama3.1-8b) generated answer and the rag retrieved answer with the aforementioned "modernbert-embed-base" model, and compute cosine similarity between both generated answers and the ground truth answer. this metric provides a robust framework for comparing rag outputs with the baseline answers vs the large-context model approach. ## experimentation we conducted experiments using a multimodal synthetic qa dataset with the goal of evaluating the effectiveness of retrieval-augmented generation (rag) versus large-context models in the task of properly answering questions about paintings. the experiments were run on a machine with an nvidia geforce rtx 3090 (24gb of vram) using python and the transformers library. the rag pipeline was built with the qdrant vector search library, for its ease of use and extensive community and support @xcite . the final dataset consists of 1020 images with textual information and captions detailedly describing them. as it was previously described, each one of the images has 12 qa pairs associated generated with gpt4 (see annex a.2). this synthetic dataset adds up to 1020*12= 12,240 total queries and their answers. the context-based answers were generated by feeding the full context (original descriptions + qwen captions) into the llama 3.1-8b model. the model was instructed to base its responses exclusively on the provided context, mitigating potential hallucination (see annex a.3). as previously said, we designated several chunking strategies to segment the texts, depending on where we want to put the focus. depending on the embedding size and structure of the data, the documents must be segmented in proper chunk sizes (number of characters in each chunk). as highlighted in for the recursive splitting we set a maximum chunk size of 300 characters with an overlap of 50, which means that chunks will be as big as 300 characters, with the first 50 being from the previous chunk, and the last 50 from the next chunk. when performing the sentence-based chunking strategy, we state a chunk size of 200 characters that will leave room for adjoint short sentences to be grouped. if a sentence is bigger than 200 characters, it will still be treated as a chunk. finally for the semantic approach, we will use a chunk size of 300 characters. note that in semantic embedding, if a sentence is not similar enough to any other one, it will remain as a standalone chunk. the "bm42" sparse embedding model, grounded in tf-idf principles, was evaluated for its efficiency in representing keyword-based similarities. sparse embeddings excel in retrieval speed and computational simplicity, particularly for short, keyword-rich contexts, and here it's no different, it performs better with the caption texts, as it is more concise and contained. particularly in "expert" questions with short answers containing keywords it performed best. on the other hand when facing more general or open ended questions, specially with questions regarding the original information which is inherently of diverse nature, the bm42 model achieved worse results. we use the "modernbert-embed-base" model, which produces 768-dimensional embeddings optimized for capturing semantic similarity. these dense embeddings performed better in scenarios where the question and context were long. this time, the model performed well with both "expert" and "beginner" questions, but questions about the original information give worse results, in favor of questions about the captions. in our experiments, colbert performance falls behind its two other opponents, but surprisingly its results are independent of the type (caption or information) and level (expert or beginner) of the question queried. the results show that the similarity is maintained along all of the questions no matter their structure, length or label. ## conclusions and future work after experimenting with different chunking approaches and parameters, the strategies chosen to test on this dataset are: semantic chunking with a chunk size of 300 characters, spacy sentence-based segmentation with a size of 200 characters, and recursive splitting with a maximum chunk size of 300. the dense embedding approach with modern-bert generally works better than its two opponents, scoring higher regardless of the chunking mechanism used. at the same time its variance is higher than the other two, scoring higher in questions derived from the captions versus the original information. the late interaction colbert model on the other hand, although performing slightly worse, keeps higher consistency, with low variance along the different labeled fields. at last, bm42 falls right in between the other two embedding models. as showcased in on the comparison of rag results with the baseline llama answers, we can definitely say that we proved what was stated in @xcite , in which they also found large-context model to outperform rag based approaches in accuracy. being this said, the approach that consistently closes the gap the most is the sentence-based document chunking strategy combined with modernbert embeddings. regarding the dataset, can observe trends that suggest that the most precise answers are generated for questions about the "description" of the artwork. a slight advantage is present in the "expert" questions, over the "beginner" ones. it must be said that we are measuring the raw answers directly from the rag pipeline, without performing any modification/augmentation technique. the evaluation at this point is being carried out by cosine similarity, but many other metrics would need exploration. there are specific libraries for this purpose such as (rag, 2024), @xcite , which evaluate rag and llm pipelines in both quantitative (precision, similarity, recall, etc.) and qualitative categories (hallucination, noise, factual correctness, etc.). the most natural approach to character recursive semantic spacy chunks 27,377 17,677 21,693 25,532 by the venetian style. from her prayer-stool at the left of the painting, the virgin listens attentively to the message of the archangel, a figure rendered very much in the style of veronese. the light and the colouring owe much to titian, a painter ei greco admired, while the arrangement of the figures and the treatment of the drapery strongly recall the work of tintoretto. here, ei greco places the figures within a simple architectural setting, loosely framing them to make the scene more realistic. caption: ### subject the painting depicts a religious scene, likely from the christian tradition, featuring two central figures: a seated woman and a standing angel. the woman, dressed in a blue robe, appears to be in a state of contemplation or prayer, with her hands raised in a gesture of supplication or meditation. the angel, dressed in a yellow robe with a white cloak, is depicted with wings and a halo, suggesting a divine or celestial presence. the angel is reaching out towards the woman, possibly conveying a message or blessing.