L a MP -Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles (2025)

Abstract: Figure captions are crucial for helping readers understand and remember a figure’s key message. Many models have been developed to generate these captions, helping authors compose better quality captions more easily. Yet, authors almost always need to revise generic AI-generated captions to match their writing style and the domain’s style, highlighting the need for personalization. Despite language models’ personalization (LaMP) advances, these technologies often focus on text-only settings and rarely address scenarios where both inputs and profiles are multimodal. This paper introduces LaMP-Cap, a dataset for personalized figure caption generation with multimodal figure profiles. For each target figure, LaMP-Cap provides not only the needed inputs, such as figure images, but also up to three other figures from the same document—each with its image, caption, and figure-mentioning paragraphs—as a profile to characterize the context. Experiments with four LLMs show that using profile information consistently helps generate captions closer to the original author-written ones. Ablation studies reveal that images in the profile are more helpful than figure-mentioning paragraphs, highlighting the advantage of using multimodal profiles over text-only ones.

## introduction figures like bar charts or line charts are widely used by scientists, companies, and governments to communicate key insights @xcite . captions-text placed next to these figures-are known to be crucial for helping readers understand and remember the figure's message @xcite @xcite . many models have been developed to generate high-quality captions to help authors compose captions more easily (hsu et al., 1 data: https://github.com/crowd-ai-lab/lamp-cap 2021; @xcite @xcite . for example, the scicap challenges in 2023 and 2024 invited global teams to generate captions for scientific figures in arxiv papers @xcite . systems like sci-capenter also emerged to assist authors by providing ai-generated captions @xcite . despite these advances, studies show that authors almost always need to revise generic ai-generated captions to match their style and the domain's style, with one participant noting, "i need to revise the facade because this is not the right way to present (the concept)" @xcite . this highlights the need for personalized caption generation. meanwhile, the rise of large language models (llms) has recently fueled interest in personal-ized text generation @xcite . benchmarks like lamp @xcite (language models personalization) and longlamp @xcite were created to study how llms can tailor text for specific contexts. however, most of these explorations focused on text-only settings, where both the input (used for generation) and profile (used for personalization) were text-based. how these text-only approaches apply to multimodal scenarios-such as figure caption generation-remains unclear. this paper introduces lamp-cap, a dataset for personalized figure caption generation with multimodal figure profiles ( §3). lamp-cap includes 110,828 target figures-scientific figures for which models aim to generate captions for-each from a distinct arxiv paper. for each target figure, lamp-cap provides the needed inputs (source)figure images and figure-mentioning paragraphs (e.g., "figure 3 shows...")-along with up to three other figures from the same paper, each with its image, caption, and figure-mentioning paragraphs, as a profile to capture context. models are then tasked with generating captions for the target figure using its image and figure-mentioning paragraphs (multimodal source), given a figure profile of sourcecaption pairs from the same paper (multimodal profile for personalization). we used lamp-cap to test caption generation with four llms and found that profile information consistently improved the similarity of generated captions to ground-truth captions ( §4). ablation studies revealed that captions are the most critical profile element, followed by images, with figure-mentioning paragraphs being the least important ( §4.1). our work provides a new benchmark for personalized text generation and demonstrates the effectiveness of using multimodal profiles beyond text-only approaches. personalized llms. personalization of llms has gained attention @xcite , primarily in two directions: (i) personalized text generation (tailoring generated text for specific contexts) and (ii) downstream task personalization (enhancing targeted applications like recommendation systems). we focus on the first direction, defining the personalization target as a group of users-all co-authors of a paper-rather than individuals. prior work, such as the lamp-5 task @xcite on personalized scholarly title generation, has also treated a paper's author group as a single entity for personalization. most prior work in this space has been centered on text-only settings ( §1). for example, lamp included tasks such as news headline generation and email subject creation-relying exclusively on text-based inputs and profiles @xcite . how these approaches extend to multimodal scenarios remains an open question. ## lamp-cap dataset we constructed lamp-cap by curating the sci-cap challenge dataset @xcite . we first selected all papers containing at least two figures. from each paper, we then randomly designated one figure as the target figure (the one needing a caption) and used the remaining figures from that paper (up to a maximum of three, since the scicap challenge allowed at most four figures per paper) as the profile to provide personalization context. following the scicap challenge dataset's split (i.e., 80/10/10 train/val/test), lamp-cap includes 110,828 target figures: 86,197 for training, 12,361 for validation, and 12,270 for testing. among these, 54,680 (49.3%) had one profile figure, 26,193 (23.6%) had two, and 30,027 (27.1%) had three, totaling 197,075 profile figures. papers with only one figure were excluded. see appendix a for details. ## experimental results experiment setups. we evaluated four llms on personalized caption generation using lamp-cap: foot_0 (i) gpt-4o @xcite , (ii) llama 4 scout (metaai, 2025), (iii) gemini 2.5 flash preview (deepmind, 2024), and (iv) gpt-4.1 mini (openai, 2024). the first three are larger models, while the last one is smaller. we used openai's api for gpt-4o and openrouter (openrouter.ai) for the others. we focused on llms because large-scale human evaluations from the scicap challenge showed a clear performance gap between model classes @xcite : only llms like gpt-4v consistently generate captions matching or exceeding those by human authors, while smaller or specialized models such as pegasus @xcite and unichart @xcite perform poorly. building on prior work showing that more profile information improves performance @xcite , we tested four caption generation settings with varying amounts and sources of profile input: (1) no profile: the model generated captions using only the target figure's image and figurementioning paragraphs. (2) one profile: the model used the same source as in (1) but additionally used one randomly selected profile figure from the same paper for personalization. (3) all profile: the model used the same source as in (1) but additionally used all profile figures from the same paper for personalization. (4) other profile: the model used the same source as in (1) but additionally used one or three randomly selected profile figure(s) from random other papers. the setup (4) tests whether performance gains come from paper-specific context or generic in-domain examples. see appendix b for the full prompt. we cleaned the output by removing unnecessary reasoning steps or explanations. we also removed cases (56 out of 12,259) where models failed to generate valid output. see appendix c and appendix d for details. using profile information (from the same paper) makes captions more similar to ground truth, especially with all profile figures. using profiles from other papers often lowered bleu and rouge scores, though there were exceptions. when profile figures shared the same type as the target figure, personalization works better. to examine how figure type affects personalization, we analyzed cases with a single profile figure, splitting them into two groups: those where the profile and target figure types matched (@xmath0,083) and those where they did not (@xmath1,120). personalization is more effective when profile captions are highly similar to the target caption. to test if personalization is more effective when profiles are similar to the target, we split our test set into two groups. we calculated the similarity (using bertscore and rouge-l) be- misaligned set. the results in figure adding more profile figures consistently improved performance, but the largest gain came from the first profile. we examined how the number of profiles affected caption quality using gpt-4o. a subset figure with exactly three profiles from the same paper was used (@xmath2,424). for the 1-profile and 2-profile settings, profiles were randomly sampled. ## conclusion and future work we introduced lamp-cap, a new dataset for personalized caption generation for scientific figures using multimodal profiles, and showed that profiles make captions more personalized across four language models. future work includes expanding profile components, exploring cross-domain generalization, and developing writer-centric evaluation metrics. we are also developing a caption writing assistant that generates personalized captions by analyzing users' local document context.