RONA : Pragmatically Diverse Image Captioning with Coherence Relations (2025)

Abstract: Writing Assistants (e.g., Grammarly, Microsoft Copilot) traditionally generate diverse image captions by employing syntactic and semantic variations to describe image components. However, human-written captions prioritize conveying a central message alongside visual descriptions using pragmatic cues. To enhance caption diversity, it is essential to explore alternative ways of communicating these messages in conjunction with visual content. We propose RONA, a novel prompting strategy for Multi-modal Large Language Models (MLLM) that leverages Coherence Relations as a controllable axis for pragmatic variations. We demonstrate that RONA generates captions with better overall diversity and ground-truth alignment, compared to MLLM baselines across multiple domains. Our code is available at: https://github.com/aashish2000/RONA

## introduction a writing assistant (wa) is a tool (e.g., grammarly, microsoft copilot, copy.ai), often powered by generative ai, that helps users in various writing tasks. was have evolved over the years to support users across a multitude of tasks, with ai-powered assistants being adept at generating a wide selection of content. image captioning (i.e., generating textual descriptions for given images) is one key task that has seen significant advancements with the introduction of multi-modal large language models (mllms). these pre-trained models have achieved remarkable success in generating captions that accurately describe the visual content of images @xcite . however, real-world image captions across different domains often require more than just a description of the visual elements-i.e., they need to convey a central message, provide context, and offer different perspectives on the image @xcite . this leads to significant lack of diversity in the generated captions, limiting the utility of was. existing approaches to fostering diversity in image captions have primarily focused on providing a richer vocabulary (i.e., syntactic variations) or selecting different components of the image to emphasize (i.e., semantic variations) @xcite . although these methods have shown promise, they often fail to capture the nuanced ways in which humans communicate through captions as shown in figure 1 . pragmatic variations, such as multi-modal implicatures and metaphors @xcite , which utilize meanings or connotations beyond the literal description of the image, are often employed by caption writers to make their messages more engaging and relatable @xcite . to address this challenge, in this work, we propose rona (relation-based coherence-aware captioning), a novel prompting strategy for mllms inspired by the concept of coherence relations (crs). based on the principles of discourse theory, crs provide a structured overview of image-text relationships @xcite @xcite , modeling both contextual and pragmatics aspects of language @xcite . we evaluate their effectiveness in image captioning by using crs as guidelines for generating captions that fulfill specific communicative functions while preserving semantic coherence. our analysis includes popular mllms: claude-3.5 sonnet v2 @xcite and @xcite on two datasets-i.e., news captions (anna) and social media captions (tweet subtitles). these datasets contain a wide range of visual objects and abstractive captions @xcite , making this a challenging task for mllms. we demonstrate that rona outperforms existing baselines on caption diversity while retaining ground truth similarity. our contributions are as follows: writing assistants mllms have enabled was to support a variety of writing tasks, with their input ranging from sentence-level suggestions @xcite to long-form writing tasks such as literature reviews and creative writing @xcite , @xcite . in order to improve the human-ai collaboration experience, there exists a need to incorporate human values into ai-based writing systems and vice versa @xcite , @xcite . although these studies have focused on broader task domains, specific writing tasks such as caption writing have been less explored @xcite , particularly pragmatically diverse captioning which we aim to address. image captioning traditional captioning models build on task-specific generative architectures to generate faithful and diverse descriptions for images @xcite . however, with the introduction of mllms, general-purpose models capable of multi-modal representations are utilized for caption generation @xcite . to improve the alignment between image-text linkages, coherence relations (cr) @xcite have been leveraged across different downstream text generation tasks @xcite @xcite . although popular mllms such as gpt-4o and claude sonnet 3.5 v2 are poor at predicting and verifying these relationships @xcite , existing research does not explore the production capabilities of these models. in our work, we investigate the ability of mllm-based was to utilize crs as a guidance mechanism for in-context learning. ## methodology coherence relations rona leverages incontext explanations of coherence relations (crs) as guidance for generating pragmatically diverse captions. we utilize crs that characterize both entity-level and scene-level linkages between an image and its expected caption @xcite . entity-level relations describe the relationships between specific objects in the image and their corresponding elaboration in the caption. scene-level relations, on the other hand, capture the overall context and narrative of the image, providing a broader understanding of the visual content. the selection of these relations are motivated by their generalizability across different domains. overall, the 5 types of relations that we utilize are: datasets popular datasets such as coco captions @xcite or flickr30k @xcite are often used for image captioning evaluation, but the ground-truth captions do not cater to sharing messages or perspectives that are more aligned with human-written captions. instead, we select datasets from 2 different task domains for image captioning: news and social media. these domains provide representative examples of realworld scenarios for the usage of was: (1) the tweet subtitles dataset @xcite ) contains 16,000 image-text pairs sourced from twitter and cleaned to remove noisy, low-quality samples, and on the other hand, for the image + caption task, we provide the model with both the image and a ground-truth caption as input. since both of these components are part of understanding the overall meaning of an image-caption pair, we wish to understand how mllms utilize both modalities to generate diverse captions without the divergence of meaning. this is similar to the prompt-guided image captioning task for mllms @xcite . in both tasks, the baseline mllm is prompted to use the inputs to generate captions with "as much diversity as possible while retaining their original meaning and message." rona utilizes in-context learning where simplified definitions of crs are provided as system prompts. we generate 5 captions per input for each type of task, with rona generating one caption per cr. additional generated caption examples are presented in appendix section e. evaluation metrics to evaluate the performance of mllms on the task of diverse captioning, we measure 4 key attributes: image-caption similarity, ground truth caption similarity, contextual diversity, and bi-gram diversity. clipscore @xcite effectively measures image-caption similarity by converting both modalities into a common representation space. for validating similarity of generated captions with the ground truth text, we turn to bleurt @xcite score. unlike traditional similarity metrics such as bleu @xcite , meteor @xcite and bertscore @xcite , bleurt is trained to balance contextual similarity and human preference judgments, making it better suited for non-descriptive captions. all similarity metrics are computed pairwise between the ground truth modality and generated captions, with the average score reported in our benchmarks. for ## conclusions we propose rona, a coherence relation-based prompting strategy, providing a framework for expressive and diverse image caption generation. our study presents a holistic evaluation of top mllms on their ability to utilize these relationships through in-context learning. our results show that rona enables the generation of a greater variety of captions while improving their overall semantic and contextual relevance across domains. rona serves as a new baseline for future work leveraging image-text relationships for improving the quality of multi-modal writing assistants.