What Makes for Good Image Captions? (2025)

Abstract: This paper establishes a formal information-theoretic framework for image captioning, conceptualizing captions as compressed linguistic representations that selectively encode semantic units in images. Our framework posits that good image captions should balance three key aspects: informationally sufficient, minimally redundant, and readily comprehensible by humans. By formulating these aspects as quantitative measures with adjustable weights, our framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements. To demonstrate its applicability, we introduce the Pyramid of Captions (PoCa) method, which generates enriched captions by integrating local and global visual information. We present both theoretical proof that PoCa improves caption quality under certain assumptions, and empirical validation of its effectiveness across various image captioning models and datasets.

## introduction image captioning, the process of translating visual content into natural language descriptions, serving pivotal roles in real-world applications ranging from assisting visually impaired individuals @xcite @xcite to facilitating content-based image retrieval @xcite @xcite @xcite . over the last decade, the field of image captioning has witnessed substantial progress, primarily driven by advancements in deep neural nets and the availability of large-scale high-quality image-text datasets. despite empirical advancements, several fundamental questions remain unanswered: what makes for good image captions? which properties should they possess, and how can we measure them? some existing models can generate captions closely resembling single-sentence human annotations @xcite , but these may not be adequate for use cases where more comprehensive coverage of fine-grained visual information is required. conversely, recent large vision language models (lvlms) @xcite have demonstrated the ability to generate multi-paragraph detailed image descriptions @xcite @xcite . yet, longer captions can sometimes be less accurate, hallucinate content, or put excessive emphasis on irrelevant details while omitting important ones. recognizing the absence of a universal standard for ideal captions, this work aims to establish welldefined principles for image captioning that address varying task requirements. we introduce an information-theoretic framework based on semantic communication @xcite and the information bottleneck principle @xcite @xcite . by leveraging this perspective, we formulate an objective function for image captioning that strikes a balance among three key criteria: our framework conceptualizes images and captions as observations of latent variables in a semantic space. this allows us to formulate image captioning as a communication process where semantic information is transmitted from the image to the caption, and measure the error at the semantic level. we then present formal quantitative measurements of the above three criteria and define the ultimate objective of image captioning as a weighted combination of them. varying the weighting coefficients of these terms suits different preferences over image captions (e.g., comprehensiveness, succinctness, readability), providing a rigorous foundation for analysis and evaluations. our framework provides a rigorous foundation for analyzing and advancing image captioning techniques. to demonstrate its practical applicability, we present the pyramid of captions (poca) method as an example application. poca employs a hierarchical approach to generate semantically rich captions by leveraging both local and global visual information. utilizing our theoretical framework, we provide formal proof that each local-global aggregation operation in poca improves caption quality under certain assumptions. empirical evaluations across various image captioning models and datasets corroborate our theoretical findings, showing that poca consistently yields more informative and semantically aligned captions while maintaining brevity and interpretability. ## proposed framework in this section, we provide a theoretical framework for image captioning as depicted in figure 1 . first, we formulate the task of image captioning by applying the concept of semantic units @xcite . we suppose that an image is an observation of a latent variable in a semantic space characterized by semantic units. an image captioning model will generate a caption for the image, and the caption can be mapped back to the latent semantic space and compared with the source semantic latent variable. based on this framework, we then introduce our proposed objectives inspired by the information bottleneck principle @xcite @xcite for feature representation learning @xcite . in our framework, we consider that the overall image captioning objective is composed of a "information sufficiency" term, a "minimal redundancy" term, and a "human comprehensibility" term. 𝑋𝑋 𝑌𝑌 � 𝑋𝑋 � 𝑌𝑌 𝑓𝑓 𝜃𝜃 𝑍𝑍 latent semantic space 𝒮𝒮 observed data space 𝒟𝒟 image captioning model semantic level error 𝑔𝑔 ℎ figure 1: overview of our formulation. some latent variable x in a latent semantic space s generates image x in data space d. the image x is then captioned by f θ producing a caption ỹ which can be mapped back to the original latent space as y . the semantic-level error @xmath0. ## proposed method: poca in this section, we introduce the pyramid of captions (poca) method, which showcases the applicability of our framework to image captionoing research. the key intuition behind the poca method is that we can have a more accurate and detailed caption by ensembling multiple captions, as shown in figure 2 . we propose to split an image into multiple local patches generate local captions for each patch, and fuse the captions to obtain a higherquality caption for the global image. formally, let σ split be a function operating in d image that represents the splitting function, which splits an image into a set of local patches: we apply an image captioning model f θ to the local patches and obtain a set of local captions: { ỹ [j] = f θ ( x[j] )} m @xmath1, and also generate a caption for the global image: ỹ = f θ ( x). the local and global information will be fused by a merging function σ merge operating in d caption : we adopt text-only llms as the merging function σ merge . as the poca method is hierarchical, we can extend it to be more layers. we can recursively split a patch into sub-patches and merge captions for each sub-patches to represent the patch. ## poca gives better captions (provably) in this section, we provide an analysis of under what condition a single local-global merging operation in poca can be guaranteed to improve the caption quality. first, we assume that there is some function φ to quantify the error z by x in a deterministic manner, and that function is concave. the deterministic assumption of φ simplifies the analysis by ignoring errors caused by factors other than the input semantics x, such as the randomness in sampling-based autoregressive generation. in other words, we assume that f θ always generates the same caption and makes the same error for the same input. the concavity of φ implies that it generates a larger volume of error when p(w i ) is far from zero and one. this means that the captioning model is more likely to make mistakes when there is high uncertainty about the presence or absence of a semantic unit in the image. assumption 1 (uncertainty-aware contentdependent error) the error z produced by the image captioning model f θ is dependent on the input semantics x. therefore, it can be expressed as a deterministic function of x: next, we introduce our assumptions on the image splitting function σ split and caption merging function σ merge . these assumptions simplify the relationship between local and global semantics by assuming linear combinations. in practice, the relationship may be more complex and depend on factors such as the spatial arrangement of the local patches and the presence of objects spanning multiple patches. assumption 2 (local-global relationship of image semantics) the σ split splits an image into local patches. the latent semantic variables corresponding to local patches satisfy the following relationship with global semantics: we now present a theorem regarding z @xmath2: theorem 1 (poca method reduces semantic error) under assumptions 1-3, the poca method is guaranteed to have the smaller error z merged than z, i.e., since a is non-negative, theorem 1 implies nondecreasing information sufficiency; if merging does not increase redundancy or decrease interpretability, then the overall quality of caption becomes better. this also aligns with the findings in @xcite that smaller-scale models combined can be as effective as a larger-scale model. additionally, it is worth noting that our assumptions require linear combinations of semantics, while it may not hold in practice for images with more complex structures of semantics. proof 1 first, we express the error of the i-th semantic unit in merged caption z merged,i as the difference between the merged caption semantics y merged,i and the source semantics x i . using assumption 3 and 2, z merged,i can be expressed as: here, ( 14 ) is derived from ( 13 ) is based on the decomposition of the global and local caption semantics into there corresponding source semantics and errors, i.e., next, we define ∆ poc,i as the gap between the norm of the global caption error and the norm of the merged caption error for the i-th semantic unit. using the triangle inequality and assumption 1, we derive a lower bound for ∆ poc,i : to yield ( 18 ) from ( 17 ), we apply the triangle inequality, which states that for any two vectors a and b, ||a + b|| ≤ ||a|| + ||b||. finally, we apply assumption 1 to obtain (20), which states by assumption 1, φ is a concave function. applying jensen's inequality and assumption 2, we have: this inequality implies that the error of the global caption is always greater than or equal to the weighted average of the errors of the local captions. intuitively, this means that the poca method, which combines information from both global and local captions, is expected to have a lower error than using only the global caption. combining this result with the lower bound for ∆ poc,i derived earlier, we can conclude that ∆ poc,i is non-negative for all i: the first inequality follows directly from the lower bound for ∆ poc,i derived earlier. the second inequality follows from the jensen's inequality result, which states that φ(x i ) ≥ m j α j φ(x this completes the proof, demonstrating that under the given assumptions, the poca method is guaranteed to reduce the semantic error compared to using only the global caption. ## conclusion our work presents a novel information-theoretic framework that provides well-defined principles for image captioning covering information sufficiency, minimal redundancy, and human interpretability. by leveraging the theoretical framework, we propose pyramid of captions (poca), a novel image captioning approach that employs a hierarchical method to generate content-rich captions by exploiting the complementary nature of local and global visual cues. through theoretical proofs and empirical evaluations, we demonstrate that poca consistently enhances the quality of image captions, making them more informative, semantically accurate, and contextually coherent while maintaining brevity and interoperability.