Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics (2022)

Abstract: Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users. Image-based NLG systems have progressed to the point where they can begin to address this persistent societal problem, but these systems will not be fully successful unless we evaluate them on metrics that guide their development correctly. Here, we argue against current referenceless metrics – those that don’t rely on human-generated ground-truth descriptions – on the grounds that they do not align with the needs of BLV users. The fundamental shortcoming of these metrics is that they do not take context into account, whereas contextual information is highly valued by BLV users. To substantiate these claims, we present a study with BLV participants who rated descriptions along a variety of dimensions. An in-depth analysis reveals that the lack of context-awareness makes current referenceless metrics inadequate for advancing image accessibility. As a proof-of-concept, we provide a contextual version of the referenceless metric CLIPScore which begins to address the disconnect to the BLV data.

## introduction in the pursuit of ever more powerful image description systems, we need evaluation metrics that provide a clear window into model capabilities. at present, we are seeing a rise in referenceless (or reference-free) metrics @xcite @xcite , building on prior work in domains such as machine translation @xcite and summarization @xcite good description bad description figure 1 : whether an image description makes an image accessible depends on the context in which the image appears. referenceless metrics like clipscore can't capture such context-sensitivity. we provide experimental evidence with blind and low vision (blv) participants that this makes current referenceless metrics insufficient for evaluating image description quality. @xcite @xcite . these metrics seek to estimate the quality of a text corresponding to an image without requiring ground-truth labels (i.e., reference descriptions), or crowd worker judgments. thus, they offer the promise of quick and efficient evaluation of image description models, and are even suggested to be more reliable than existing reference-based metrics @xcite . here, we investigate the value of such metrics for a high social-impact domain: assessing the usefulness of image descriptions for blind and low vision (blv) users. automatically generating descriptions to make images accessible is an important goal: though images are omnipresent in digital communication @xcite @xcite @xcite , user-generated descriptions are rare @xcite , which has serious implications for blv users @xcite . can referenceless metrics help guide models to generate descriptions that align with what blv users value? studies with blv users emphasize the impor-tance of the context in which an image appears. for example, while people's clothing is highly relevant when browsing shopping websites, their identities become central when reading the news @xcite @xcite . not only the domain but also the immediate context matters for selecting what is relevant. consider the image in figure 1 , showing a park with a gazebo in the center and a sculpture on a pedestal in the foreground. a description written for the image's occurrence in the wikipedia article of gazebos ("a freestanding, open, hexagonal gazebo with a dome-like roof in an idyllic park area.") is unhelpful if the image instead appears in the article on sculpture. this simple example illustrates that context could play a central role in the assessment of description quality. in this work, we report on studies with sighted and blv participants that seek to provide rich, multidimensional information about what people value in image descriptions for accessibility. in contrast to current practices, we elicit and evaluate image descriptions within contexts the images could appear in, here wikipedia articles. we find that, for both sighted and blv participants, the description's relevance to the context is a major driver of their overall assessments. we then use this experimental data to evaluate two very different referenceless metrics: clip-score @xcite , which assesses a description's quality relative to its associated image, and spurts @xcite , which relies only on linguistic properties of the text. by their very design, these metrics don't capture the effects of context seen in our user studies, since they treat description evaluation as a context-less problem. this shortcoming goes undetected on most existing datasets and previously conducted human evaluations, which presume that image descriptions are context-independent, but it is immediately apparent in our evaluations. these results suggest that current referenceless metrics are not reliable guides due to their lack of context integration, but offers a path forward: perhaps referenceless metrics can be modified to include this missing context. as a proof-of-concept, we show that a context-sensitive adaptation of clipscore results in improved correlations with human judgments -a promising signal for the development of future context-sensitive referenceless metrics. ## experiment: the effect of context on human image description evaluation efforts to obtain and evaluate image descriptions through crowdsourcing are mainly conducted outof-context: images that might have originally been part of a tweet or news article are presented in isolation to obtain a description or evaluation thereof. following recent insights on the importance of the domains an image appeared in @xcite @xcite , we seek to understand the role of context in shaping how people evaluate descriptions. figure 2 provides an overview of the two main phases. firstly, we obtained contextual descriptions by explicitly varying the context each image could occur in (figure 2a ). we then explored how context affects sighted and blv users' assessments of descriptions along a number of dimensions (figure 2b ). finally, in section 4, we compare these contextual evaluations with the results from the referenceless metrics clipscore @xcite and spurts (feinglass and yang, 2021). ## referenceless metrics for image accessibility referenceless metrics have been shown to correlate well with how sighted participants judge description quality when descriptions are written and presented out-of-context @xcite @xcite . while image accessibility is one of the main goals referenceless metrics are intended to facilitate @xcite @xcite , it remains unclear whether they can approximate the usefulness of a description for blv users. inspired by recent insights into what makes a description useful, we argue that the inherently decontextualized nature of current referenceless metrics makes them inadequate as a measure of image accessibility. we focus on two referenceless metrics to support these claims: clipscore @xcite and spurts @xcite . appendix b briefly considers other referenceless metrics. clipscore uses the similarity of clip's image and description embeddings as the predictor for description quality, as formulated in (1). denoting there are no significant correlations between clipscore and human ratings. spurts correlates significantly with all responses provided by blv participants but negatively with irrelevance and not at all with sighted participant ratings, indicating a fundamental mismatch. ## the potential for integrating context into clipscore can referenceless metrics like clipscore be made context sensitive? to begin exploring this question, as a proof of concept, we amend (1) as follows: here, quality is a function of (a) the description's similarity to the context (first addend) and (b) whether the description captures the information that the image adds to the context (second addend). these two addends can be seen as capturing aspects of (ir)relevance and imaginability, respectively, though we anticipate many alternative ways to quantify these dimensions. however, despite these promising signs, there are also reasons to believe that clip-based metrics have other restrictive limitations. due to clip's training, images are cropped at the center region and texts need to be truncated at 77 tokens @xcite . clip relies on embeddings learned for each absolute token position in the text window and each patch position in the image. these can therefore not be easily extended to avoid any context or image cropping that is currently limiting clipscore. specifically for the purpose of accessibility, the information this removes can be crucial for determining whether a description is useful or not. for instance, our experiments show that the length of a description is an important indicator for description quality -information lost in clipbased metrics. moreover, this disproportionately affects the ability to encode the context paragraphs, which are often longer than a typical description. these decisions are therefore likely reflected in any resulting metric and should therefore be reconsidered when devising a new metric. ## conclusion the context an image appears in shapes the way high-quality accessibility descriptions are written. in this work, we reported on experiments in which we explicitly varied the contexts images were presented in and investigated the effects of this contextual evaluation on participant ratings. these experiments reveal strong contextual effects for sighted and blv participants. we showed that this poses a serious obstacle for current referenceless metrics, but we also see promise for future efforts since the inclusion of context to a prominent metric such as clipscore begins to address the disconnect from blv needs.