Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts (2022)

Abstract: Recent breakthroughs in NLP research, such as the advent of Transformer models have indisputably contributed to major advancements in several tasks. However, few works research robustness and explainability issues of their evaluation strategies. In this work, we examine the behavior of high-performing pre-trained language models, focusing on the task of semantic similarity for visual vocabularies. First, we address the need for explainable evaluation metrics, necessary for understanding the conceptual quality of retrieved instances. Our proposed metrics provide valuable insights in local and global level, showcasing the inabilities of widely used approaches. Secondly, adversarial interventions on salient query semantics expose vulnerabilities of opaque metrics and highlight patterns in learned linguistic representations.

## introduction semantic similarity between pairs of sentences serves a large variety of applications in the field of natural language processing, such as document retrieval, text classification, question answering and others. even though such tasks have risen in popularity since the introduction of the transformers @xcite , and despite the attention given on robustness and transparency of nlp transformers @xcite @xcite few efforts have addressed explainable evaluation @xcite . text-image retrieval is a real world semantic similarity application where the task is to feed a textual input to a system, and receive an image as a response. visual details of the retrieved instance need to accurately correspond to the textual descriptions, often in a fine-grained fashion. any mismatch between modalities can be easily perceived by humans, and captured by automated metrics. such evident disagreements can act as starting points for further investigation, revealing inner processes on the semantic matching procedures. in this work, we aim to unveil the evaluation strategy of semantic similarity models. specifically, we apply pre-trained transformers on visual vocabularies and obtain results via ranking. first, we address the shortcomings of traditional ranking metrics @xcite , which provide either a binary answer (item found in top-k items or not), or position-informed variants (item found in the k-th position). however, such measures cannot provide detailed insights regarding the contribution of the scene constituents to the rank position. for example, if an instance is ranked in the k-th position, items in previous k-1 positions may be highly relevant to the ground truth one or on the contrary, highly irrelevant. to this end, we propose novel explainable ranking evaluation metrics that decompose and quantify the conceptual differences between ground truth and retrieved instances in local and global level. even then, we observe that existing metrics lack a way to assess whether the top-ranked items are actually relevant to the query. for this reason, we construct adversarial queries where an attribute is replaced with a conceptually divergent one, in order to evaluate the response of a ranking system to distorted inputs. in all cases, frequently misperceived semantics captured by our evaluation framework reveal patterns imprinted in the learned representations of language models. our overall approach is applicable regardless of the chosen language model or ranking system. ## related work a whole new world of possibilities in nlp has opened since the advent of the transformer @xcite , with successful milestones such as bert @xcite and roberta @xcite serving as backbone models for many applications. mpnet @xcite combines permuted language modeling with masked language modeling to overcome the shortcomings of its predecessors. towards reducing model sizes, knowledge distillation followed in distil-bert/distilroberta @xcite , minilm @xcite and tinybert @xcite , as well as parameter reduction techniques implemented in albert @xcite achieve more compact models while maintaining performance. the textual semantic similarity task was greatly benefited by sentence-bert (sbert) @xcite , a siamese-bert variant that allows efficient embedding representations using the aforementioned models accordingly. traditional evaluation metrics such as hits, mean reciprocal rank (mrr), precision, recall and f-score @xcite have dominated the field of information retrieval. while these metrics serve the purpose of assessing the retrieved information, they do not provide explainable means of justification. explainable evaluation metrics @xcite aim to address this challenge. lack of trust of neural methods due to biases, outdated training, and inaccurate assumptions has led to the need for explainable methods in language models. research towards that direction has utilized concept attributions @xcite , chunk alignments @xcite , feature importance @xcite , or explanations by simplification @xcite . adversarial examples can also provide insights regarding the inner workings of obscure models, and are closely related to counterfactual explanations, placing them in the broader area of explainability @xcite . numerous works address the problem of adversarial examples for natural language models @xcite , with recent methods addressing the robustness of nlp models such as bert through adversarial examples/attacks @xcite . in this work, we approach adversarial examples from a different perspective, first of all tackling a different problem than classification that most works do, and secondly by realizing the creation of adversarial examples on the semantic rather than the linguistic level, investigating the effect of semantic changes on the ranking of textimage retrieval systems. ## overview our workflow consists of three stages: representation, ranking and explainable evaluation. we view text-image retrieval as a query-corpus retrieval problem, exclusively exploiting linguistic information for representation and ranking, while revealing visual information only at the evaluation stage, where we compare retrieved images with the ground truth ones. as input, we consider a dataset of size n that contains complex scene images i @xmath0, accompanied by query-corpus pairs (q i , c i ), q @xmath1, c @xmath2, @xmath3, 2, ..., n with each corpus c i consisting of an arbitrary number of sentences s j , @xmath4, 2, ..., l c . in the representation stage, pre-trained sentence similarity transformers @xmath5, c instances in a common vector space u . cosine similarity scores between query-corpus embedding pairs in u are sorted to provide a rank r i per query q i in the ranking stage, with r i either lead to success, if the ground truth image i g i with corpus c i is returned at the top of the rank, or failure otherwise. we provide a visual demonstration of the ranking procedure in figure 2 . all failures per model m , i.e. image pairs (i g , i r ) for which i g ̸ = i r are stored in a set f, which is further passed to the evaluation stage. we then employ three methods to provide an understanding and evaluation towards failures: transparent ranking metrics (section 4), human evaluation and adversarial re-ranking (section 5). the overview of our proposal is presented in figure 1 . ## explaining failures we count as failure f @xmath6, i r ) @xmath7; instead another image i r i ̸ = i g i with c r ̸ = c i achieved rank @xmath8. following the 'blind' evaluation strategy of traditional ranking metrics, we provide a measure of retrieval failures as the cardinality of the failure set: @xmath9. however, it is not possible to verify if i r i can accurately satisfy q i without exploiting visual information. to this end, we exploit visual annotations and human perception to quantify the suitability of each i r @xmath10. by decomposing all semantics that contribute to the suitability of each i r i we obtain a discrete and transparent conceptual measure of similarity between (i g , i r ) i . ## adversarial re-ranking we create adversarial queries @xmath11, upon which adversarial rankings r * per q * are extracted. figure 3 provides the causal graph of adversarial interventions for any q @xmath12. ## experiments we implement the same experimental strategy on flickr8k, flickr30k and vg ∩ coco datasets, obtaining query-corpus representations in a common semantic space u for a variety of models @xmath13. the code for our experiments is provided in submitted supplementary material. ## conclusion in this work, we presented an evaluation framework for text-image retrieval and experimented with pretrained transformer-based semantic similarity models. our approach achieved in capturing representation patterns and evaluation shortcomings of widely used metrics in local and global level. as future work, we aspire to extend our automated metrics to include attributes and spatial relationships between concepts, and produce adversarial re-rankings using verb antonyms, singular-plural sentence transformations and rare synonyms of salient concepts.