Image Retrieval for Arguments Using Stance-Aware Query Expansion (2021)

Abstract: Many forms of argumentation employ images as persuasive means, but research in argument mining has been focused on verbal argumentation so far. This paper shows how to integrate images into argument mining research, specifically into argument retrieval. By exploiting the sophisticated image representations of keyword-based image search, we propose to use semantic query expansion for both the pro and the con stance to retrieve “argumentative images” for the respective stance. Our results indicate that even simple expansions provide a strong baseline, reaching a precision@10 of 0.49 for images being (1) on-topic, (2) argumentative, and (3) on-stance. An in-depth analysis reveals a high topic dependence of the retrieval performance and shows the need to further investigate on images providing contextual information.

## introduction images are a prominent form of non-verbal communication. images convey messages in diverse ways including scribbles, concept drawings, (produced) photos, data visualizations, or internet memes. moreover, images can serve as an inspiration or an overview of the topic at hand. they play a key role in public discourse @xcite and in expressing personal opinion @xcite , e.g., in the form of political memes. spread "virally," they can gain a large followership on social media and influence political decision-making @xcite or serve as a form of evidence for troublesome events. for example, about 150,000 images are uploaded to facebook every minute. 1 all of these uses often form an integral part of argumentative writing or speechmaking. composing an argumentative text or speech hence does not only include the retrieval and arrangement of written arguments, but also that of relevant "argumentative images" to go along with them. by proposing stance-aware query expansion, this paper shows how to harness existing keywordbased image retrieval technology for the task of retrieving images for arguments. the key assumption of this expansion is that adding terms to the user's query that indicate a stance (i.e., pro or con the query's underlying issue), then images retrieved for the expanded queries support that stance. figure 1 illustrates the desired effect: the shown images stem from google's image search after having expanded the query nuclear energy with either the term good (images on the left-hand side) or the term anti (right-hand side). in what follows, section 3 introduces the task of "argumentative image" retrieval, section 4 outlines our approach to this task-stance-aware query expansion-including three methodological variants, and the sections 5 and 6 detail important elements of our experimental setting as well as evaluation results for the three proposed methods. ## related work al-khatib, 2019 defines an argumentation strategy as "a set of principles that guides the selection and arrangement of arguments (plus contextual information) in an argumentative discourse." while research in computational argumentation has so far focused on verbal argumentation, real-world discourse often integrates images to great effect, ranging from memes emerging on the spur of of a moment to figures summarizing months of research. discourse analyses thus often include images (e.g., @xcite . images for arguments images can provide contextual information and express, underline, or popularize an opinion @xcite , thereby taking the form of subjective statements @xcite . some images express both a premise and a conclusion, making them full arguments @xcite . other images may provide contextual information only and need to be combined with a conclusion to form an argument. in this regard, a recent semeval task distinguished a total of 22 persuasion techniques in memes alone @xcite . moreover, argument quality dimensions like acceptability, credibility, emotional appeal, and sufficiency @xcite all apply to arguments that include images as well. and as a kind of visual argumentation scheme (a "stereotypical pattern of human reasoning"; @xcite , some images are frequently adapted to different topics @xcite . social groups even create their own symbolisms and use them to express opinions (e.g., fringe web communities; @xcite . the potentially high emotional impact of images to a vast audience @xcite can cause changes in the social discourse and eventually politics @xcite . examples include photos of a drowned refugee child @xcite , police violence @xcite , or the recent departure of western military forces from afghanistan. image search keyword-based image search analyzing the content of images or videos has been studied for decades @xcite , pre-dated only by approaches relying on metadata and similarity measures @xcite . in a recent survey, @xcite categorize image features into color, texture, shape, and spatial features. current commercial search engines also index text found in images, surrounding text, alternative texts displayed when an image is unavailable, and their urls @xcite . also related to the retrieval of argumentative images is that of "emotional images", which relies on image features like color and composition @xcite . argumentation goes hand in hand with emotions, so that emotional features may be promising for retrieving images for arguments in the future. for lack of labeled data (argumentativeness plus emotionality), we start by exploiting keyword-based web search to retrieve images for arguments from the web, as did the earliest image search approaches (e.g., @xcite . argument search based on argument mining from texts (cf. @xcite , argument search engines aim to support decisionmaking and persuasion. conceptually, a query to an argument search engine may either name an issue without a stance @xcite (e.g., nuclear energy), or represent a conclusion for which supporting and attacking premises are to be retrieved (e.g., nuclear energy mitigates climate change). the first collection of arguments from the web is the internet argument corpus @xcite , containing 400,000 posts from an online debate portal. the first argument search engine, args.me, indexes a similar dataset of arguments @xcite . not relying on retrieval in collections of arguments, ar-gumentext @xcite first searches for documents relevant to a user's query in generic web crawls, and then mines arguments on the fly within retrieved documents. regarding the evaluation of argument search engines, judging the topic relevance of a retrieved text alone is insufficient, it must also be argumentative @xcite . research on argumentation has identified many further quality criteria for arguments @xcite ), yet few have been investigated for argument retrieval, and hardly anything has been said on the argumentative quality of images. ## image retrieval for arguments similar to textual argument retrieval, we define the task of retrieving images for arguments as follows: given a keyword query suggesting an issue or a claim for a topic, retrieve as two ranked lists those and only those images that can assist someone in (1) supporting and (2) attacking it. this definition presumes that a given user intends to search for images suitable to assist in persuading others, an intent that encompasses a diversity of real-life scenarios, such as sending a pertinent image to a friend, using it as a cover for a news article or blog post, or on a slide in a presentation. furthermore, it encompasses deliberation scenarios, such as forming an own opinion, and creating a collage for a school project. we expect search engines for argumentative images to offer facets as shown in figure 1 to meet such needs. to assess the relevance of an image to a given query, a three-fold judgment is required: topic relevance the image content is related to the query topic. this criterion corresponds to the notion of relevance in keyword-based image retrieval @xcite . ## stance-aware query expansion the basic hypothesis of our approach is that the task of image retrieval for arguments can be tackled effectively by the structure shown in figure 3a , which issues for a user's query several expanded queries to a keyword-based image search engine and then fuses the result lists to a list of pro and con images. however, we assume that the semantic capabilities of modern keyword-based image search engines can be harnessed further to already provide a classification of the images, thus separating pro and con images as well as omitting neutral ones. in particular we propose a stance-aware query expansion as depicted in figure 3b . it generates focused queries for both relevant stances (superscript + for pro andfor con), which are processed independently of each other, presuming that a sufficient diversity of expanded stance-aware queries recalls relevant images for each stance on the top ranks. in that case, the development of a post-retrieval image stance classifier can be omitted. the result lists of each stance are interlaced, i.e., composed by taking the first result of each list for a stance, then the second of each, and so on. we devise three methods: (1) appending always the same stance-indicating terms to the user's query, (2) appending sentiment-indicating terms that co-occur with the query's terms, and (3) appending topic-specific stance-indicating terms obtained from a text argument search engine. in retrieval tasks, human relevance judgments of retrieval results for a fixed set of topics allows for evaluating the effectiveness of competing retrieval models. known as the cranfield paradigm or trec-style evaluation @xcite , it is also employed in textual argument retrieval within the touché shared tasks @xcite . all our expansion methods retrieve results for the same set of queries, which are then pooled and manually judged with respect to their relevance to a given query's information need. to evaluate the methods presented in section 4, we employ a sample of 20 controversial questions from the touché 2020 task 1 test set @xcite , from which we derive one query each. 4 we calculate the number of relevant images in the methods' ten top-ranked images (precision@10) for both rankings of pro and con images (thus actually for 20 images), which is straightforward to interpret for multiple relevance levels. to ensure state-of-the-art keyword-based retrieval and a large image index, all methods retrieve images using the google image search. figure 4 shows the annotation interface for gathering relevance judgments. as described in section 3, we distinguish relevance on three levels: topic, argumentativeness, and stance. the first three options, the image being able to support the pro, contra, or both stances, indicate the image stance(s) and that it is argumentative and on topic. the fourth option, the image not being able to support a stance, indicates that the image is on-topic only. the final option indicates that the image has no stance and is neither argumentative nor on-topic. using the annotation interface of figure 4 , we collected 2988 annotations on 993 images foot_3 and 20 topics from 12 layperson annotators: three annotators per image and topic. the images' order was randomized for each annotator within a topic to avoid order biases. the annotator agreement is fair to moderate (fleiss' κ of 0.39; fleiss, 1981; figure 7 shows κ per topic). we compute a single label for each image and topic pair as per majority vote (two out of three), treating an annotation both as a vote for both pro and con, and assigning the groundtruth label both if there are at least two votes for both pro and con. 6 to ensure consistency in the face of difficulties reported by the annotators, we reviewed the ground-truth and changed the label for 223 images. the vast majority of changes (85%) were due to vagueness in our instructions: annotators labeled images that can introduce the topic or question but have no argumentative value otherwise (see figure 5 for examples) often as being able to support both stances though they should have labeled them as non-argumentative (neither). pros-cons positive-negative good-anti 0.35 0.44 0.48 0.52 0.62 0.64 0.92 0.95 0.94 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 precision@10 figure 6: precision@10 achieved by the three query expansion methods at three relevance levels, averaged across 20 topics and each topic's individual pro and con rankings. the dashed lines indicate the expected precision@10 of a random image stance classifier dividing the retrieved images into the pro and con rankings. the bars are overlaid, not stacked. ## evaluation this section reports on a comparative evaluation of the three query expansions methods by their retrieval effectiveness, followed by a topic-wise error analysis. last, we also carried out an investigation into the counterintuitive case of ambiguous images that can support both stances. ## conclusion this paper introduces the task of image retrieval for arguments, proposes with stance-aware query expansion a family of methods to tackle the task that exploits existing keyword-based image search technology, and carries out a first empirical analysis for the task using human annotations on 993 images and 20 topics. in the experiments, the basic goodanti heuristic, which expands a user query with the same two terms (good for pro images, anti for con ones), outperforms more sophisticated query expansion approaches that employ terms specific to the query's topic. an error analysis unveils a high topic-dependence of the retrieval effectiveness. finally, the paper investigates ambiguous images labeled as supporting both the pro and con stances. in cases where many neutral images are found, we suggest displaying them separately. relying on google's image search engine ensures a state-of-the-art retrieval model at the price of exact replicability. since commercial image search engines are subject to frequent changes it must be investigated how reproducible our results are. developing an open image retrieval system for arguments would enable laboratory evaluation. it would also allow for creating image representations tailored to argumentation, but requires creating and labeling an extensive image collection that covers a manifold of controversial topics. there is a lot of room to improve precision. filtering images by analyzing their web pages may avoid ones that merely show a product or otherwise fail to make a point. further analyses of the web pages with argument mining technologies could provide for a better stance classification of the images. also images search engine feature to retrieve similar images may be helpful in this regard. moreover, classifiers to identify persuasion techniques or emotions in images may prove beneficial to identify images for arguments. assessing the quality of argumentative images could improve the imagined search engine's evaluation and utility to users. to foster research, we run a corresponding shared task to be held as part of the touché 2022 lab. 7 this paper seeks to extend argument mining in general and argument retrieval in particular to considering images. other argument mining tasks (e.g., relation extraction in articles) could similarly be extended to account for the widespread use of images in argumentative texts or speechmaking.