Paraphrase Acquisition from Image Captions (2023)

Abstract: We propose to use image captions from the Web as a previously underutilized resource for paraphrases (i.e., texts with the same “message”) and to create and analyze a corresponding dataset. When an image is reused on the Web, an original caption is often assigned. We hypothesize that different captions for the same image naturally form a set of mutual paraphrases. To demonstrate the suitability of this idea, we analyze captions in the English Wikipedia, where editors frequently relabel the same image for different articles. The paper introduces the underlying mining technology, the resulting Wikipedia-IPC dataset, and compares known paraphrase corpora with respect to their syntactic and semantic paraphrase similarity to our new resource. In this context, we introduce characteristic maps along the two similarity dimensions to identify the style of paraphrases coming from different sources. An annotation study demonstrates the high reliability of the algorithmically determined characteristic maps.

## introduction two texts that convey "semantically equivalent information" via a different wording are called paraphrases, or said to be in a paraphrase relation. a variety of natural language processing tasks have been approached by using paraphrase resources or paraphrase generation and detection algorithms: textual entailment @xcite @xcite , semantic similarity @xcite , machine translation @xcite ), text reformatting @xcite , or question answering @xcite . paraphrasing can help to improve sentence compression and text summarization @xcite as well as natural language understanding @xcite . to tackle the task of paraphrasing, large amounts of paraphrases usually are helpful. hence, paraphrase acquisition has gained popularity over the flourine-containing durable water repellent makes a fabric waterresistant. water bead on a fabric that has been made non-wetting by chemical treatment. recent years and paraphrase datasets have been created manually, (semi-)automatically, and via distant supervision resulting in different levels of "naturalness", diversity (topic, genre, language, etc.), granularity (word, sentence, or passage level), and scale (see section 2 for details). for model training, @xcite strongly emphasize the importance of negative examples that are lexically similar but semantically inappropriate as paraphrases. with image captions, we propose an as yet underutilized source of natural paraphrases. the underlying hypothesis is that different captions for the same image form paraphrase candidates because the captions may describe the same image content. we study the phenomenon of 'image reuse and captioning' in the english wikipedia (figure 1 shows an example) by mining captions of images that have been reused. carefully optimized filtering heuristics for captions and associated images yield a large set of captions, which we analyze qualitatively and quantitatively for their usefulness as paraphrases. the set of mined image captions forms our wikipedia-ipc dataset, foot_0 , foot_1 which contains caption pairs as paraphrases along with their image source three different quality levels; 30,237 caption pairs are gold-, 229,877 are silver-, and 656,560 are bronze-quality paraphrase candidates. image content icons, symbols, pictograms pictures, graphics, paintings, maps, schematics image purpose ideological (used to convey atmosphere and sentiment) specific, explanatory, pragmatic (used in discourse) brazilian flag up arrow wikimedia community logo yellow card icon for plain stage in cycling stage race motorway exit icon pictograms of olympic sports athletics paraguayan goalkeeper justo villar was awarded as the best goalkeeper of the tournament. romain grosjean was once again the center of controversy when he collided on the first lap with mark webber. ac-130 spectres were highly effective during the battle. j. c. a. corea was born in the historic town of chilaw, on the west coast of sri lanka. february 10: new delhi becomes india's capital. toronto's mars discovery district is a centre for research in biomedicine. dostana was filmed almost entirely in miami. the armenian genocide (pictured) was the first event officially condemned as ¨crimes against humanity¨. "i blame myself" was co-written by ariel rechtshaid (pictured), who additionally assisted with the songwriting and production of each track on night time, my time. semi hollow-body electric guitar gibson es-335 has a ¨solid center blockï nside a body. this ¨circular warming stripes¨graphic depicts average global temperature using chronologically ordered, concentric coloured rings. water bead on a fabric that has been made non-wetting by chemical treatment. dikes are used to protect the rice paddy fields from the channels of saltwater which overflow during high tide. the camouflage grouper epinephelus polyphekadion is the type-host of p. viscosus; the photograph shows a specimen from new caledonia. the stone is against the half-timbered wall of exmewe hall, on st. peter's square, ruthin. a sub-surface metropolitan line a stock train (left) passes a deep-tube piccadilly line 1973 stock train (right) in the siding at rayners lane. the x-38 crv prototype makes a gentle lakebed landing at the end of a july 1999 test flight at the dryden flight research center with a fully deployed parafoil. tylosaurus proriger mounted skeleton in the rocky mountain dinosaur resource center in woodland park, colorado. the usefulness of an image caption as a paraphrase depends strongly on its corresponding image. ## related work paraphrase corpora to collect high-quality paraphrases, some studies applied manual acquisition. for instance, for the webis-cpc-11 @xcite , crowdworkers were asked to rewrite one of 4,096 passage-level literature excerpts "so that the rewritten version has the same meaning, but a completely different wording and phrasing." the resulting 7,859 pairs form one of the few passage-level paraphrase corpora. later, also @xcite crowdsourced sentences with the same meaning as a shown sentence from some tweet for their pit-2015 dataset (18,862 sentence pairs, 5,641 considered paraphrases). since crowdsourcing is costly to scale, automatic paraphrase acquisition often simply relies on machine translation. for instance, @xcite scrambled words and forth-and-back translated sen-tences to obtain a gold dataset of 108,463 paraphrase pairs and a silver dataset of 656,000 pairs. also, the paranmt-50m dataset (wieting and gimpel, 2018) with 30 million "strong" paraphrases (assessment by the authors), was created by translating sentences from english-czech parallel pairs. trying to combine the advantages of manual and automatic methods (quality vs. scale), most paraphrase acquisition methods use distant supervision. for instance, @xcite suggested aligned sentences from monolingual parallel corpora as paraphrases-an idea later used by @xcite to create the microsoft research paraphrase corpus (5,801 manually annotated sentence pairs, 3,900 considered paraphrases). instead of monolingual sources, @xcite exploited bilingual parallel corpora by aligning phrases that translate into the same pivot phrase in another language. based on this idea, the english/spanish ppdb 1.0 @xcite and its extension to 23 additional languages @xcite with an english portion of 7.6 million lexical (i.e., synonymous) and 68.4 million phrasal paraphrases form the biggest paraphrase collection. translation pivoting was also used for the opusparcus @xcite and tapaco @xcite datasets. opusparcus was created from movie and tv subtitles in six european languages. the automatically filtered training set contains 7 million english sentence and phrase pairs ("good" or "mostly good" paraphrases according to the authors) while the manually labeled development and test sets contain 3,088 pairs (1,997 paraphrases). analogously, tapaco is created from the multilingual tatoeba data, a crowdsourced collection of sentence translations. 3 the english part of tapaco contains 158,000 sentences that are clustered in 62,000 paraphrase sets. instead of translation, @xcite used a different pivoting method for their twitter news url corpus. they mined tweets that contain the same hyperlink and labeled 51,524 respective sentence pairs via crowdsourcing. our paraphrase acquisition methodology is also based on a pivoting technique, but the pivot media are images. captions and paraphrases our idea to use images from the web as pivots for distant supervision paraphrase acquisition is inspired by studies on crowdsourced caption datasets. for instance, @xcite extracted a collection 3 https://tatoeba.org/eng/about ## caption-based paraphrase acquisition we collect captions of images that are used on the english wikipedia, grouping those captions that describe the same image. this section outlines the mining and filtering steps. data source: wikipedia as a source for paraphrases, we use captions from wikipedia articles. wikipedia is a collection of well-formulated, human-written text that is constantly being improved by a community of volunteer editors. the guidelines for the use of images on wikipedia encourage editors to write a caption to explain the relevance of an image in a particular context, resulting in 60% of images having a caption. we use the english wikipedia for the paraphrase acquisition (wikimedia, september 1st, 2022). 4 two versions of wikipedia dumps are available for the paraphrase acquisition-a reduced dataset with all articles, templates, media descriptions, and primary meta pages and a full-size dump with all pages, including their full revision history. image and caption mining wikimedia commons is a service which hosts free-to-use images and other media files which are used across all wikimedia projects (like wikipedia). the markup language for wikipedia articles, wikitext, features the extended image syntax, which allows editors to insert images into wikipedia pages from wikimedia commons. the specified image is referenced by its unique identifier in wikimedia commons, which is a concatenation of a generic media type prefix (e.g., image) followed by a colon and its (also unique) filename. the extended image syntax allows the specification of style properties, metainformation, and a caption. if a caption is present, we store it along the image reference. however, not all images in wikipedia pages are included by the use of the extended image syntax. wikitext allows the use of (user-created) style templates, often with inconsistent image reference syntax. infoboxes on wikipedia are template-based tables placed at the beginning of an article that contain a collection of the most important information about its subject. they are commonly used in wikipedia-20% of all the pages contain one. important to us is that one third of the infoboxes contain one or even multiple images, and about 56% of these images have a caption. wikitext represents infoboxes as lists of key-value pairs whose sets of allowed keys and values are defined in templates. there are close to 2,000 different infobox templates based on semantic categories of an article. the responsible keyword for image referencing varies among these templates, and to cover all cases is not feasible. thus, we identified the most commonly used keyword which is "image", followed by an optional counter for multiple images. these captions are mined as individual image-caption pairs. no wikitext parser other than wikipedia's own implementation (which is monolithically intertwined with wikipedia's software stack) is capable of reliably parsing all the different template syntaxes. our heuristic for image mining from infoboxes yields an increase of 24% of image references. in addition to a caption, an image reference may also have an alternative text that accurately describes the visual content of an image. alternative texts are intended to serve as a substitute in case the user is visually impaired, or the image cannot be displayed correctly. according to wikipedia's policy, all images are obliged to have an alternative text, except those with a purely decorative purpose. we extract them as paraphrase candidates, too. altogether, 355 million caption pairs are mined that we subsequently filter as follows (see image filter a necessary condition for an image to be useful is that it has been referenced at least twice, so we discard all image captions whose images have only one reference. as illustrated in ## corpus analysis and evaluation we analyze and evaluate our paraphrase corpus based on the quantification of paraphrase similarity, for which we select well-known syntactic and semantic measures, and compare our corpus with five state-of-the-art paraphrase and three image-caption corpora from the literature. therefore, we propose a new measure ∆ sem,syn that measures paraphrase sophistication based on the difference of semantic and syntactic similarities. to validate the proposed measure we analyze the correlation between human judgments and automatically computed semantic and syntactic similarities. ## conclusion and future work we propose a new approach to use image captions as a resource for paraphrasing. from wikipedia, we extracted 30,237 caption pairs in gold quality, 229,877 in silver quality, and 656,560 in bronze quality for our new wikipedia-ipc dataset. as part of our analysis, we found that many caption pairs are "sophisticated" paraphrases in the sense of being semantically similar but dissimilar at the lexical and syntactic levels. we have introduced a respective measure for assessing paraphrase sophis-tication based on semantic and syntactic similarity. the new measure correlates well with manual judgments, and we have shown that paraphrases from different sources have individual characteristics along the two similarity dimensions. studying large sets of paraphrases with characteristics required by a specific task (e.g., sophisticated vs. only word order changed) will be useful for task-specific paraphrase recognition and generation approaches, and for natural language understanding as a whole. in future work, we plan to extend the mining process to larger web resources than the wikipedia and to apply image analysis as a new equivalence criterion. identifying highly similar but not identical images may help to identify even more captionbased paraphrase candidates.