Query Generation for Multimodal Documents (2021)

Abstract: This paper studies the problem of generatinglikely queries for multimodal documents withimages. Our application scenario is enablingefficient “first-stage retrieval” of relevant doc-uments, by attaching generated queries to doc-uments before indexing. We can then indexthis expanded text to efficiently narrow downto candidate matches using inverted index, sothat expensive reranking can follow. Our eval-uation results show that our proposed multi-modal representation meaningfully improvesrelevance ranking. More importantly, ourframework can achieve the state of the art inthe first stage retrieval scenarios

## introduction as more documents on the web are generated and consumed by mobile devices with cameras, documents are often multimodal, containing information in both text and image modalities. this poses a new challenge of finding relevance documents across modalities. more formally, the relevance of document, consisting of text t and image i, to the given query keywords q, should be modeled as a trimodal function f (q, t, i), rather than a simple lexical match between q and t (or, bm25 baseline) assuming the semantics of image i is fully represented by the surrounding text (or, paired-text assumption). prior research observes that paired-text assumption is often violated (henning and ewerth, 2017)for example, some semantics can be better captured in image modality, and may not (or, cannot) be described in text. meanwhile, bm25 baseline @xcite would fail to serve queries for such semantics. to overcome this limitation of bm25, one may model relevance from query as a trimodal function f (q, t, i) @xcite instead, but they require runtime invocation of f for the given query q with all potential document matches. this incurs a prohibitive runtime overhead, unacceptable for search engines finding results online. a common practice is to use a cheap bm25 ranking as a "first-stage retrieval", efficiently supported by inverted index, to quickly narrow down to a few candidate documents, then evaluate f (q, t, i). however, due to simple nature of bm25 using exact term matching, a document will be missed, if the query term is absent in t, even though it semantically matches image or another term in the text. our contribution is to keep first-stage retrieval as efficient as bm25, but enable multimodal semantic matching, using query generation (qg) before indexing. more specifically, we generate a likely query q from a joint modeling of t and i, to create an expanded text @xmath0, t ) pair has more lexical overlaps, or better paired than (q, t), for first-phase retrieval. specifically, we train a sequence-to-sequence model, such that given the representation of multimodal document, this model generates possible queries that users may ask to retrieve such document. this is analogous to doc2query @xcite approach used for first-stage retrieval of textual relevance ranking, though this model, dealing with text modality only, cannot apply to our problem of retrieving multimodal documents. for such multimodal representation for qg, a naive baseline is bimodal representation shown in figure 1a : we may assume (t, i), even when lexical overlaps are low, is semantically paired in the embedding space. note this is a relaxed version of paired-text assumption. given this relaxed assumption, common architecture of bimodal representation for qg consists of the encoders for image i and text t. each generates vector representation, which is later fused into a multimodal space, then decoded into a textual caption. specifically, we consider two strong baselines: (a) cross-modal representation, pretrained from a large-scale paired corpus of image and caption, such as lxmert (tan and bansal, 2019), vilbert @xcite , and @xcite , finetuned for our task, and (b) memory network structure @xcite . based on these baselines, we design bimodal qg combining the advantages of the two as a strong baseline. then, we extend into trimodal qg leveraging text, image and query (q, t, i). alternatively, we may further relax paired-text assumption and propose trimodal qg in figure 1b: (t, i) can be partially paired, where some semantics is conveyed in one modality. to deal with that challenge, the query given at training, helps "disentangle" shared and private semantics as additional loss terms. another role of query is improving image representation, to de-emphasize semantics not discussed in either text or query. in summary, our contributions are as follows: our work is closely related to the following three areas of research. ## bimodal baselines: lxmert and memory-based generator this section compares and contrasts two bimodal baselines: lxmert foot_0 and memory-based generator encode text t and image i into vector representations (section 3.1), then the two are aggregated into a multimodal representation (section 3.2), such that this vector can feed a decoder to generate a text sequence (section 3.3). specifically, we build bimodal qg baseline, combining lxmert representation and memory-based generator decoding. figure 2 shows overall architecture of our bimodal qg baseline. ## trimodal qg: query-aware representation our proposed bimodal qg partially contributes to relax paired-text assumption, but neither image and text representation is aware of queries. we argue that, queries carry rich semantics and contribute significantly to to relax paired-text assumption, specified as two key contributions c1 and c2 below. motivated, we propose two new loss functions l 1 and l 2 , addressing c1 and c2 respectively. ## experiment the goal of our evaluations is to validate the effectiveness of our approach in public dataset and real-world web search queries and settings. in particular, we have two research questions: we use public dataset for rq1 and real-life queries and quality annotation for ad-hoc web search task from naver for rq2. ## conclusion we study the problem of representing a multimodal document to be indexable for efficient first-stage. our contribution is posing the problem as trimodal qg to augment the given text, by proposing a trimodal joint representation of image, text, and query without paired-text assumption. we validate our approach over both public dataset and real-life web search data collected from commercial search engines.