Vector Space Interpolation for Query Expansion (2022)

Abstract: Topic-sensitive query set expansion is an important area of research that aims to improve search results for information retrieval. It is particularly crucial for queries related to sensitive and emerging topics. In this work, we describe a method for query set expansion about emerging topics using vector space interpolation. We use a transformer model called OPTIMUS, which is suitable for vector space manipulation due to its variational autoencoder nature. One of our proposed methods – Dirichlet interpolation shows promising results for query expansion. Our methods effectively generate new queries about the sensitive topic by incorporating set-level diversity, which is not captured by traditional sentence-level augmentation methods such as paraphrasing or back-translation.

## introduction in web-search scenario, users may input queries that are not offensive (or controversial) by themselves; but may leak controversial queries through auto-suggest or return offensive (or controversial) documents in the search engine results pages (serp). these queries, denoted as threat queries, often pertain to sensitive topics, such as antisemitism, and climate change. to minimize such inadvertent leakage, search engine companies identifies a list of potentially sensitive topics that such queries pertain to. then they build topic-wise classifiers to categorize queries into such topics, so that queries from sensitive topics can be handled specifically (such as post-processing the serp results). formally, a topic is defined as an emerging subarea of a broader sensitive issue that leads to adverse public relations issues and user experience. often emerging topics and trends become a hive of threat queries, for instance, the 2020 usa elections, and covid19 vaccination. such topics could be timesensitive as well (such as michael jackson's death, black lives matter movement). annotating a sizable amount of training data for these topics is challenging due to time-sensitivity, and ever-growing number of the topics. hence, our central motivation is to propose an efficient topic-sensitive query set expansion technique from a small set of annotated queries for emerging topics and trends. given a small number of queries about a sensitive topic, we aim to automatically generate a larger extensive set of queries while maintaining topical consistency. this expanded set could be later used to improve the classifier / post-processing techniques to provide better search results. initially, human annotators tag this initial set of queries (denoted as seed queries) as belonging to the a sensitive topic. the seed query set is assumed to be small (typically around ten queries), as it is impractical for annotators to create a large set of seed queries with the increasing number of sensitive topics. in literature, query expansion is performed using paraphrasing @xcite , and template transformation @xcite . however, these methods attempt to preserve the semantic meaning of the original query, which is not our intent. although some work have focused on improving the sentence-level diversity of the paraphrases @xcite , they are unable to broaden the coverage of the seed queries, where set-level diversity and generalization are crucial. in this paper, we propose a query expansion method based on vector space interpolation. we consider the interaction between the queries in the seed set that leads to extensive and completely new queries (within the sensitive topic), which is impossible to generate using traditional text augmentation techniques, such as paraphrasing or back-translation. our contributions are as follows: i) we propose a method for set expansion using vector space interpolation (specifically dirichlet interpolation) which ii) ensures set-level diversity and generalizes to create new queries within the given topic. ## vector space interpolation we use the variational autoencoder language model optimus @xcite for vector space interpolation. it consists of a bert encoder @xcite and a gpt2 decoder @xcite . the encoder and the decoder are linked through a latent connector, which serves as the bottleneck layer of the autoencoder. it is pre-trained on a large text corpus, where it learns to reconstruct a sentence x from the decoder, given that sentence x as input to the encoder. optimus learns to organize sentences according to high-level semantics (topic, sentiment, tense, etc.) in the intermediate latent space through the pre-training objective. thus, it allows easy manipulation of the dense sentence representations in the latent vector space. as argued in @xcite , this is possible because: i) sentences are densely represented in the latent space as a result of pretraining, and ii) kl regularization of the vae and the continuity property of neural networks allow latent vectors with similar semantics to be smoothly organized together. it is thus possible to combine two or more sentences by performing vector operations over their latent representations. the resultant vector could then be used to perform controlled generation through the decoder. we denote a topic t as consisting of seed queries s 1 , s 2 , .., s n . the latent vectors from optimus corresponding to the seed queries are z 1 , z 2 , .., z n . we combine the latent vectors in different ways to create the modified latent vector z. the modified latent vector z is then used in the decoder to generate new synthetic queries. we use the following methods to manipulate the latent vectors of the seed queries: linear interpolation: the interpolation technique used in @xcite . the modified latent vector z is created from a linear combination of two vectors, where the weights sum to 1. ## conclusion in this paper, we proposed a method for query expansion using different vector space interpolation techniques. we use the optimus variational autoencoder language model to perform the task of query expansion using linear, polar, and dirichlet interpolation methods. we also propose several automatic and human evaluation metrics to compare the different interpolation techniques. the dirichlet interpolation method shows the strongest results and is able to create set-level diverse queries about the given sensitive or emerging topic.