Embedding Meta-Textual Information for Improved Learning to Rank (2020)

Abstract: Neural approaches to learning term embeddings have led to improved computation of similarity and ranking in information retrieval (IR). So far neural representation learning has not been extended to meta-textual information that is readily available for many IR tasks, for example, patent classes in prior-art retrieval, topical information in Wikipedia articles, or product categories in e-commerce data. We present a framework that learns embeddings for meta-textual categories, and optimizes a pairwise ranking objective for improved matching based on combined embeddings of textual and meta-textual information. We show considerable gains in an experimental evaluation on cross-lingual retrieval in the Wikipedia domain for three language pairs, and in the Patent domain for one language pair. Our results emphasize that the mode of combining different types of information is crucial for model improvement.

## introduction the recent success of neural methods in ir rests on bridging the gap between query and document vocabulary by learning term embeddings that allow for improved computation of similarity and ranking @xcite . so far vector representations in neural ir have been confined to textual information, neglecting information beyond the raw text that is potentially useful for retrieval and is readily available in many data situations. for example, meta-textual information is available in the form of patent classes in prior-art search, topic classes in search over wikipedia articles, or product classes in retrieval in the e-commerce domain. a straightforward approach to use such meta-textual information to improve retrieval would require an exact match of meta-textual categories of queries and relevant documents. however, in the majority of cases the sets of categories assigned to queries and documents overlap only at a small percentage, for both relevant and irrelevant documents. thus, more sophisticated techniques are needed to aid similarity computation and ranking by meta-textual category information. in this paper, we show how to apply neural embedding methods to meta-textual categories. we show that meta-textual information is an incomplete representation of queries and documents and does not suffice for a standalone computation of similarity and ranking. however, an incorporation of pre-trained embeddings of meta-textual categories into a neural learning-to-rank approach yields significant improvements over a learning-to-rank approach that uses text-only embeddings @xcite . in our approach, enhanced embeddings are created by concatenating text embeddings with meta-textual category embeddings, and a fully connected weight layer is learned on top of the concatenation of the enhanced embeddings of a query-document pair by a deep multilayer perceptron that is optimized for pairwise ranking. we present experiments on three different language pairs for cross-lingual retrieval in the wikipedia domain, and show improvements of up to 2 ndcg points by incorporating meta-textual embeddings in a learning-to-rank framework. additional experiments on a single language pair in the patent domain also show improvements of up to 1.3 ndcg points by incorporating embeddings for patent classifications, and up to 6.3 ndcg points if separate models for text and meta information are combined in an ensemble. ## related work embeddings of meta-textual information via so-called "side constraints" has been done successfully in neural machine translation applications @xcite @xcite @xcite @xcite . metainformation such as domain labels, product categories, or politeness level, has been incorporated in various ways, including a choice of feeding a meta-information tag at the source side or the target side, the choice of attaching it to each sentence or to each word, and the choice of learning embeddings vi learning-to-rank training or via devising a specific representation for the meta-information embeddings. an incorporation of meta-information into learning-to-rank approaches for ir has previously been presented by @xcite , albeit not in a neural ir framework. the neural learning-to-rank architecture used in our work builds on @xcite . while they present an option to use learned text embeddings for cosine similarity scoring, our approach focuses on direct learning of a similarity function in order to learn a proper weigthing of textual and meta-textual embeddings. ## data sets for our clir experiments, we conduct experiments on two different domains, namely wikipedia articles and patents. for the wikipedia retrieval task, we extend data from @xcite with category in- ## experiments to evaluate the efficacy of our model, we conduct experiments on wikipedia data for three language pairs, and on patent data for a single language pair. we conduct additional experiments in the patent domain to evaluate alternative models that combine information differently. finally, we integrate standard tf-idf and evaluate if our models scale up to realistic retrieval scenarios in prior art search. ## conclusion we presented an approach to incorporate retrieval-relevant meta-textual information into learning-to-rank models. such information is readily available in many retrieval tasks and can be integrated by learning dense embeddings that allow for inexact matching. our main motivation was to investigate the relative benefit of incorporating meta information into a manageable neural architecture presented in previous work @xcite . if more advanced and potentially better performing embedding models like bert @xcite or elmo @xcite can be further enhanced by meta-textual information is an open question for future work. our results in the wikipedia domain show that adding category information yields significant gains across several language pairs. the contribution of meta information increases with the size of the documents' lists, but is also dependent on the amount of meta information. in the patent domain, adding patent classification embeddings shows significant improvements over the textual models. existing models that make use of the tf-idf metric can be further improved if the system components are combined properly. integrating the tf-idf metric also makes the ranking models applicable to large scale retrieval if a machine translation system is available to translate queries into the target language. finally, our experiments on two different data sets and up to three language pairs showed that combining different types of information in retrieval is not straightforward. joint learning worked particularly well for integrating noisy wikipedia categories, while for the cleaner hierarchical patent ipc classifications we observed the largest gains when individual models are are combined in an ensemble setup. on patent data, joint models were significantly better than single models, but weighted ensembles outperformed joint models by a large margin.