Textual Representations for Crosslingual Information Retrieval (2021)

Abstract: In this paper, we explored different levels of textual representations for cross-lingual information retrieval. Beyond the traditional token level representation, we adopted the subword and character level representations for information retrieval that had shown to improve neural machine translation by reducing the out-of-vocabulary issues in machine translation. We found that crosslingual information retrieval performance can be improved by combining search results from subwords and token level representation. Additionally, we improved the search performance by combining and re-ranking the result sets from the different text representations for German, French and Japanese.

## introduction cross-lingual information retrieval (clir) systems commonly use machine translation (mt) systems to translate the user query to the language of the search index before retrieving the search results @xcite @xcite . traditionally, information retrieval and machine translation systems convert search queries to tokens and n-grams level textual representation @xcite @xcite . modern neural machine translation (nmt) systems have shown that subwords and character representations with flexible vocabularies outperform fixed vocabulary token-level translations @xcite @xcite . this study explores the shared granularity of textual representations between machine translation and cross-lingual information retrieval. textual representations of varying granularity encode queries differently, resulting in more diverse and robust search retrieval. potentially, subwords and character-level representations are less sensitive to irregularities in noisy user-generated queries, e.g. misspellings and dialectal variants. ## related work neural machine translation had shown to outperform older paradigm of statistical machine translation models significantly and even "achieved human parity in specific machine translation tasks" @xcite @xcite . moving from fixed token-level vocabulary to a subword representation unlocks open vocabulary capabilities to minimize out-of-vocabulary (oov) issues foot_0 . byte-pair encoding (bpe) is a popular subword algorithm that splits tokens into smaller units @xcite . this is based on the intuition that smaller units of character sequences can be translated easily across languages. for instance, these smaller units appear when handling compound words via compositional translations, such as for instance, subword units can better handle compound words via compositional german to english translations, @xmath0. suwbords can also cope with translations where we can easily copy or translate part of the source tokens or translate cognates and loanwords via phonological or morphological transformations, e.g. @xmath1. while bpe reduces the oov instances, it requires the input to be pre-tokenized before applying the subword compression. alternatively, @xcite proposed a more languageagnostic approach to subword tokenization directly from raw string inputs using unigram language models. completing the whole gamut of granular text representations, @xcite explored characterlevel neural machine translations that do not require any form of pre-processing or subword or token-level tokenization. they found that multilingual many-to-one character-level nmt models are more efficient and can be as competitive as or sometimes better than subwords nmt models. moreover, character-level nmt can naturally handle intra-sentence code-switching. in the context of clir, they will be able to handle mixed language queries. following this, @xcite found that using byte-level bpe vocabulary is 1/8 the size of a full subword bpe model. a multilingual nmt (many-to-one) setting achieves the best translation quality, outperforming subwords models and character-level models. while finer granularity of text representations was exploited for machine translation, to our best knowledge, information retrieval studies have yet to study the impact of using these subword representations on traditional information retrieval systems @xcite @xcite . however, many previous works have leapfrogged to using fully neural information retrieval systems representing text with underlying various subword representations and neural dense text representation. often, these neural representations are available in multilingual settings in which the same neural language model can encode texts in multiple languages. @xcite explored using the popular multilingual bidirectional encoder representations from transformers (bert) model to learn the relevance between english queries and foreign language documents in a clir setup. they showed that the model outperforms competitive non-neural traditional ir systems on a few of the sub-tasks. alternatively, previous researches have also used a cascading approach to machine translation and traditional ir where (i) the documents are translated to the foreign languages with neural machine translation and/or (ii) the foreign queries are trans-lated before retrieval from the source document index @xcite @xcite . saleh and pecina (2020) compared the effects of statistical machine translation (smt) and nmt in a cascaded traditional clir setting. they found that the better quality translations from nmt outperforms smt and translating queries to the source document language that achieved better ir results than using foreign language queries on an index of translated documents. although fully neural ir systems are changing the paradigm of information retrieval, traditional ir (e.g. tf-idf or bm25) approaches remain very competitive and can still outperform neural ir systems for some tasks @xcite . in this regard, we follow up on the cascading approach to machine translation and information retrieval on traditional ir systems. this study fills the knowledge gap of understanding the effects of subword representation in traditional ir indices. ## experiments we report the experiments on different textual representations on traditional ir in a cross-lingual setting using a large-scale dataset derived from wikipedia @xcite . @xcite focused their work on a supervised re-ranking task using relevance annotations. we use those annotations from the same wikipedia dataset to perform the typical retrieval task. the dataset was designed so that the english queries are expected to retrieve the wikipedia documents in the foreign languages, and the foreign documents with the highest relevance are annotated with three levels of relevance. formally, the ground truth data is a set of tuples: (english query, q, foreign document, d and relevance judgement r, where @xmath2, 1, 2}). we note that the wikipedia documents in the dataset are not parallel (i.e. not translations of each other) but they are comparable in nature depending on the varying amounts of contributions available on the official wikipedia dumps across different languages. for our study, we use the german, french and japanese document collections and report retrieval performance of english queries translated to these languages. 3 the wikipedia corpus came pre-tokenized, so we had to detokenize the documents foot_3 @xcite before putting them through the subword tokenizer. we used pre-trained sentencepiece subword tokenizers used by the opus machine translation models(tiedemann and thottingal, 2020) foot_4 . additionally, we emulated the typical pre-processing steps for character-level machine translation and split all individual characters by space, replacing the whitespaces with an underscore character. the english queries were translated using the same opus machine translation models. 6 although these machine translation models are open source and free to use under a permissive cc-by license, it takes a significant amount of gpu computation and major changes to the huggingface api @xcite to efficiently translate the query samples parallelized inference. we will release the modified code for parallel gpu inference and translation outputs for the data used in this experiment for future convenience to improve the replicability of this paper. ## results for german and french documents, the intuition behind the poor performance of the character-level representation can be attributed to the meaningless and arbitrary nature of the unordered bag of characters. whereas in japanese, with its mix of syllabic and logographic orthography, the individual characters can potentially encode crucial semantic information. we can see that both search result expansion and re-ranking techniques can improve the final search results for some languages. ## conclusion we explored the different granularity of textual representations in a traditional ir system within the clir task by re-using the subword representation from the neural machine translation systems. our experiments in this paper provide empirical evidence for the underwhelming impact of subwords in traditional ir systems for latin-based languages as opposed to the advancements that subword representation has made in machine translation. 7 in some scenarios, it is possible to achieve better clir performance by combining and expanding retrieval results of token and subword representations. we conducted the experiments in this study using well-formed queries and documents. our intuition is that a combination of the different textual representations can improve the robustness of the indexing and retrieval systems in realistic situations with noisier data (e.g. queries spelling or translations errors). for future work, we want to explore similar experiments with noisy e-commerce search datasets. 8