Document Translation vs. Query Translation for Cross-Lingual Information Retrieval in the Medical Domain (2020)

Abstract: We present a thorough comparison of two principal approaches to Cross-Lingual Information Retrieval: document translation (DT) and query translation (QT). Our experiments are conducted using the cross-lingual test collection produced within the CLEF eHealth information retrieval tasks in 2013–2015 containing English documents and queries in several European languages. We exploit the Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) paradigms and train several domain-specific and task-specific machine translation systems to translate the non-English queries into English (for the QT approach) and the English documents to all the query languages (for the DT approach). The results show that the quality of QT by SMT is sufficient enough to outperform the retrieval results of the DT approach for all the languages. NMT then further boosts translation quality and retrieval quality for both QT and DT for most languages, but still, QT provides generally better retrieval results than DT.

## introduction multilingual content has been growing significantly in the last few years simultaneously with rapid internet access growth all over the world. monolingual information retrieval task allows users to find information in documents that are written in the language that they use to write their queries. this ignores a vast amount of information that is represented in other languages. cross-lingual information retrieval (clir) breaks this language barrier by allowing users to look up information that is represented in documents written in languages different from the language of the query. we reinvestigate the effectiveness of two principal approaches to clir: document translation (dt) and query translation (qt). the existing comparison studies of the two approaches are outdated (e.g. @xcite and do not reflect the current advances in machine translation (mt). even in very recent works, the authors have blindly assumed that dt is superior to @xcite , giving the argument that in dt, the text is translated in a larger context compared to the translation of short isolated queries in qt. the larger context should help in translation disambiguation and better lexical selection during translation, which should subsequently lead to better retrieval results. this hypothesis needs to be revised, taking into consideration the significant improvement of machine translation quality in recent years, despite the strong practical disadvantages of dt over qt: dt is computationally expensive and hard to scale (every document needs to be translated into each supported language and then indexed) while qt is performed in query time and only a short text (the query) is translated into the document language. in this work, state-of-the-art statistical machine translation (smt) and neural machine translation (nmt) systems are deployed for document translation and query translation to investigate their effect on retrieval quality in the cross-lingual setting. the experiments are conducted using the cross-lingual test collection produced within the clef ehealth tasks on patient-centered information retrieval in 2013-2015 extended with additional relevance assessments and manual query translations @xcite . though this is a very specific domain and the results cannot be thoughtlessly generalized to other domains, the choice of this test collection was motivated by two facts: first, it provides resources for large-scale experimentation (1 million in-domain documents, 166 queries in 8 languages, thorough relevance assessment). second, the medical domain in mt has been well studied @xcite , and there are enough resources to develop wellperforming mt systems for multiple languages. ## related work in clir, documents and queries are written in different languages. the traditional term-matching retrieval methods require both documents and queries to be represented in the same language. in practice, either the queries need to be translated into the document language (qt), or the documents need to be translated into the query language (dt). not many studies and experiments have been conducted in order to compare these two approaches. oard (1998) investigated the performance of dt, qt, and a hybrid system combining both. they found that the system translating english queries into german (the document language) outperformed the system translating the documents from german into english (the query language). they hypothesized that documents, which are typically longer than queries, provide more contextual and linguistic information that helps reduce translation ambiguity and thus improves translation quality. @xcite presented a hybrid dt/qt system, which averaged the retrieved document scores from dt and qt systems and thus outperformed both of them. @xcite employed a two-step method where qt was first used to retrieve a limited number of documents that were translated into the query language and reranked by their dt retrieval scores. pirkola (1998) presented a new method for clir, which was referred to as structured queries. the idea was that a document containing one possible translation candidate of a query term is more relevant than a document that contains multiple translations of that term. this probabilistic structured queries approach was also applied to cross-language speech retrieval @xcite . @xcite also exploited alternative translations of query terms. their experiments showed that combining multiple translations outperformed the selection of one best translation. nikoulina et al. (2012) investigated reranking smt translation hypotheses towards better clir performance and showed that smt systems are usually trained to give the best results in terms of translation accuracy, adequacy, and fluency. however, an improvement will be achieved when they are optimized towards retrieval quality. we followed this approach in our previous work and introduced a richer set of features and adopted the hypothesis reranker for multiple languages in the medical domain @xcite . several recent papers employed methods based on deep learning. @xcite presented an unsupervised clir approach employing shared cross-lingual word embedding model, which was trained using monolingual data only. they used those embeddings to translate query terms word by word into the document language. @xcite trained nmt model for clir using out-domain data and synthetic data (created by translating in-domain monolingual english into german) to retrieve answers to german questions from english collection in the technical domain (askubuntu and stackoverflow). clir in the medical domain has been investigated within the series of clef share/ehealth labs since 2013 which focused on improving access of laypeople (non-medical experts) to reliable medical information @xcite @xcite @xcite @xcite . in this paper, we compare the performance of both qt and dt using the traditional smt and state-of-the-art nmt methods trained on the same data to make the comparison as fair as possible. we present a novel approach for nmt model selection that is optimized towards clir performance and investigate the effect of morphological pre-and post-processing on the performance on clir. ## data two types of data were used in our experiments: the data for training, tuning, and testing mt (section 3.1) and the clir test collection (section 3.2). ## retrieval system the document collection is indexed using terrier @xcite , an open-source tool for information retrieval experiments. for retrieval, we use terrier's implementation of the language model with bayesian smoothing and dirichlet prior @xcite with the default value of the smoothing parameter. ## machine translation systems in this section, we provide details on training the smt and nmt systems used in the clir experiments. the smt systems fully replicate the work by @xcite ; we only provide the most important information. the nmt systems are described in full detail. ## conclusions we presented a comparative study between querytranslation (qt), and document translation (dt) approaches in the cross-lingual information retrieval (clir) task. to conduct this study, we investigated various mt systems and their configurations and performed a thorough large-scale evaluation based on the test collection produced within the clef ehealth tasks on patient-centered information retrieval during 2013-2015, and extended with additional relevance assessments. we experimented with both statistical and neural mt paradigms. the smt systems for qt were specifically trained and tuned to translate medical search queries. for dt, we trained two smt systems: the first one was built to produce word forms, and the second one to produce word lemmas. we then used these two systems to translate the test collection into seven european languages. furthermore, we performed lemmatization and stem-ming on the collection that was translated using the smt system that produces word forms. the results showed that a well-tuned qt system outperforms dt, which is a positive result with an important impact on practical applications. so far, the qt approach has been preferred mainly for efficiency reasons (less space and computation needed). our experiments suggest that this approach is even more effective (better retrieval results). we also investigated the effect of using neural machine translation, which is now considered the state-of-the-art in many domains. this completely new paradigm in machine translation tends to improve the fluency of generated output (which is appreciated by humans), but often mismatches content and adequacy (which might hurt the performance in ir). in our experiments, nmt improved retrieval results in both qt and dt, but the qt approach is still superior, so the results are consistent with the findings from the smt experiments. however, we emphasize that the way we trained our mt systems is very domain-specific (medical domain), and we made use of a vast amount of medical data (monolingual and parallel). this makes our comparative study very task-oriented. when dealing with general domain test collection, some search terms might have a different meaning in different domains. for example, the word "development" probably in most cases means in medicine the growth or spread of a disease (or a tumor), while in the general domain we can not say without a context, and in that case, the need for linguistics information in the queries will be more important to solve the translation ambiguity. this should be considered when comparing qt and dt approaches; thus, the reader should be careful when drawing the same conclusion of this work while working on a different domain.