Language Modelling with NMT Query Translation for A mharic- A rabic Cross-Language Information Retrieval (2019)

Abstract: This paper describes our first experiment on Neural Machine Translation (NMT) based query translation for Amharic-Arabic Cross-Language Information Retrieval (CLIR) task to retrieve relevant documents from Amharic and Arabic text collections in response to a query expressed in the Amharic language. We used a pre-trained NMT model to map a query in the source language into an equivalent query in the target language. The relevant documents are then retrieved using a Language Modeling (LM) based retrieval algorithm. Experiments are conducted on four conventional IR models, namely Uni-gram and Bi-gram LM, Probabilistic model, and Vector Space Model (VSM). The results obtained illustrate that the proposed Uni-gram LM outperforms all other models for both Amharic and Arabic language document collections.

## introduction information retrieval (ir) is the activity of retrieving relevant documents to information seekers from a collection of information resources such as text, images, videos, scanned documents, audio, and music as well. these resources can be structured, indexed, and navigated through language technology (lt), which includes computational methods that are specialized for analyzing, producing, modifying, and translating text and speech @xcite . the increasing necessity for retrieval of multilingual documents in response to a query in any language opens up a new branch of ir called cross-language information retrieval (clir). its goal is to accept the query in one language, transform it into a searchable format and provide an interface to allow a user to search and retrieve information in different languages as per their information need @xcite . the amharic language is the official language of ethiopia spoken by 26.9% of ethiopia's population as mother tongue and spoken by many people in israel, egypt, and sweden. arabic is a natural language spoken by 250 million people in 21 countries as the first language and serving as a second language in some islamic countries. ethiopia is one of the nations, which have more than 33.3% of the population who follow islam, and they use the arabic language to teach religion and for communication purposes. arabic and amharic languages belong to the semitic family of languages, where the words in such languages are formed by modifying the root itself internally and not simply by the concatenation of affixes to word roots @xcite . nowadays, it is widely used to solve clir problems for many language pairs. however, much of the research on this area has focused on european languages despite these languages being very rich in resources. so this study is aimed to develop the nmt query translation based amharic-arabic clir system. an essential part of clir is mapping between query and document collections by translating queries to the target document language or the source document to the target document language. we follow the first approach to translate the query words by using a pre-trained nmt model. for the purpose of this translation, we have constructed a small parallel text corpus by modifying the existing monolingual arabic and its equivalent translation of amharic language text corpora available on tanzile @xcite , as amharic-arabic parallel text corpora are not available for mt task. the rest of the paper is organized as follows. clir approaches are discussed in section 2. related works are reviewed in section 3. the proposed clir approach based on lm is described in section 4. resources and configurations of experiments for evaluating the system and the results are detailed in section 5, followed by a conclusion in section 6. ## clir approaches in clir, the query and the document collection needs to be mapped into a common representation to enable users to search and retrieve relevant documents across the language boundaries @xcite . based on the resources used to map the query and the documents in different languages, clir approaches can be categorized as; dictionarybased approach, latent semantic indexing (lsi), machine translation (mt) approach, and probabilistic-based approach @xcite . ## related works most of the researchers have studied clir works related to different language pairs. however, the only work reported on amharic and arabic languages pair is "dictionary based amharic-arabic cross-language information retrieval system" @xcite . the performance was affected by incorrect translation due to outof-dictionary words and unnormalized arabic words; specifically, diacritics not mapped with the dictionary words, and the query was formulated by selecting words available in the dictionary. some of the prominent works reported on amharic and arabic languages paired with other languages are discussed below. in bilingual amharic-english search engine @xcite , limitation of word coverage includes a large-size commercial bilingual dictionary and on-line bilingual dictionary for query translation and short data size. the system can perform best only on the selected query terms which are available in the dictionary. the lack of electronic resources such as morphological analyzers and large mrd have forced a. @xcite to spend considerable time to develop those resources themselves. solving the problem of word sense disambiguation will enhance the effectiveness of clir systems. andres @xcite , studied to choose the best dictionary for cross-lingual word sense disambiguation (clwsd), which is focused only on english-spanish cross-lingual disambiguation and the disambiguation task is dependent on the coverage of dictionary and corpus size. query suggestion that exploits query logs and document collections by mapping the input query of french language to queries of english language in the query log of a search engine by w. @xcite showed the strong correspondence between the french input queries and english queries in the log, but languages may be more loosely correlated. for example, english and amharic. m.al-shuaili and m.garvalho (2016), proposed a technique to map characters automatically from different languages into english, without human interference and prior knowledge of the language. while mapping helps transliterations of oov names to have the same or, at least, very similar pronunciations in any language, word structure, and writing direction add complexity for character mapping and originality of the names also affects the result of character mapping. in the corpus-based clir system for amharic and english language pairs @xcite , the size and the quality of document constructed highly affected the performance of the system. nigussie eyob (2013), have developed a corpus-based afaan oromo-amharic clir system to enable afaan oromo speakers to retrieve amharic information using afaan oromo queries. the scarcity of aligned corpus creates a problem of translation disambiguation, and the dictionary is limited to translate words only. f. @xcite , explores combinationof-evidence techniques for clir using three types of statistical translation models: context-independent token translation, token translation using phrase-dependent con-texts, and token translation using sentencedependent contexts. experiments on retrieval of arabic, chinese, and french documents using english queries show that no one technique is optimal for all queries, but statistically significant improvements in mean average precision (map) over strong baselines can be achieved by combining translation evidence from all three techniques. in all the above-mentioned cases, the key element is the mechanism to map between languages that can be encoded in different forms as a data structure of the query and documentlanguage term correspondences in an mrd or as an algorithm, such as an mt or machine transliteration system. nowadays, the direction of clir is on utilizing neural approaches. quing @xcite , proposed a neural approach to english-chinese clir, which consists of two parts; bilingual training data and kernel-based neural ranking model (k-nrm). external sources of translation knowledge are used to generate bilingual training data which is then fed into a kernel-based neural ranking model. the bilingual training approach outperforms traditional clir techniques given the same external translation knowledge sources. k-nrm learns translation relationships from bilingual training data by capturing soft-matches from bilingual term pairs and combine softmatches to generate final scores with a set of bins. kazuhiro seki (2018) explores a neural network-based approach to compute similarities of english and japanese language text. they focus on nmt models and examine the utility of an intermediate state. the intermediate state of input texts is indeed beneficial for computing cross-lingual similarity outperforming other approaches, including a strong machine translation baseline. many of clir works related to neural approaches are focused on neural ranking methods not directly using nmt for query translation. in this work, an nmt based query translation is employed to map between amharic and arabic languages using traditional ir ranking methods. ## proposed amharic-arabic clir system traditional ir in cross-language environment settings mainly allows measuring the similarity between the information need (query) in source language and collection of documents in both languages. in a clir environment, queries and documents are written in two different languages. in order to match terms between the two languages, a retrieval system needs to establish a mapping between words in the query vocabulary and words in the document vocabulary. deep learning nmt is a recent approach of mt that produces high-quality translation results based on a massive amount of aligned parallel text corpora in both the source and target languages. deep learning is part of a broader family of ml methods based on artificial neural networks (memoq, 2019). it allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. these methods have improved the state-of-the-art research in language translation @xcite . nmt is one of the deep learning end-to-end learning approaches to mt that uses a large artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model. the advantage of this approach is that a single system can be trained directly on the source and target text no longer requiring the pipeline of specialized systems used in statistical mt. many companies such as google, facebook, and microsoft are already using nmt technology @xcite . nmt has recently shown promising results on multiple language pairs. nowadays, it is widely used to solve translation problems in many languages. however, much of the research on this area has focused on european languages despite these languages being very rich in resources. our research has been focused on resolving query translation ambiguity. the opensource nmt system, called opennmt @xcite , which is an open-source toolkit for nmt, is used to construct the amharic-arabic nmt model. the pre-trained model is used to translate the text in amharic to the arabic language. once the query is translated into arabic, standard ir algorithms can be used to retrieve the relevant documents from amharic and arabic document collections. as shown in figure 1 , prepossessing (tokenization, punctuation, and stop-word removal) is done for amharic and arabic document collections first. then language models are produced for both languages, which will be used to estimate the query likelihood of the given query. the search module is used to input amharic language queries and retrieve relevant documents in both languages. a sample screenshot of the proposed system displaying relevant documents as a list of a hyperlink for a sample user query is shown in figure 2 . a sample amharic text which is preprocesed after sentence spliting, tokenizing words, punctuation and stop-word removal is shown in a language model, which is a probability of words in each document p(w|d) in the collection, is used to rank the documents according to the probability of generating the query. the query likelihood is given by p (q|d) = ∏ m @xmath0. but this will assign zero probability for the words that are not available in the specific documents. therefore the following maximization technique, which is lm with jelineck-mercer smoothing @xcite , is used to optimize the likelihood of a given query, as shown in equation 1 . ## experiments and results to design, develop, and maintain effective ir system, evaluation is very crucial as it allows the measurement of how successfully an information retrieval system meets its goal of helping users fulfill their information needs. there are two approaches for evaluating the effectiveness of ir systems: (i) user-based evaluation and (ii) system-based evaluation. in the system-based evaluation method, several human experts evaluate the system to prepare a set of data that can be reused in later experiments. the user-based evaluation method quantifies the satisfaction of users by monitoring the user's interactions with the system @xcite . in this work, the focus is on system-oriented evaluation that focuses on measuring how well an ir system can rank the most relevant documents at the top for a given user query. to evaluate the proposed amharic-arabic clir system, test collections (document corpus, search queries, and relevance judgments) have been prepared as bench-marked datasets are not available. amharic is used as a source language to retrieve target language documents in arabic as well as in amharic. experiments are conducted on four conventional ir models, namely uni-gram and bigram lm, probabilistic model, and vsm. unigram lm is the bag-of-words model where the probability of each word only depends on that word's own probability in the document. bigram lm denotes n-gram models with @xmath1. it is assumed that the probability of observing the i th word w i in the context history of the preceding i-1 th word can be approximated by the probability of observing it in the preceding n -1 th word. the probabilistic model makes an estimation of the probability of finding if a document d j is relevant to a query q, which assumes that the probability of relevance depends on the query and document representations. vsm is an algebraic model for representing queries and documents as vectors of identifiers. relevant judgments can be created using crowdsourcing @xcite , @xcite , which is a timeconsuming and expensive task. therefore, we considered the topmost ranked documents and took the union of all intersections between unigram and bigram, unigram and vsm, bigram and probability, and probability and vsm. if the number of documents in this set is less than 10, the symmetric difference of the uni-gram model is taken. as it is shown in figure 3 , the documents (amtext1.txt, am-text43.txt, amtext27.txt, amtext39.txt, am-text26.txt, amtext41.txt, amtext81.txt, am-text67.txt, amtext28.txt, amtext34.txt) are selected as the top-ranked documents relevant for the query "ምስጋና ለአላህ ይገባው የዓለማት ጌታ ለኾነው" (all praise is due to allah, lord of the worlds). for evaluation, we configure our test collection as 75 amharic search queries, 114 arabic and equivalent translation of amharic documents (each verse of the quran is organized as a single document), and relevant judgments are extracted using equation 2. the description of this test collection is shown in the most frequently used and still the dominant approach to evaluating the performance 1 https://trec.nist.gov/ of information retrieval systems are precision and recall. precision is defined as the proportion of retrieved documents that are actually relevant, and recall is defined as the proportion of relevant documents that are actually retrieved. both precision and recall can be expressed as; p @xmath2. the evaluation metrics used in this work are; map and recall. map and recall are computed as the sum of average precision (ap) of each query divided by the number of queries and sum of average recall of each query divided by the number of queries, respectively. the other measurement technique used for evaluation is discount cumulative gain (dcg) that measures the usefulness or gain of a document based on its position in the result list. the gain is accumulated from the top of the result list to the bottom, with the gain of each result discounted at lower ranks. dcg adopted from @xcite is accumulated at a particular rank position p as given in equation 4 . we also used normalized ndcg to measure the usefulness of documents at first, fifth, and tenth position of ranked lists. evaluation results of all models are presented in