An Investigative Study of Multi-Modal Cross-Lingual Retrieval (2020)

Abstract: We describe work from our investigations of the novel area of multi-modal cross-lingual retrieval (MMCLIR) under low-resource conditions. We study the challenges associated with MMCLIR relating to: (i) data conversion between different modalities, for example speech and text, (ii) overcoming the language barrier between source and target languages; (iii) effectively scoring and ranking documents to suit the retrieval task; and (iv) handling low resource constraints that prohibit development of heavily tuned machine translation (MT) and automatic speech recognition (ASR) systems. We focus on the use case of retrieving text and speech documents in Swahili, using English queries which was the main focus of the OpenCLIR shared task. Our work is developed within the scope of this task. In this paper we devote special attention to the automatic translation (AT) component which is crucial for the overall quality of the MMCLIR system. We exploit a combination of dictionaries and phrase-based statistical machine translation (MT) systems to tackle effectively the subtask of query translation. We address each MMCLIR challenge individually, and develop separate components for automatic translation (AT), speech processing (SP) and information retrieval (IR). We find that results with respect to cross-lingual text retrieval are quite good relative to the task of cross-lingual speech retrieval. Overall we find that the task of MMCLIR and specifically cross-lingual speech retrieval is quite complex. Further we pinpoint open issues related to handling cross-lingual audio and text retrieval for low resource languages that need to be addressed in future research.

## introduction cross-lingual information retrieval (clir) is an extension of the information retrieval (ir) task where query and documents are in different languages @xcite . the goal of clir is to retrieve documents matching a user's query to satisfy their information need. in general, a user would pose a query in their own language (l1), retrieve a document in a foreign language (l2) that is translated into the user's language l1. machine translation (mt) of some form is thus one of the fundamental components in enabling clir @xcite . clir has been the focus of much research since its definition in the 1990s. since this time significant progress have been made in clir, and in the associated research areas of automatic speech recognition (asr) and machine translation (mt). however not much work has been done in the area of multimodal cross-lingual retrieval (mmclir), apart from notable examples such as @xcite @xcite , which bring these topics together. with the increasing interest in information access for diverse multimodal content, there is a need to learn and provide better retrieval tools and technologies to support users, in their desire to satisfy their information needs and quest for new knowledge. the expanding volume and diversity of data made electronically available every day pushes the limits of ir research and development further to facilitate retrieval over different modalities, i.e., multi-modal ir @xcite . this work is a step in this direction to investigate and study the challenges and the performance of mmclir while combining individual component solutions of mt+ir+sp for mmclir, and in particular the situation where limited training resources for the technologies are available. the mmclir task rests around four main pillars which need to be addressed adequately both independently and in combination: 1. cross-lingualism: input queries and documents to be retrieved are in different languages; 2. document and query modalities: documents to be retrieved can be in different modalities than the query, but also differ among themselves e.g. text documents and audio recordings; 3. information retrieval: in which the ir mechanism depends on document indexing, query processing, ranking and retrieval, 4. low resource constraints: how to build effective models without having access to the resources typical for mt and speech systems is a major challenge for mmclir tasks. openclir challenge campaign: this benchmark challenge 1 focused on cross-lingual text and speech retrieval, under a low-resource data setting. in this challenge there was insufficient parallel data available to train state-of-theart mt and asr systems. in this task, queries are written (text) keywords in english and the documents are text or audio in swahili. the work presented in this paper was conducted within the scope of this challenge. we outline the data provided by the openclir task organizers later (see section 4) and report our results and findings from the openclir evaluation. we investigate a general mechanism for mmclir, which can be applied for other such similar low resource languages for which there is insufficient data to train effective mt, asr and ir systems. having this use case in mind we present our analysis of the challenges and alternative solutions guided by the aforementioned four pillars. the main contributions of our work are as follows: ## related work the strategy for crossing the language barrier between queries and documents in clir can be either query translation, document translation or both. document translation is the preferred method when users need to both search and access documents in their own language (l1) @xcite . in query translation, the query is translated into the target language (l2), and then used to retrieve indexed documents in the original language l2 @xcite . multiple approaches have been explored to address query translation (sub)task over the years. these can be divided into several categories: dictionary-based, mt-based, corpus-based and ontology-based @xcite . dictionary-based methods were predominant in early work on query translation @xcite @xcite . however, out-ofvocabulary (oov) issues may easily arise as these dictionaries are limited and require exact matches, thus the whole ir performance may be negatively impacted. in corpus-based approaches translations of keywords in l1 are extracted from parallel or comparable corpora in l2 based on statistical methods @xcite . improvements in mt systems mean that most recent work on clir has focused on the use of mt for query translation @xcite . state-of-the-art mt now uses neural approaches (nmt) @xcite . however, it is challenging to train effective nmt systems using limited amounts of parallel data, thus our work focuses on statistical mt approaches. recent work on mmclir has focused on document translation @xcite using smt and nmt, learning a shared embeddings space for both queries and documents @xcite , and learning better word translation probabilities using neural network @xcite . the authors found that retrieval results using smt for document translation are relatively better than nmt, possibly due to the limited nature of data @xcite . a neural network based approach has also been explored to estimate word translation probabilities for clir @xcite . the authors found that the neural network model estimates better probabilities for word translations than automatic word alignments alone, since using neural network they can encode the character sequences of input source words to generate translations of out-of-vocabulary words. following on from our overview, the approach adopted in this paper aims to address our task using a combination of dictionary and statistical mt due to the very limited amounts of bilingual training data available and initial findings that retrieval performance is relatively better when using smt rather than nmt for clir @xcite . ## approach for our experiments, we used an mt system to translate the input queries from english to swahili (described later in section 5). as the resources available for building translation models were very limited, we focused on translation of input queries rather than attempting to translate the target documents. we divided our investigation of the mmclir task into two phases: ## dataset & evaluation mechanism in this section we describe the dataset used to train mt and asr systems, and to tune the retrieval system to address the task of text and speech retrieval. we also describe the evaluation mechanism used for this task. ## automatic translation automatic translation (at) foot_2 between english and swahili is a challenging task due to the lack of parallel data available for training high-quality systems. furthermore, the specifics of the clir task, e.g. queries can be single words or phrases with specific constraints, impose additional constraints on how to approach the at task. ## speech processing searching for a textual query in speech documents is often performed on speech transcripts of the documents created using an asr system. in low-resource scenarios, it is difficult to build a high quality asr system for the target language due to the shortage of labelled speech corpora. to alleviate the quality of our asr system, we combined output of a keyword spotting system and a phoneme search system with asr output. ## information retrieval in this section we describe the different components of our ir system, and present the tools and resources that are used for the development of the ir components. ## results and analysis in this section we present our results on the analysis set for text-and speech-based retrieval before moving on to present our results on the evaluation set. foot_5 ## conclusion and future work in this work we investigated a mmclir task focusing on english-swahili search carried out within the openclir challenge. we examine solutions to several challenges for mmclir in the context of low-resource availability. we investigated two research questions and examined alternative at approaches for effective query translation. we build we anticipate that work on mmclir will open new avenues and increase the scope of future research and promote interesting new research collaborations and pathways as the amount of multi-modal content is expected to rise very significantly as we consume and interact with more applications and content @xcite . in the future we would like to be able to explore new language pairs, and already plan to work on building better mt and speech models to boost retrieval effectiveness.