A Multilingual Reading Comprehension System for more than 100 Languages (2020)

Abstract: This paper presents M-GAAMA, a Multilingual Question Answering architecture and demo system. This is the first multilingual machine reading comprehension (MRC) demo which is able to answer questions in over 100 languages. M-GAAMA answers questions from a given passage in the same or different language. It incorporates several existing multilingual models that can be used interchangeably in the demo such as M-BERT and XLM-R. The M-GAAMA demo also improves language accessibility by incorporating the IBM Watson machine translation widget to provide additional capabilities to the user to see an answer in their desired language. We also show how M-GAAMA can be used in downstream tasks by incorporating it into an END-TO-END-QA system using CFO (Chakravarti et al., 2019). We experiment with our system architecture on the Multi-Lingual Question Answering (MLQA) and the COVID-19 CORD (Wang et al., 2020; Tang et al., 2020) datasets to provide insights into the performance of the system.

## introduction recent advances in open domain question answering (qa) have mostly revolved around machine reading comprehension (mrc) @xcite . the mrc task is to read and comprehend a given text and then answer questions based on it. our monolingual mrc approach @xcite has the capability of being applied to train many language models (lms) such as bert @xcite and roberta @xcite . we achieve the 2nd rank 1 on the google natural questions @xcite ) leaderboard 2 . in this paper, we expand our approach by introducing new multilingual capabilities using models such as multilingual-bert (m-bert) @xcite and xlm-r @xcite . this addition has the capability of transcending language boundaries to 104 languages. figure 1 shows examples of qa pairs from the mlqa dataset @xcite . to the best of our knowledge, this is the first published demo of a multi-lingual qa system. we achieve this by introducing a novel multilingual component to our qa gaama (go ahead, ask me anything) @xcite pipeline. we introduce m-gaama, a new system that performs cross-lingual mrc where a given question and context can be in the same or different languages. the system extracts the answer from the context language. then, the demo utilizes the sota ibm watson machine translation widget to return the answer translated in the desired language of the user 3 . this breaks the language barrier for users who don't understand the given source text but want their question answered effectively and accurately. in addition, we also show how m-gaama can be used in downstream tasks by incorporating it with cfo @xcite , in an end-to-end qa system and demo. we show that this can be extended to perform multilingual qa by utilizing a language identifier to first gather the (target) language in which what record company did kesha sign with? after failing to negotiate with lava records and atlantic records in 2009, kesha signed a multialbum deal with 𝑅𝐶𝐴 𝑅𝑒𝑐𝑜𝑟𝑑𝑠 +++++++++++++++++ through dr. luke's imprint. having spent the previous six years working on material for her debut album, she began putting finishing touches to the album with luke and max martin. english ¿con qué compañía discográfica firmó kesha? poco después, kesha firmó un contrato por varios discos con rca a través de luke, después de haber sido buscada por lava records y el sello de flo rida, como también atlantic records. 𝑅𝐶𝐴 ++++++ había notado sus seguidores en los medios sociales cuando negoció su contrato, por lo tanto se basó en construir su primer sencillo, «tik tok», ofreciendo la canción en myspace en julio. ## related work few other qa demos exist; bertserini @xcite , leverages the anserini ir toolkit @xcite to extract relevant documents given a question, then uses bert-based techniques @xcite to extract the correct answer. however, their demo is designed to perform only mono-lingual english qa. the gaama and cfo @xcite demos also only performs english qa. in contrast, m-gaama and our downstream end-to-end-qa task perform cross-lingual qa. several cross lingual large scale representations have been created by training a large scale transformer @xcite based masked language model on text in multiple languages. the use of pretrained multilingual language models such as m-bert @xcite , xlm @xcite , and xlm-r @xcite achieve the previous sota on cross-lingual tasks including question answering @xcite . we train our underlying mrc system with these pre-trained language models and achieve results that are consistently as strong as prior work. many datasets for english mrc have been introduced with annotated wikipedia documents including @xcite @xcite . fewer resources are available for the cross-lingual setting. the mlqa @xcite dataset contains parallel instances in 7 languages where the context is found in wikipedia. the tydiqa @xcite dataset containes instances in 11 languages. however, tydiqa is not parallel and it only has instances where the question and context are in the same language. for an individual case with exposure lying between 1 e and 2 e , the likelihood function for an incubation observation was 12 (\ncommission of china, reporting an incubation time ## data we test the multilingual capabilities incorporated into our qa system by running experiments on the mlqa dataset @xcite . the dataset consists of seven languages: english (en), spanish (es), german (de), arabic (ar), hindi (hi), vietnamese (vi), and chinese (zh). to achieve a multilingual parallel qa benchmark the authors apply a novel alignment strategy on wikipedia articles by identifying wikipedia sentences with the same meaning in multiple languages. passages containing these sentences are then presented to the annotators who write questions that are now answerable in multiple languages. we consider this a good resource for evaluating multilingual capabilities on different pairs of languages (e.g., context (c) in english, question (q) in german) due to the parallel q/c pairs available in the corpus. in addition, we use squad 1.1 @xcite , which is significantly larger, but only contains english data, for training in a zero-shot scenario. we also explore qa on covid-19 articles in a zero-shot scenario to show the relevance and importance of multilingual qa in current events. covidqa 0.1 @xcite ) contains 124 question and document pairs. the dataset comprises of (question, scientific article, exact answer) triples that have been manually created from the literature review page of kaggles covid-19 open research dataset challenge @xcite . they manually identified the exact answer span as a verbatim extract from the document. we converted their data into squad format for our experiments. we create a multilingual covid-19 qa dataset using machine translation. we translate both the questions and context in spanish and chinese. we align the gold answer between the english and the translated dataset by marking the gold answer with pseudo-html tags prior to translation. we recovered the translated answers for all questions. an example of a qa pair in english and spanish is shown in figure 2 . ## demo in this section we describe the interface for our m-gaama demo. we then show an end-to-end-qa demo as an example that builds upon m-gaama using prior work @xcite . m-gaama is a grpc @xcite server which wraps our lms for mrc. m-gaama provides an mrc interface which can answer questions in over 100 languages. we use m-bert and xlm-r lms to drive m-gaama's multilingual support. in addition, we provide a language translation component made available as a javascript widget foot_0 to allow the user to see the answer in the question language or any other language of choice. the m-gaama interface weaves the components together using the reactjs framework foot_1 . providing m-gaama as a grpc server allows it to be quite flexible. this enables it to seamlessly transition between being a standalone system and integrating with larger systems. we show this via the downstream end-to-end-qa task described below. end-to-end-qa builds upon m-gaama, with a full ir-mrc pipeline. information retrieval is obtained using an elasticsearch index foot_2 for each language foot_3 . the user can ask a question in any language for which an index exists. the language of the question is identified using the 'langid' toolkit @xcite to determine the appropriate index. the appropriate index is then searched for documents in the target language. these documents are then evaluated together with the user's question by m-gaama. finally, answer spans are de-duplicated and sorted by score before being returned to the user. the end-to-end-qa demo weaves these components together using the cfo framework @xcite , which is a novel approach for orchestrating services. ## model architecture our mrc qa model accepts a single query-document pair as its input and produces a span from the document along with a prediction score as its output. the underlying qa model is based on @xcite . the base layer of the qa system encodes the question and the candidate paragraph using the cross-lingual m-bert @xcite and xlm-r @xcite representations. an output feed forward layer is added on top of the base layer to produce 3 sets of scores: scores at each token offset marking the likelihood of an answer chunk (1) starting at this offset, (2) ending at this offset, and (3) the entire sequence marking the likelihood of the question being answerable given the current context. ## experiments we experiment with several multi-lingual models on the mlqa test set prior to integration in our system. we explore zero-shot learning as in prior work @xcite by training and fine-tuning on squad 1.1 @xcite for m-bert and the xlm-r qa models. refer to @xcite for additional details about model architecture and implementation. we train our models using the huggingface code foot_4 with the default parameters except 3e-5 learning rate, 2 training epochs, 32 batch size and, 790 warmup steps. we show results for the cross-lingual task (xlt), where the question and context are in the same language (e.g. question (q) and context (c) in chinese) on the left side of we also provide additional analysis for the mlqa results by showing the difference per each question type in further, we show the value of having a multilingual model by also exploring qa for covid-19 using the @xcite dataset on english and translated data in spanish and chinese. the ability to answers questions in other languages is especially important in this use-case finally, while the best performing model is xlm-r large, there is merit to including the m-bert model in the demo due to its reduced size which makes deployment more scalable. we use our m-bert model for runtime experiments. a single x86-64 intel r core is used as the cpu whereas one nvidia r tesla r v100 is used as the gpu. for brevity we show results for a random subset of five context question language pairs in ## conclusion in this paper we present our m-gaama demo, an interface for interacting with our multilingual qa mrc system. to the best of our knowledge we are the first to present a qa demo with multilingual capabilities in over 100 languages. we enable the user to be able to ask a question in one language, find the answer in another language, and with the use of machine translation the user can see the answer in the question language or another desired language. we also show how m-gaama can be used in a downstream task in our end-to-end-qa demo. finally, we show that our system achieves results that are consistently as strong as prior work on the mlqa dataset @xcite using xlm-r-large on all seven languages. it can also be used to perform qa in current events via the cord-19 covid-19 @xcite dataset. in the future we plan on experimenting with additional qa datsets such as natural questions @xcite and tydiqa @xcite .