P ar C our E : A Parallel Corpus Explorer for a Massively Multilingual Corpus (2021)

Abstract: With more than 7000 languages worldwide, multilingual natural language processing (NLP) is essential both from an academic and commercial perspective. Researching typological properties of languages is fundamental for progress in multilingual NLP. Examples include assessing language similarity for effective transfer learning, injecting inductive biases into machine learning models or creating resources such as dictionaries and inflection tables. We provide ParCourE, an online tool that allows to browse a word-aligned parallel corpus, covering 1334 languages. We give evidence that this is useful for typological research. ParCourE can be set up for any parallel corpus and can thus be used for typological research on other corpora as well as for exploring their quality and properties.

## introduction while ≈7000 languages are spoken @xcite , the bulk of nlp research addresses english only. however, multilinguality is an essential element of nlp. it not only supports exploiting common structures across languages and eases maintenance for globally operating companies, but also helps save languages from digital extinction and fosters more diversity in nlp techniques. there are extensive resources that can be used for massively multilingual typological research, such as wals @xcite , glottolog @xcite , babelnet @xcite or http://panlex.org . many of them are manually created or crowdsourced, which guarantees high quality, but limits coverage, both in terms of content and languages. we work on the parallel bible corpus (pbc) @xcite , covering 1334 languages. more specifically, we provide a wordaligned version of pbc, created using state-of-theart word alignment tools. as word alignments themselves are only of limited use, we provide an interactive online tool foot_0 that allows effective browsing of the alignments. the main contributions of this work are: i) we provide a word-aligned version of the parallel bible corpus (pbc) spanning 1334 languages and a total of 20m sentences @xcite . for the alignment we use the state-of-the-art alignment methods @xcite and eflomal @xcite . ii) we release parcoure, a user interface for browsing word alignments, see the multalign view in figure 1 . we demonstrate the usefulness of parcoure for typological research by presenting use cases in §6. iii) in addition to browsing word alignments, we provide an aggregated version in a lexicon view and compute statistics that support assessing the quality of the word alignments. the two views (multalign and lexicon views) are interlinked, resulting in a richer user experience. iv) parcoure has a generic design and can be set up for any parallel corpus. this is useful for analyzing and managing parallel corpora; e.g., errors in an automatically mined parallel corpus can be inspected and flagged for correction. ## features parcoure's user facing functionality can be divided into three main parts: multalign and lexicon views and interconnections between the two. ## experimental setup corpus. we set up parcoure on the pbc corpus provided by @xcite . the version we use consists of 1758 editions (i.e., translations) of the bible in 1334 languages (distinct iso 639-3 codes). pbc aligns bible editions on the verse level by using verse-ids that indicate book, chapter and verse (see below). although one verse may contain multiple sentences, we do not split verses into individual sentences and consider each verse as one sentence. retrieval. elasticsearch 2 is a fast and scalable open source search engine that provides distributed fulltext search. the setup is straightforward using an easy-to-use json web interface. we use it as the back-end for parcoure's search requirement. we find that a single instance is capable of handling the whole pbc corpus efficiently, so we do not need a distributed setup. for bigger corpora, a distributed setup may be required. we created two types of inverted indices for our data: an edge-ngram in-2 https://www.elastic.co/ dex to support search-as-you-type capability and a standard index for normal queries. alignment generation. simalign (jalili sabet et al., 2020) is a recent word alignment method that uses representations from pretrained language models to align sentences. it has achieved better results than statistical word aligners. for the languages that multilingual bert @xcite supports, we use simalign to generate word alignments. for the remaining languages, we use eflomal @xcite , an efficient word aligner using a bayesian model with markov chain monte carlo (mcmc) inference. the alignments generated by simalign are symmetric. we use atools foot_1 and the grow-diag-final-and heuristic to symmetrize eflomal alignments. lexicon induction. we exploit the generated word alignments to induce lexicons for all 889,111 language pairs. to this end, we consider aligned words as translations of each other. for a given word from the source language, we count the number of times a word from the target language is aligned with it. the higher the number of alignments between two words, the higher the probability that the two have the same meaning. we filter out translations with frequency less than 5%. ## backend design an overview of our architecture can be found in figure 4 . the code is available online. 4 parallel data format. we use the pbc corpus format @xcite : each verse has a unique id across languages / editions, the verse-id. the verse-id is an 8-digit number, consisting of two digits for the book (e.g., 41 for the gospel of mark), three digits for the chapter, and two digits for the verse itself. there are separate files for each edition. in each edition file, a line consists of the id and the verse, separated by a tab. indexing. we identify a pbc verse using the following format: {verse-id}@{language-code}-{edition-name}. we use this identifier to save and retrieve sentences with elasticsearch. in addition, we store all metadata identifiers within elasticsearch. thus, we can search for a sentence by keyword, sentence number (= verse-id), language code, or edition name. parcoure also supports the corpus alignment encoding (ces) foot_3 format. one can download parallel corpora in ces format and use our tools to adapt them to parcoure's input format. alignment computation. since eflomal's performance depends on the amount of data it uses for training, we concatenate all editions to create a bigger training corpus for languages that have more than one edition. if language l 1 has two, and language l 2 three different editions, then the final training corpus for this language pair will contain six aligned edition pairs. system architecture. parcoure is built on top of modern open source technologies, see figure 4 . the back-end uses the flask web framework, 6 gunicorn web server, foot_5 and elasticsearch. 8 the frontend utilizes the bootstrap css framework, foot_7 and the d3 visualization library. 10 since all these tools are free and open-source, there is no restriction on setting up and releasing a new parcoure instance. to extract word alignments, one can use any tool, such as eflomal, fast align or simalign. performance improvements. for good runtime performance, we precompute the word alignments. regarding lexicon, given a query word and a target language, parcoure first looks for a precomputed lexicon file; if it does not exist, par-coure obtains the translations for the query word online. to accelerate the translation process, par-coure employs python's multiprocessing library. the number of cpu cores is decided online based on the number of editions available for source and target languages. for a corpus with 1334 languages, we will end up with 890,445 alignment files and the same number of lexicon files. we cache alignment / lexicon files to speed up access. we use the last recently used (lru) cache replacement algorithm. ## parcoure use cases languages differ in how they encode meanings/functions. there are various aspects that make such differences an interesting problem when dealing with a dataset that has good coverage of the entire variation of the world's languages. (i) many such differences between languages are not widely acknowledged in linguistic theory, so to document the extent of variation becomes a discovery of sorts. for example, the fact that interrogative words might distinguish between singular and plural (figure 6 ) turns out to be a typologically salient differentiation @xcite . (ii) the variation of linguistic marking is even stronger in the domain of grammatical function, like the differentiation between the interrogative and relative pronoun in figure 6 . (iii) in lexical semantics, parcoure supports the investigation of how languages carve up the meaning space differently (cf. figure 5 ), especially when it comes to the ≈1000 low-resource languages covered in pbc. massively parallel texts are an ideal resource to investigate such variation @xcite . grammatical differences between languages, like differences in word order, have a long history in research on worldwide linguistic variation @xcite . however, being able to look at the usage of word order in specific contexts (and being able to directly compare exactly the same context across languages) is only possible by using parallel texts. for example, specific orders of more than two elements can be directly extracted from the parallel texts, like the order of demonstrative, numeral and noun "these two commandments" in figure 7 @xcite . for lack of space, we describe four more use cases only briefly: grammatical markers vs. morphology as devices to express grammatical features (figure 8 ); differences in how languages use gramfigure 6 : use case 2, grammatical differentiation. english "who" has three different translations in this spanish example: relative pronoun ("que"), and singular ("quién)" and plural ("quiénes") interrogative pronoun. matical case (figure 9 , ablative/dative in latin can correspond to five different cases in croatian); and exploration of paraphrases (figure 10 ). see the captions of the figures for more details. ## extension to other corpora our code is available on github and can be generically applied: you can create a parcoure instance for your own parallel corpus. parallel corpora are essential for machine translation (mt); parcoure's functionality is useful for analyzing the quality of a parallel corpus and the difficulty of the translation problem it poses. we give three examples i) incorrect sentence alignments can be identified, e.g., cases in which a target sentence is matched with the merger of two sentences in the source: cf. figure 11 where a short sentence in english is aligned with german and french sentences that also contain a second sentence that is missing in english. this functionality is particularly helpful for mined parallel corpora that tend to contain erroneous sentence pairs. ii) suppose an mt system trained on the parallel corpus makes a lexical error in a particular context c by mistranslating source word w s with target word w t . the lexicon view can be consulted for w s and the user can then click on the erroneous target word w t to get back to a multalign view of aligned sentence pairs containing w s and w t . she can then analyze why the mt system mismatched c with these contexts. examples of the desired translation are easy to find and inspect to support the formation of hypotheses as to the source of the error. iii) for multi-source approaches to mt @xcite @xcite , parcoure supports the inspection of all input sentences together. the mt system output can also be loaded into parcoure for a view that contains all input sentences and the output sentence. since any of the input sentences can be responsible for an error in multi-source mt, this facilitates analysis and hypothesis formation as to what caused a specific error. ## conclusion progress in multilingual nlp is an important goal of nlp and requires researching typological properties of languages. examples include assessing language similarity for effective transfer learning, injecting inductive biases into machine learning models and creating resources such as dictionaries and inflection tables. to serve such use cases, we have created parcoure, an online tool for browsing a word-aligned parallel corpus of 1334 languages, and given evidence that it is useful for typological research. parcoure can be set up for any other parallel corpus, e.g., for quality control and improvement of automatically mined parallel corpora.