EM Corpus: a comparable corpus for a less-resourced language pair M anipuri- E nglish (2021)

Abstract: In this paper, we introduce a sentence-level comparable text corpus crawled and created for the less-resourced language pair, Manipuri(mni) and English (eng). Our monolingual corpora comprise 1.88 million Manipuri sentences and 1.45 million English sentences, and our parallel corpus comprises 124,975 Manipuri-English sentence pairs. These data were crawled and collected over a year from August 2020 to March 2021 from a local newspaper website called ‘The Sangai Express.’ The resources reported in this paper are made available to help the low-resourced languages community for MT/NLP tasks.

## introduction the web is immense, free, and available to all @xcite . several studies have proposed the use of the web as a corpus for teaching and research @xcite @xcite @xcite . languages such as english and chinese are widely published and are well-equipped with resources and tools. availability of data for lowresource languages on the web is increasing day by day @xcite contributing hugely to bridge the gap between high-resource and lowresource languages. in addition, it is important to mention the language in discussion states #ben-derrule @xcite to minimize the existing divide of languages in nlp. in this paper, our work is to equip a less-resourced language pair, manipuri-english with resources. our objective is to increase the size of available data for manipuri-english language pairs. our goal is to build a sentence-level comparable corpus for manipuri-english 2 from a newspaper website 1 the corpus is available from http:// lepage-lab.ips.waseda.ac.jp/en/projects/ meiteilon-manipuri-language-resources/ 2 the codes from iso 639-2 for these languages are as follows: manipuri (mni) and english (eng) called "the sangai express". 3 we introduce the creation of a comparable corpus named "ema-lon manipuri corpus", (translation: our mother tongue manipuri corpus) abbreviated as the em corpus, of the low-resourced language pair, manipuri-english. we report on the method for creating the comparable corpus. we also tried to extract parallel corpus from our comparable data. additionally, we provide the table that maps the corresponding glyph points to its unicode codepoints for manipuri. the structure of the paper is as follows. section 2 describes previous work. sections 3 and 4 describes the characteristics of the language and the data. section 5 presents the methodological aspects. section 6 provides the details of the experiment and its analysis. section 7 concludes and proposes future directions. ## related work several works on the web as a corpus @xcite @xcite @xcite for many languages have been reported from the past decades @xcite . the use of web-based manipuri corpus has been reported by @xcite for the identification of reduplicated multiword expression (mwe) and multi-word named entity recognition (ner). pmindia is yet another crawled data set of 13 indian languages with english. this data set includes manipuri-english language pair data. indiccorp, sourced from news crawls, is a large monolingual corpus of 11 indian languages from two different language families (indo-aryan branch and dravidian) @xcite . some of the familiar datasets obtained from web crawls are the leipzig corpus @xcite , commoncrawl, and the oscar project @xcite , none of which contains the manipuri-english language pair in it. @xcite analyses different types of bilingual corpora, ranging from parallel, noisy parallel, comparable, very-non-parallel corpora. types of comparable corpus includes: (i) nonsentence-aligned, non-translated bilingual documents that are topic-aligned. example, newspaper articles that are published on the same date in different languages, and (ii) non-aligned sentences that are mostly bilingual translations of the same document. our work is close to the former. ## manipuri (meiteilon) manipuri, locally known as meiteilon, is an indian language from the sino-tibetan language family. it is highly agglutinative in nature. manipuri follows the sov (subject-object-verb) syntax structure. as the predominant language of the indian state manipur, manipuri has about two million native speakers. as a language classified as 'vulnerable language' by unesco @xcite , it is one of the two indian languages listed in the 8th schedule of the indian constitution as endangered. manipuri has two writing systems: eastern nagari script (also known as the bengali script) and meitei mayek. we use manipuri written in eastern nagari script for all of our works. again, manipuri is a low-resourced language that has not been explored much in computational linguistics. one of the reasons being the limited amount of available resources. in this paper, we aim to bridge this gap by sharing our resources publicly. ## ema-lon manipuri corpus (em corpus) the amount of resources for manipuri-english language pair is limited for performing machine translation (mt)/natural language processing (nlp) tasks @xcite . for example, there are 41,669 sentences monolingually and 7,419 parallel sentences with english in the open-sourced monolingual and parallel data from the pmindia dataset foot_1 @xcite . other sources include tdil-dc foot_2 , where the data is available upon an undertaking agreement. a standard site such as opus foot_3 @xcite is limited in the coverage of low-resource asian and south asian languages including, manipuri. this motivated us to create our comparable corpus. em corpus is built for manipuri-english language pair. this corpus is created by collecting news articles daily from a newspaper website known as "the sangai express," which is available in both languages. an average of 14,000 sentences is crawled for this language pair daily. the reported data is being collected from august 2020 to march 2021, as shown in figure 1 . the domain of the em corpus includes general articles, news on state, national and international affairs, sports and entertainment news, and the editorial. the english articles are topic-aligned with the manipuri articles, however, they are not the exact bilingual translation of each other but rather the summary or the gist of the manipuri news. the monolingual datasets contain 1.88 million manipuri sentences and 1.45 million english sentences and the parallel corpora contain 124,975 sentences. the number of words per sentence in manipuri and english is reported to be 17 and 23 monolingually and, 21 and 26 in parallel. it is to note that the number of word types in each language reflects the number of sentences and the structure of the language: it is natural that the more the sentence pairs, the higher the number of word types as reported in ## methodology in this section, we introduce the creation of em corpus and extraction of parallel corpus from the comparable data. the news updates in 'the sangai express' are available in a section-based format and, each section contains articles in an infinite scroll format. the request for the lists of urls follows a simple form, and so we source our data with a web scraper for each language which we built. since the class in the html of each article corresponds to each other, document alignment was straightforward. figure 1 shows the statistics of the data collection obtained per month from august 2020 to march 2021. to extract the content of the articles from the html, we use beau-tifulsoup foot_4 @xcite which is a rich python library for parsing html/xml documents, which on inspection performs well in extracting the body of the articles. additionally, we use crontab @xcite to automate our news crawl. we use hunalign @xcite , a sentence aligner that aligns bilingual text based on the heuristics of sentence-length information and a bilingual dictionary (if available). it is to be noted that hunalign does not deal with changes of sentence order like most sentence aligners. due to the absence of the dictionary for manipuri, we use the automatic dictionary built based on the alignment. we retain 1-1 alignments obtained from filtering sentences with a threshold that discards score lower than 0.3. ## experiment and result the paper discusses the creation of a comparable corpus from scratch and extracts parallel sentences from the comparable data. as mentioned earlier, the nature of the sentences in the two documents is such that the english news provides a summary of the manipuri news. although our documents are topic-aligned, the sentences are not present in a one-to-one correspondence. this explains the difference in the number of sentences monolingually. further, we use hunalign to extract the parallel sentences from em corpus. we wanted to study the relevance of the parallel sentences extracted from the comparable data of the aforementioned nature. we designed a simple experiment on statistical machine translation. the models were trained on 5,000 sentences from pmindia dataset and incremented by 10,000 parallel sentences from our em corpus in each iteration. our validation and test data are from the pmindia (haddow and kirefu, 2020) dataset whose domain is the official documents from the prime minister office of india. figure 2 shows the result of the experiment. as we progress with the adding of more training data, we observe a decrease in the bleu score which is expected. it is to be noted that the decrease is not linear in nature. the data that we add other than the baseline are obtained from the news crawls which are not standardised translated data. although, the sentences are aligned, the parallel sentences are not exact translations of one another, instead comparable. the sudden increase of the bleu score could be the result of seeing similar sentences crawled from the news articles related to the prime minister office while training. ## conclusion this work provided an insight into corpus creation for manipuri-english language pair. firstly, we studied the creation of the comparable corpus, em corpus for the low-resourced language pair manipuri-english. secondly, we discussed the nature of the comparable corpus for manipuri-english language pair. we report the statistics on these data which is built by collecting from the web for over a year, from august 2020 to march 2021. the appendices provide information on mapping the glyph points to the unicode codepoints for manipuri. this is a necessary step due to the nature of the news articles that were crawled. the sangai express uses its custom web font file. this table can be referred if you are crawling independently to build your own corpus. in the future, we would like to inspect the possibility of increasing the size of data by using dataaugmentation techniques. also, we welcome everyone in improving and contributing to these resources.