A Parallel Corpus and Dictionary for A mis- M andarin Translation (2022)

Abstract: Amis is an endangered language indigenous to Taiwan with limited data available for computational processing. We thus present an Amis-Mandarin dataset containing a parallel corpus of 5,751 Amis and Mandarin sentences and a dictionary of 7,800 Amis words and phrases with their definitions in Mandarin. Using our dataset, we also established a baseline for machine translation between Amis and Mandarin in both directions. Our dataset can be found at https://github.com/francisdzheng/amis-mandarin .

## introduction amis is a minority language spoken on the east coast of taiwan and has been described as a vulnerable or endangered language @xcite @xcite . though there have been some efforts to preserve the language through education and linguistic research @xcite @xcite , amis and its preservation have not yet benefited from (to the best of our knowledge) data-based methods used in machine learning and natural language processing. low-resource machine translation has recently attracted more attention in the field of natural language processing for languages such as amis that have a relatively low amount of data due to a small population of speakers. because neural machine translation (nmt) systems typically do not perform well for low-resource languages, which lack parallel data @xcite , approaches such as collaborating with language communities to increase parallel data, transfer learning from other machine translation systems, and using multilingual models, among others are being explored @xcite . however, despite all these new approaches to low-resource machine translation, it is clear that parallel data is still essential for training state-of-the-art machine translation systems @xcite , as high-resourced language pairs still require large amounts of data to achieve state-of-the-art translation quality @xcite . due to the lack of amis resources available for use in machine translation, we developed an amis-mandarin parallel corpus and dictionary. our contributions can be summarized as follows: ## amis amis (iso 639-3 language code ami) is an east formosan language @xcite spoken on the east coast of taiwan between hualien and taitung @xcite by the amis, one of taiwan's several indigenous ethnic groups. formosan languages are spoken by the indigenous peoples of taiwan @xcite and are part of the austronesian language family. they are believed to be the most diverse of the austronesian languages @xcite , and because high diversity in a group of genetically-languages found in a geographical area implies earlier settlement in that area @xcite , taiwan is considered to be the homeland of austronesian languages @xcite . figure 1 shows the distribution of these languages in taiwan with the region where amis is spoken being shaded in gray. though the vast majority of taiwan's population is han chinese, taiwan is home to several groups of indigenous peoples who are austronesian @xcite . according to official government statistics 1 , taiwan's indigenous population is 582,008, which is approximately 2.4% of taiwan's total population. the amis people have a population of 217,216, making up approximately 37.3% of taiwan's indigenous population. mandarin chinese is the language of education (scott and tiun) and is spoken along with other chinese languages (e.g. hokkien) by the majority of the population, whether indigenous or not. despite the amis population being over 200,000, the amis language has just roughly 30,000 speakers @xcite . these roughly 30,000 speakers, however, do not all speak the same dialect of amis. according to tsuchida @xcite @xcite , amis has five major dialects: (i) sakizaya (撒 奇萊雅群), (ii) northern/nanshi amis (北部/南 勢阿美群), (iii) tavalong-vata'an (太巴塱-馬太 鞍群), (iv) central/haian amis (中部/海岸阿美 群), and (v) southern/peinan and hengchun amis (南部/卑南恆春阿美群). the dataset described in section 3 in this paper uses data from central amis. the existence of several dialects of amis means that any one dialect has a relatively low number of speakers. amis is classified as a vulnerable language by unesco @xcite , meaning that most, but not all, children and families of the amis, an indigenous austronesian ethnic group native to taiwan, speak amis as a first language but that their use of amis may be limited to specific social settings (such as the home, where it is used amongst family) @xcite . however, amis has also been described as an endangered language by several linguists @xcite @xcite . @xcite , who researched the amis language in taiwan since 1995, performing extensive fieldwork and data gathering with native amis speakers, notes that only those over 50 are proficient in amis. whether amis is "vulnerable" or "endangered," it is clear that the language is at risk and would benefit from more attention from linguists and natural language processing technologies to help preserve the language digitally and enable its use in modern technologies. ## amis-mandarin dataset we compiled amis-mandarin parallel data from an amis-mandarin online dictionary (原住民族 語 言 線 上 辭 典) foot_1 published by the indigenous languages research and development foundation (原 住 民 族 語 言 研 究 發 展 基 金 會). this dictionary consists of words, phrases, and example sentences of these words and phrases in central amis with their mandarin translations. it was compiled in 2012 by national taiwan normal university. though these data have been made searchable in the format of an online dictionary, they are not designed for computational use. the website allows one to download the dictionary in parts or in whole as a pdf or odt file. however, due to some inconsistencies in how words, translations, and example sentences are laid out in these files, neither is easy to use for computational tasks. thus, we downloaded pdfs of the dictionary made available by this online dictionary, converted them to html using pdfminer foot_2 , and extracted data using beautiful soup foot_3 indepen864 55 , 672 7 , 289 6 , 903 dently from the authors of this dictionary. each amis word/phrase and its mandarin dictionary entry were extracted, and when available, example sentences in amis along with their mandarin translations were also taken. dictionary entries and their mandarin definitions were put into one pickle file, while example sentences and their mandarin translations were put into another pickle file. these files can be opened using pandas 5 . the dictionary is also available as a tab-delimited text file, and the parallel sentence data are also available as text files split into train, dev, and test sets. the parallel data were shuffled before being split into the train, dev, and test sets, which were taken from 80%, 10%, and 10% respectively from the shuffled data. the dataset we compiled can be found at https://github.com/ francisdzheng/amis-mandarin . our dictionary dataset contains 7,800 unique entries in amis along with their mandarin equivalents. amis dictionary entries that had more than one definition in mandarin were added to our dictionary dataset as separate entries for each additional definition. thus, there are a total of 7,926 pairs of amis and mandarin words/phrases in our dictionary dataset. our parallel corpus dataset contains 5,751 amis sentences and their mandarin translations. this parallel corpus dataset is summarized in ## amis-mandarin machine translation using our amis-mandarin dataset, we trained models for machine translation between amis and mandarin in both directions. ## conclusion we presented an amis-mandarin parallel corpus and dictionary, which is the first, to the best of our knowledge, amis-mandarin dataset documented in english for the natural language processing community. though the online dictionary from which we obtained the data is available to anyone, the dictionary interface is only available in mandarin, and the data is not in a form that nlp researchers can easily use. other amis-mandarin data we found on the web were also not in an easily usable format and not friendly for englishspeaking researchers. the dataset we compiled consists of 5,751 parallel sentences and 7,800 amis words and phrases paired with their definitions in mandarin. using this dataset, we experimented with amis-mandarin machine translation and established baseline bleu and chrf scores. using both the parallel corpus and dictionary during training produced models that performed the best on our test data from our parallel corpus. aside from sentence translation, we also envision that the dataset we compiled can be used for exploring how dictionary entries can be predicted using parallel data (instead of using dictionary entries to aid in the translation of sentences). defining words is also an important part of language documentation, and it would be interesting to see how machines can draw meaning for individual words or short phrases given parallel sentence data. in the future, we would like to take a closer look at how tokenization can be optimized for the two languages and try using other existing tokenizers that have been trained more specifically for mandarin. we also want to try to incorporate more external knowledge and perhaps acknowledge that parallel data may never be enough. our dataset is small, and we hope to explore how knowledge from grammars and other literature written on amis can be incorporated into our model or into the creation of synthetic parallel data. as amis is an austronesian language like indonesian, which is widely spoken and has much more literature available, it is possible that knowledge from indonesian can be helpful in nlp tasks involving amis. we hope that our dataset can spark more interest from the machine learning community in not only amis, but other formosan languages as well.