Data and R epresentation for T urkish N atural L anguage I nference (2020)

Abstract: Large annotated datasets in NLP are overwhelmingly in English. This is an obstacle to progress in other languages. Unfortunately, obtaining new annotated resources for each task in each language would be prohibitively expensive. At the same time, commercial machine translation systems are now robust. Can we leverage these systems to translate English-language datasets automatically? In this paper, we offer a positive response for natural language inference (NLI) in Turkish. We translated two large English NLI datasets into Turkish and had a team of experts validate their translation quality and fidelity to the original labels. Using these datasets, we address core issues of representation for Turkish NLI. We find that in-language embeddings are essential and that morphological parsing can be avoided where the training set is large. Finally, we show that models trained on our machine-translated datasets are successful on human-translated evaluation sets. We share all code, models, and data publicly.

## introduction many tasks in natural language processing have been transformed by the introduction of very large annotated datasets. prominent examples include paraphrase @xcite , parsing @xcite , question answering @xcite , machine translation (mt; @xcite , and natural language inference (nli; @xcite . unfortunately, outside of parsing and mt, these datasets tend to be in english. this is not only an obstacle to progress on other languages, but it also limits the field of nlp itself: english is generally not a representative example of the world's languages when it comes to morphology, syntax, or spelling conventions and other kinds of standardization @xcite , so it's risky to assume that models and results for english will generalize to other languages. a natural response to these gaps in our dataset coverage might be to launch new annotation efforts for multiple languages. however, this would likely be prohibitively expensive. for example, based on the costs of snli @xcite and multinli @xcite , we estimate that each large dataset for nli would cost upwards of us $50,000 if created completely from scratch. at the same time, commercial mt systems have improved dramatically in recent years @xcite @xcite @xcite . they now offer high-quality translations between hundreds of language pairs. this raises the question: can we use these mt systems to translate englishlanguage datasets and use the translated versions to drive more genuinely multilingual development in nlp? in this paper, we offer evidence that the answer is "yes". using amazon translate, we translated snli and multinli from english into turkish to create the first large turkish nli data sets, nli-tr, at a tiny fraction of the cost of creating them from scratch. turkish is an interesting challenge in this context since it is very different from english, most notably in its very free word order and complex morphology. a word in turkish bears morpho-syntactic properties in the sense that phrases formed of several words in languages like english can be expressed with a single word form. in our validation phase (section 3), a team of turkish-english bilingual speakers assessed the quality of a large sample of the translations in nli-tr. they found the quality to be very high, which suggests that translated datasets can provide a foundation for nli research on a resourceconstrained language, even if it has significantly different characteristics from english. we then use these datasets to study the roles of pre-trained language models and morphological parsing in successful nli systems for turkish (section 4). for these experiments, we fit classifiers on top of pre-trained bert parameters @xcite and compare the original bert-base release, the multilingual bert embeddings released by the bert team, and the turkish bert (berturk) embeddings of @xcite . we find berturk to be superior to the others for nli-tr. morphological parsing is a natural preprocessing step for turkish due to its complex morphology. thus, we assess the use of three morphological parsers as the second case study: @xcite , @xcite , and turkish morphology @xcite . we find that the parsers help where training data is sparse, but the need for a parser disappears as the training data increases. this is a striking finding: one might expect that turkish would require morphological parsing given its complex word-formation processes. it might be regarded as welcome news, though, since the parsers are expensive to run. in section 4.2, we report on some new optimizations of existing tools to make the relevant parsing jobs feasible, but we would still like to avoid these steps if possible, and it seems that we can for nli. finally, we investigate how models trained on the machine translated datasets perform on the human translations from xnli @xcite . we find that machine translated and human translated sentences yield similar results, suggesting that it is safe to apply models trained on machinetranslated datasets to human-written sentences. ## related work early in the development of textual entailment tasks, @xcite argued for multilingual versions of them. this led to subsequent explorations of a variety of techniques, including crowdsourcing translations @xcite , relying on parallel corpora to support reasoning across languages @xcite , and automatically translating datasets using mt systems @xcite @xcite . this research informed se-meval tasks in 2012 @xcite and 2013 @xcite followed by assin 1 (fonseca et al., 2016) and 2 @xcite shared tasks exploring the viability of multilingual nli. from the perspective of present-day nli models, these datasets are very small, but they could be used productively as challenge problems. more recently, @xcite reinvigorated work on multilingual nli with their xnli dataset. xnli provides expert-translated evaluation sets from english into 14 other languages, including turkish. though they are valuable resources to push nli research beyond english, test sets alone are insufficient for in-language training on target languages, which is likely to lower the performance of the resulting systems. although it was not the main focus of the xnli effort, @xcite distributed machine translations of multinli into other languages, including turkish, which we call multinli-tr xnli in this paper. the translations helped them form a strong baseline for their cross-lingual models, which proved superior in their assessments. however, the quality of the translations is crucial, as the authors note. our hope for nli-tr is that it supports effective in-language training. xnli's primary focus on test sets rather than training is justified by a wide body of recent results on cross-lingual transfer learning. multilingual embeddings (embeddings trained on multilingual corpora) have played an important role in these developments. the bert team @xcite released multilingual embeddings and demonstrated their value using xnli. at the same time, bert models have been released for a variety of individual languages (see @xcite and specialized domains @xcite . while we might expect the language-and domain-specific embeddings to be superior for the kind of data they were trained on, the multilingual versions might be more efficient in large-scale deployments in diverse environments. balancing these trade-offs is challenging. here, we offer some insight into these trade-offs for turkish. turkish is a morphologically-rich language in which new word forms are freely created using suffixation. several morphological parsers @xcite @xcite and morphological disambiguation systems @xcite have been developed for turkish. the state-of-the-art morphological analyzers can parse with success rates around 95%. we use three of these parsers in this work to evaluate the role of morphology in nli systems (section 4.2). ## conclusion we created and released the first large turkish nli dataset, nli-tr, by machine translating snli and multinli. though english and turkish have very different grammars and thus stress-test automatic approaches, our team of experts judged the translations to be of very high quality and to preserve the original nli labels consistently. these results suggest that mt can help address the paucity of datasets for turkish nli. we release code, models, and data publicly for further research. we also used nli-tr to investigate central issues in turkish nli. first, we used nli-tr to analyze the effects of in-language pretraining. second, we compared three morphological parsers for turkish with simpler tokenization schemes. we found that a turkish-only pretraining regime can enhance turkish models significantly, and that morphological parsing is arguably worth its costs only when the training dataset is small. in our final case study, we returned to the general issue of translation quality, but now from the perspective of developing nli systems. we showed that models trained on multinli-tr perform well on the expert-translated test set from xnli. on the basis of these findings, we argue that mt can be more widely adopted for advancing nlp studies on resource-constrained languages. though language-dependent tasks like dependency parsing are challenging to translate, mt can efficiently transfer large and expensive-to-create labeled datasets from english to other languages in many nlp tasks, including text classification, question answering, and text summarization. in addition, mt will presumably get cheaper, faster, and better over time, thereby further strengthening our core claims. e n t a i l m e n t c o n t r a d i c t i o n n e u t r a l b r o k e n majority-level label entailment contradiction neutral broken gold label 469 0 2 29 0 488 3 9 1 1 494 4 0 0 0 0 confusion matrix -annotations (all) 0 100 200 300 400 figure 3: confusion matrix for the majority-level label consistency results of all annotations in