V yākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages (2021)

Abstract: While there has been significant progress towards developing NLU resources for Indic languages, syntactic evaluation has been relatively less explored. Unlike English, Indic languages have rich morphosyntax, grammatical genders, free linear word-order, and highly inflectional morphology. In this paper, we introduce Vyākarana: a benchmark of Colorless Green sentences in Indic languages for syntactic evaluation of multilingual language models. The benchmark comprises four syntax-related tasks: PoS Tagging, Syntax Tree-depth Prediction, Grammatical Case Marking, and Subject-Verb Agreement. We use the datasets from the evaluation tasks to probe five multilingual language models of varying architectures for syntax in Indic languages. Due to its prevalence, we also include a code-switching setting in our experiments. Our results show that the token-level and sentence-level representations from the Indic language models (IndicBERT and MuRIL) do not capture the syntax in Indic languages as efficiently as the other highly multilingual language models. Further, our layer-wise probing experiments reveal that while mBERT, DistilmBERT, and XLM-R localize the syntax in middle layers, the Indic language models do not show such syntactic localization.

## introduction the indian subcontinent is home to more than 450 languages spanning seven language families. multilingualism and code-switching are common phenomenon across these languages. traditionally, the region has been at the center of many linguistic studies due to its rich linguistic diversity @xcite ). the indo-aryan and the dravidian language families are the most prominent ones in the subcontinent, with more than a billion speakers combined. recent work in computational linguistics has focused on clubbing together the major languages from these two language families into a single group called as indic languages. 1 even with such a huge potential user base, the progress in building language technologies for the indic languages has been limited. most of the indic languages fall under the category of low-resourced and mid-resourced languages. some recent opensourced efforts have tried to address this by developing various toolkits @xcite , frameworks @xcite , language models, benchmarks, and datasets @xcite . to perform well on natural language understanding tasks, a language model should have a good understanding of the various general aspects of the structure of language, like its grammar and syntax. the structure of indic languages is quite rich in its morphosyntax, which is partially shared across the languages. thus, performing language modeling and downstream tasks on a multilingual dataset of indic languages helps capture typologically generalized stimuli across the languages and implicitly addresses the issue of data-scarcity. while there have been some efforts to analyze the role of multilingual training for indic languages, it has been limited to downstream nlu tasks and large multilingual models which are not exclusively trained for indic languages @xcite . in this work, we introduce a syntactic evaluation benchmark of colorless green 2 sentences in indic languages: vyākarana 3 , which provides a challenging envi-ronment to evaluate multilingual language models for their syntactic abilities specifically. we use the dataset to conduct layer-wise probing of five multilingual language models to inspect their understanding of the syntax in indic languages. we include the indicbert @xcite and muril @xcite foot_3 models in our experiments, which are trained on indic languages and english exclusively. we probe the models for four syntax-related tasks: pos tagging (pos), syntax tree-depth prediction (stdp), grammatical case marking (gcm), and subject-verb agreement (sva). in an attempt to capture the linguistic diversity in the indian subcontinent, we include one language each from the indo-aryan (hindi) and dravidian (tamil) language families in the dataset. in order to incorporate the prevalent phenomena of code-switching in these languages, we also include english-based script-mixed versions of these languages in our experiments. our layer-wise probing experiments show that the indic language models do not show syntactic localization and do not capture the syntax in indic languages as efficiently as the other "highly multilingual language models". 5 2 background language models are usually evaluated with information theory-based perplexity measures. while these perplexity measures might show good agreement with a language model's natural language understanding (nlu) capabilities, they do not capture language models' syntactic abilities efficiently @xcite . evaluating language models' syntactic abilities is quite important in furthering the research towards developing human-like robust language models @xcite . recent work has focused on the targeted syntactic evaluation of language models @xcite @xcite @xcite , which takes inspiration from various psycholinguistic generalizations found in humans and assesses the role of syntax in the models' ability to perform various nlu tasks. while there has been significant progress towards building nlu evaluation benchmarks @xcite , the work in developing syntactic evaluation benchmarks has been quite recent and limited @xcite @xcite . probing is an alternative paradigm that can be used for syntactic evaluation of language models. probing deals with quantifying the amount of linguistic information encoded in the pre-trained representations of language models @xcite . while this does not provide a very detailed analysis of models' syntactic behavior, it can be used to compare the amount of syntactic information captured by the models, as well as the inner dynamics of how and where this information is encoded in the models @xcite @xcite @xcite . probing methods are also relatively compute-efficient, as they do not involve training or fine-tuning the language models. with the recent development in multilingual language modeling, numerous evaluation studies have been performed to test the models' multilingual and cross-lingual abilities @xcite @xcite @xcite . similar to monolingual language modeling paradigms, there has been significant work towards building multilingual evaluation benchmarks @xcite @xcite . however, these benchmarks do not cover syntactic evaluation efficiently, with pos tagging being the sole syntax-related task. some recent studies have tried to address this by building multilingual and crosslingual syntactic evaluation suites with subject-verb agreement tasks @xcite . alongside the developments in multilingual nlp, there has been some recent progress towards advancing indic nlp @xcite @xcite . while there are numerous datasets available to benchmark and compare multilingual language models on nlu tasks in indic languages, there is no such resource available for syntactic evaluation in indic languages. 6 most recent works @xcite @xcite rely on pos tagging with universal dependencies treebanks as the sole task to compare the syntactic abilities of indic language models. neha eats mangoes and tom eats apples. ने हा आम खाती ह और टॉम से ब खाता ह । a. b. [ nnp ] [ nns ] [ vb ] ह और [ nnp ] [ nns ] [ vb ] ह । c. रया पतं ग पीती ह और जॉन प र ख चता ह । d. riya drinks kites and john draws stones. e. ( english translation of the original sentence ) ( original sentence ) ( template sentence without the content words ) ( colorless green variant of the original sentence ) ( english translation of the colorless green sentence ) feminine singular masculine plural masculine singular syntax in indic languages has specific characteristics which make it more challenging to model. unlike english, most of the indic languages follow a free linear word-order, with the default order being sov (subject-object-verb). hence, the linear distance (in terms of the number of intervening words) between the verb and its subject is usually larger in indic languages as compared to english (figure 1a ; figure 1b ). indic languages like hindi and tamil also have grammatical genders and grammatical number, where the words are morphologically inflected in agreement with the gender and count of their corresponding head nouns (figure 1b ). usually, the syntactic evaluation of language models is conducted with syntax-related probing tasks @xcite , or with targeted syntactic evaluation through controlled psycholinguistic experiments @xcite . performing such syntactic evaluations under a colorless green setting ensures that the models do not leverage any semantic or lexical cues and biases from the text 7 example sentences from the benchmark dataset can be found in appendix a to process its syntactic structure @xcite . most of the evaluation tasks from both of these categories can be designed with a syntactic dependencies treebank. hence, we construct a colorless green treebank for indic languages, which can be used to perform various syntactic evaluation tasks under a colorless green setting. ## conclusion this work presents a gender-balanced benchmark evaluation of colorless green sentences in indic languages for syntactic testing of multilingual language models. by doing so, we aim to address the existing research gap in syntactic testing in indic nlp. we introduce four new treebanks: cg-hdtb, cscg-hdtb, cg-ttb, and cscg-ttb for this purpose. we use the datasets from the treebanks to perform four syntax-related benchmarking evaluation tasks. in our experiments, we perform layer-wise probing of token-level and sentencelevel representations from five different multilingual language models. our experiments reveal that performing syntactic evaluation under a colorless green setting ensures that the semantic and lexical cues do not add evaluation artifacts in the form of higher-than-actual performance metrics. further, the experiments also reveal that the multilingual language models suffer significantly under a codeswitched setting, which is an important aspect in indic languages. overall, our experiments show that the currently available indic language models do not capture syntax as efficiently as the other highly multilingual language models. indic models seem to outperform other multilingual language models on various nlu tasks in indic languages even with a certain lack of syntactic knowledge. this might suggest that the indic language models do not rely significantly on syntax while making inferences on such nlu tasks. we also observe that training on large artificial corpora of transliterated texts do not help indic language models in capturing syntax under a code-switched setting. further, we find that unlike the other highly multilingual language models, the currently available indic language models do not show any syntactic localization in the middle layers. this is a unique