Do Multilingual Transformers Encode P aninian Grammatical Relations? A Layer-wise Probing Study (2025)

Abstract: Large multilingual transformers such as XLM-RoBERTa achieve impressive performance on diverse NLP benchmarks, but understanding how they internally encode grammatical information remains challenging. This study investigates the encoding of syntactic and morphological information derived from the Paninian grammatical framework—specifically designed for morphologically rich Indian languages—across model layers. Using diagnostic probing, we analyze the hidden representations of frozen XLM-RoBERTa-base, mBERT, and IndicBERT models across seven Indian languages (Hindi, Kannada, Malayalam, Marathi, Telugu, Urdu, Bengali). Probes are trained to predict Paninian dependency relations (by edge probing) and essential morphosyntactic features (UPOS tags, Vibhakti markers). We find that syntactic structure (dependencies) is primarily encoded in the middle-to-upper-middle layers (layers 6–9), while lexical features peak slightly earlier. Although the general layer-wise trends are shared across models, significant variations in absolute probing performance reflect differences in model capacity, pre-training data, and language-specific characteristics. These findings shed light on how theory-specific grammatical information emerges implicitly within multilingual transformer representations trained largely on unstructured raw text.

## introduction multilingual pre-trained transformer models such as xlm-roberta @xcite have become foundational in natural language processing (nlp), achieving remarkable cross-lingual transfer capabilities. however, their internal representations remain only partially understood. identifying where and how linguistic structures are encoded across the layers of these transformers improves their interpretability, efficiency, and adaptability across diverse languages and linguistic tasks @xcite . a fundamental question is whether multilingual transformer models implicitly encode theoryspecific grammatical information despite not being explicitly trained with linguistic annotations grounded in such formalisms. much prior work has analyzed transformer representations using probing tasks based mainly on universal dependencies (ud) @xcite . while valuable, the ud framework aims for cross-lingual consistency and potentially abstracts away from linguistic phenomena better captured by language-specific or theory-specific grammars. indian languages (il), characterized by rich morphology and relatively free word order, are often analyzed using frameworks derived from the classical paninian grammatical tradition @xcite . this tradition introduces distinctive linguistic concepts such as kārakas (syntactico-semantic roles) and emphasizes morphological cues such as vibhakti markers as fundamental in signaling syntactic structures. a detailed overview of the specific paninian annotation scheme @xcite relevant to this work is provided in appendix a. in this work, we carry out the first paninian analysis of multilingual transformer representations across layers. we probe the hidden layers of xlm-roberta, indicbert-v2, and mbert across seven indian languages annotated under a shared paninian annotation scheme @xcite . to measure these linguistic representations, we employ edge probing @xcite for syntactic dependency structures and tokenclassification probes for morphosyntactic features (universal pos, vibhakti markers). specifically, the paper aims to answer the following research questions clearly: our key findings can be summarized as follows: ## experimental setup to investigate and compare the layer-wise encoding of paninian grammar in different multilingual transformers, we conducted probing experiments using three base models: xlm-roberta-base @xcite , multilingual bert @xcite and indicbert-v2 @xcite . for each base model, parameters were kept frozen foot_0 , and we extracted hidden states from all layers (including embeddings; layer 0 up to layer 12 for base models). our analysis covers seven indian languages: hindi (hi), kannada (ka), malayalam (ml), marathi (mr), telugu (te), urdu (ur), and bengali (be). the datasets utilize annotations following an extended version of the paninian dependency annotation scheme proposed by @xcite . probing tasks and models: we designed diagnostic classifiers @xcite for three distinct tasks reflecting key aspects of the formalism. following best practices for probing @xcite , we aimed for low-capacity probes: separate probe models were trained for each task, base model, language, and layer combination. data handling: input sequences were processed using the respective tokenizer for each base model (xlm-r, mbert, indicbert-v2). sequences exceeding a tokenized length of 128 tokens were filtered out. remaining sequences were padded/truncated to 128 tokens. target labels were aligned to the first sub-word token, with others ignored. detailed dataset statistics are provided in appendix b. training and evaluation: probes were trained independently for 3 epochs using the adamw optimizer @xcite with crossen-tropyloss. learning rates and batch sizes were tuned per model and task type based on preliminary experiments (edge probes: @xmath0, @xmath1; token probes: @xmath2.5e-3, @xmath3. the model state yielding the best validation performance (las for dependencies, accuracy for features) was selected. evaluations were performed on held-out validation sets. ## results we evaluated the performance of probes trained on each layer's representation across the three base models (xlm-roberta, mbert, indicbert-v2), seven languages, and three probing tasks (dependency relations, upos, vibhakti). key findings are presented below. dependency relations (las): figure 1 presents the best labeled attachment score (las) achieved by dependency probes trained on each layer (0-12) for the three base models across the seven indian languages. a consistent trend across all three models and most languages is the distribution of syntactic information across layers. las is low at the initial embedding layer (l0), increases through the lower layers, generally peaks in the middleto-upper-middle layers (typically layers 7-9), and then declines towards the final layer (l12). this confirms the widely observed phenomenon that structural syntactic information is most saliently represented in the intermediate representations of transformer models @xcite . model-wise, xlm-roberta (fig. 1a ) and mbert (fig. 1b ) achieve comparable peak performance on higher-resource languages like hindi (peak las 61.6% for xlm-r at l8; 59.1% for mbert at l7) and urdu (peak las 48.1% for xlm-r; 47.9% for mbert, both at l7). indicbert-v2 (fig. 1c ), despite its indic pretraining, does not consistently outperform the general multilingual models on this task for all languages. while reaching a high peak for hindi (62.1% at l6), its las scores for several other languages (ka, ml, be, mr, te) are often lower than those achieved by xlm-r or mbert, particularly in the upper layers. indicbert-v2 shows earlier peaks in several languages (e.g., l6 for hindi compared to l8 and l7 for xlm-r and mbert respectively). upos tagging (accuracy): figure 2 illustrates the layer-wise accuracy for predicting universal part-of-speech (upos) tags. generally, upos tagging accuracy is high, significantly exceeding las scores, confirming that basic lexical category information is more readily extractable. performance tends to peak relatively early compared to las, often plateauing across several lower-to-middle layers (e.g., layers 3-9 for many languages in xlm-r and mbert) before potentially declining slightly in the final layers. model-wise, xlm-roberta and mbert show strong results, with peak accuracies often reaching 85-90%+ for languages like hindi and kannada. indicbert-v2 generally achieves somewhat lower peak accuracies on this task compared to the other two models (e.g., peaking around 75% for hindi). ## analysis and discussion our multi-model, multi-task probing experiments reveal consistent layer-wise patterns for encoding paninian grammar, alongside notable performance variations. layer specialization for paninian grammar: across all three models, a functional specialization ## conclusion we presented a layer-wise probing analysis comparing the encoding of paninian grammatical information within xlm-roberta, mbert, and indicbert-v2 across seven indian languages, ex-amining dependency relations, upos tags, and vibhakti features. our findings reveal that paninian dependency structure generally peaks in the uppermiddle transformer layers, following lexical and morphological feature encoding in lower-to-middle layers, consistent with known patterns of linguistic representation. substantial cross-lingual and cross-model variations were observed. while indicbert-v2 showed strengths in vibhakti prediction for core indic languages, it did not uniformly surpass general multilingual models in representing paninian dependency structures. performance differences across languages correlate strongly with probing dataset sizes and likely reflect variations in pre-training data. our results confirm that probing effectively reveals how theory-specific grammatical formalisms are represented within standard multilingual models.