Type and Complexity Signals in Multilingual Question Representations (2025)

Abstract: This work investigates how a multilingual transformer model represents morphosyntactic properties of questions. We introduce the Question Type and Complexity (QTC) dataset with sentences across seven languages, annotated with type information and complexity metrics including dependency length, tree depth, and lexical density. Our evaluation extends probing methods to regression labels with selectivity controls to quantify gains in generalizability. We compare layer-wise probes on frozen Glot500-m (Imani et al., 2023) representations against subword TF-IDF baselines, and a fine-tuned model. Results show that statistical features classify questions well in explicitly marked languages and structural complexity prediction, while neural probes lead on individual metrics. We use these results to evaluate when contextual representations outperform statistical baselines and whether parameter updates reduce availability of pre-trained linguistic information.

## introduction multilingual contextual embeddings show promise for accessing fine-grained morphosyntactic properties across hundreds of languages. probing how transformer models encode certain linguistic properties has practical implications for language typology research, where systematic comparison of structural features often relies on automated analysis. additionally, evaluations targeting specific linguistic phenomena can test common architectural assumptions about transformer models. examples include the often discussed layer-wise specialization from syntactic to semantic processing @xcite and the ability of shared embedding spaces to effectively capture cross-linguistic patterns. researchers rely on these assumptions in order to describe the internals of the models when testing on benchmarks @xcite ; şahin ara eng fin ind jpn kor rus tydi qa ud 2.15 what do you think of air france ? pron aux pron verb adp propn propn punct 1 2 3 4 5 6 7 8 root obj aux nsubj obl case compound punct labels type content token count 8 max. tree depth 2 avg. verbal edges 3 lexical density 0.428 avg. edge length 2 avg. sub. chain length 0 complexity metrics 15.428 sentence representations subword tf-idf glot500 encoder fine-tuned glot500 et al., 2020), but also when evaluating their general linguistic capabilities outside of specific tasks @xcite . however, comparisons with appropriate baselines are often left out. without those, we cannot determine whether observed linguistic capabilities reflect genuine structural processing or are the result of patterns that simpler statistical methods capture equally well. this presents a challenge when investigating universal sentence-level phenomena where the relationship between surface form and underlying structure varies extensively @xcite . we focus specifically on interrogative sentences, which illustrate this variation particularly well. for example, arabic uses explicit particles like " ‫ه‬ ‫ل‬ " for polar (yes/no) questions and overt subordinating conjunctions for complex clauses. alternatively, japanese relies on contextual cues for question interpretation and implicit hierarchical embedding through casemarking for structural complexity. the differences in how languages encode both categorical distinctions and continuous complexity metrics create a natural setup for evaluating whether contextual rep-resentations capture structural patterns better than surface-level statistical correlations obtained by, for example, tf-idf features. we explore this question through controlled comparisons of neural representations with statistical baselines across seven typologically diverse languages. our framework extends probing methods to continuous linguistic properties, including token count, lexical density, dependency length, tree depth, verbal arity, and subordination patterns. figure 1 illustrates our method: we start with existing multilingual datasets, process and annotate these for categorical (interrogative types) and continuous labels, and we finally evaluate three representation types (subword tf-idf features, contextual embeddings, and a fine-tuned model) using our annotated data in arabic, english, finnish, indonesian, japanese, korean, and russian. we present three key findings: ## related work probing methods assess what linguistic knowledge is encoded in neural representations by training classifiers to predict specific properties in word embeddings @xcite . early work demonstrated that contextualized and static representations encode syntactic information like part-of-speech categories, dependency rela-tions, and word order variation @xcite . most probing studies focus on token-level properties, with fewer approaches looking at variation in sentence-level regularities. şahin et al. (2020); @xcite introduce comprehensive evaluation frameworks for sentence level probing tasks. these reveal how models encode structural linguistic properties such as morphological case marking, agreement patterns, and syntactic hierarchies, as well as functional properties including semantic roles, discourse relations, and pragmatic features. question type classification represents a natural extension of this work, as it requires models to integrate both formal markers (interrogative particles, auxiliary inversion) and functional understanding (information-seeking intent, presupposition structure). two assumptions motivate current probing approaches. first, the layer-wise specialization hypothesis suggests lower layers encode syntax while higher layers capture semantics @xcite . this informs decisions about which layers to probe for different linguistic tasks. second, multilingual models develop shared embedding spaces that capture cross-linguistic patterns @xcite , enabling efficient transfer across languages. probes mainly target categorical properties through classification tasks @xcite . however, @xcite argue that complex linguistic phenomena require more sophisticated probing architectures that can approximate a wider range of information content. regression-based probing is a simple approach that investigates linguistic properties like syntactic complexity, processing difficulty, and structural density. complexity measures derived through dependency parsing allow us to generate target labels that reveal how models encode syntactic structure along continuous and discrete dimensions. we investigate these to assess how accessible structural features are from learned embeddings. determining whether probes capture genuine structural encoding requires appropriate baselines. @xcite introduced selectivity controls comparing performance on real versus shuffled labels to distinguish linguistic encoding from spurious correlations. most studies, however, evaluate neural representations without statistical baselines, making it difficult to assess whether contextual embeddings offer genuine advantages over similarly, while probing typically uses frozen representations extracted from a specific layer of the encoder, the relationship between pre-trained knowledge and task-specific optimization remains underexplored. understanding when end-to-end optimization preserves or degrades pre-trained linguistic knowledge requires direct comparison of frozen probes and fine-tuned models, particularly for structural properties that may be disrupted by task-specific adaptation. ## dataset we introduce the question type and complexity (qtc) dataset containing ∼ 9, 000 annotated questions across seven languages: arabic, english, finnish, indonesian, japanese, korean, and russian. qtc combines tydiqa-goldp training data @xcite with universal dependency treebank test data @xcite to balance natural language variation with standardized syntactic annotation, with approximately 70% of sentences drawn from tydiqa and 30% from ud treebanks. the choice of languages was informed by different question formation strategies. languages using explicit interrogative marking include arabic with " ‫ه‬ ‫ل‬ ", finnish with suffix "-ko/-kö", and russian with particle "ли". languages using implicit strategies like context or prosody include japanese, indonesian, and korean. lastly, auxiliary inversion in english can be seen as a mixed strategy. categorical and continuous labels were created using parallel annotation pipelines. for question type classification, tydiqa data already contained human annotations from three independent annotators. we adopted annotations where all three an-notators agreed and manually resolved disagreements. ud treebank sentences were annotated for question type using language-specific rule-based systems targeting morphosyntactic patterns: interrogative particles, wh-phrase positioning, and auxiliary structures. we label polar questions as '1' and content questions as '0'. for complexity metrics, we used udpipe 2.0 @xcite to parse all sentences, then applied the profiling-ud framework @xcite to extract six raw complexity features capturing processing difficulty (see appendix a for details). we validated complexity metrics through statistical outlier detection and (partial) manual verification of parse quality. foot_0 4 probing tasks ## experiments the experiments were carried out on glot500-m @xcite , a multilingual encoder-only transformer. glot500-m was created by extending the xlm-r-base architecture @xcite using continued pre-training on a custom multilingual corpus and expanding the vocabulary from 250k to 401k tokens to cover 511 languages, including all seven languages in our dataset. ## results our statistical baselines employ logistic regression for classification and ridge regression for complexity prediction, with xgboost capturing nonlinear feature interactions. results across the two tasks reveal trade-offs in the ability of our models to capture different kinds of linguistic information. for question type classification, neural probes consistently perform the best, with the majority of highest accuracy and selectivity scores. regression results show more variety, with different representation types leading on different complexity metrics. submetric tf-idf ridge s tf-idf xgboost s glot500 best probe s layer glot500 fine-tuned ## conclusion we investigated how multilingual transformers encode question patterns by comparing contextual embeddings against statistical baselines across seven typologically diverse languages. glot500 probes show advantages in question type classification, particularly for languages requiring contextual integration (japanese, korean, english, finnish), while arabic's unambiguous particles favor statistical methods. for complexity regression, statistical baselines show better selectivity on most individual metrics, though neural methods excel at token count and verbal arity. different complexity metrics exhibit distinct layer-wise encoding patterns. fine-tuning compensates for unstable neural encoding (high oscillations) but struggles on metrics with otherwise stable layer-wise representations, suggesting task-specific optimization can disrupt pre-trained knowledge. our qtc dataset and regression-based probing setup using selectivity controls provide tools for investigating continuous linguistic properties. we find that understanding when and why neural models capture linguistic structure requires careful comparison with principled baselines. future work should examine applications to other architectures, investigate why certain complexity metrics resist neural encoding, and develop training procedures that preserve linguistic information while improving task performance.