Linguistic Features in G erman BERT : The Role of Morphology, Syntax, and Semantics in Multi-Class Text Classification (2025)

Abstract: Most studies on the linguistic information encoded by BERT primarily focus on English. Our study examines a monolingual German BERT model using a semantic classification task on newspaper articles, analysing the linguistic features influencing classification decisions through SHAP values. We use the TüBa-D/Z corpus, a resource with gold-standard annotations for a set of linguistic features, including POS, inflectional morphology, phrasal, clausal, and dependency structures. Semantic features of nouns are evaluated via the GermaNet ontology using shared hypernyms. Our results indicate that the features identified in English also affect classification in German but suggests important language- and task-specific features as well.

## introduction even today, with large language models (llms) like @xcite , llama @xcite , or mistral @xcite representing the de facto state-of-the-art systems for most nlp tasks in english, the exploration of bert-like models still provides extremely useful insights for low-resource and non-english scenarios @xcite @xcite , often offering more efficient and lightweight solutions. despite the extensive research evaluating the linguistic knowledge encoded in various english versions of bert @xcite using interpretative methods like attention analysis @xcite @xcite , monolingual models pre-trained on languages other than english have received considerably less attention. given that languages can differ quite significantly in their morphological, syntactic, and semantic complexity, it is crucial to identify which behaviours observed for english translate to other languages and which, instead, are language-specific. for example, @xcite found that different types of linguistic information are distributed across different layers of english bert; surface-level information like phrasal structure is processed by layers closer to the input, syntactic information by the middle layers, and semantic information by the layers closer to the output. the ability of bert-like models to process syntactic information has been evaluated by assessing their performance on subject-verb agreement in english @xcite . more recently, @xcite assessed the quality of the semantic representations for general function words (e.g. negations, coordinating conjunctions, and quantification terms) in these models. their findings suggest that bert-like models struggle to accurately complete sentences based on these function words alone, often relying on other indicators, like named entities (nes), for their predictions. our work investigates which morphological, syntactic, and semantic features are the strongest predictors in an eight-class text classification task for a german bert model. building on evidence from english, we analyse similarities and differences, particularly exploring how the richer inflectional morphology of german (eisenberg, 2020) affects model performance. former studies on german have analysed morphological or syntactic features separately @xcite . claeser (2022) conducts a study on the same corpus we use in this work, but considers only the influence of morphology with regard to cnns. our study covers a larger selection of morphological, syntactic, and semantic features and focuses on bert. 1 2 materials and methodology ## results and discussion all steps in the following analysis focus on the top 10% tokens with positive shap values in correctly classified texts; in this way we inspect only words that positively contribute to the correct classification decision. in addition to the usual quantitative analysis of shap values, we run a statistical analysis to identify which features significantly affect model performance. we make this decision to ensure that all reported effects are significant above chance, which is especially important for less frequent features. as null hypothesis, we assume that the distribution of each shap feature within a category matches its original distribution in the corpus for the same category. positive contributions are reported when values significantly exceed the null hypothesis, and negative contributions when they are significantly lower (refer to appendix a for more details). to reduce data sparsity and increase generalizability, we group all linguistic features into coarse-grained categories (e.g., "verbs" would include all types of verbs; refer to appendix b for the detailed mapping). ## conclusion this work investigates the role of linguistic information in a monolingual german bert model for a multi-class classification task. it replicates prior findings on the dilution of phrasal information in a full 12-layer model @xcite ) and bert's preference for nes @xcite . the results suggest that german bert's syntactic representation prioritizes dependency relations over clausal or phrasal ones by focusing on word clusters in dependency relations, showing opportunities for further research. additionally, german noun inflection has a minor influence, with a preference for nominative over accusative, possibly due to the syntactic function outweighing its morphological form. the semantic analysis shows that classification accuracy depends not only on class size but also on smaller categories forming a coherent semantic space, and consequently, increasing their distinctiveness. overall, this study indicates some crosslinguistic consistency in bert's linguistic representations while emphasizing the need for further analyses of language-specific phenomena, especially in low-resource contexts.