Memory-enhanced Large Language Model for Cross-lingual Dependency Parsing via Deep Hierarchical Syntax Understanding (2025)

Abstract: Large language models (LLMs) demonstrate remarkable text generation and syntax parsing capabilities in high-resource languages. However, their performance notably declines in low-resource languages due to memory forgetting stemming from semantic interference across languages. To address this issue, we propose a novel deep hierarchical syntax understanding approach to improve the cross-lingual semantic memory capability of LLMs. First, we design a multi-task joint fine-tuning strategy to implicitly align linguistic knowledge between source and target languages in LLMs, which is leveraged to initially parse the target text. Second, we automatically construct the multilingual dependency label banks based on the statistical structure information from the Universal Dependencies (UD) data. Third, we obtain each label’s memory strength via in-depth analysis of the initial parsing tree and its dependency label bank. Finally, memory strength is further exploited to guide LLMs to learn the linguistic commonalities from multilingual dependency label banks, thus activating the memory ability of weak labels. Experimental results on four benchmark datasets show that our method can dramatically improve the parsing accuracy of all baseline models, leading to new state-of-the-art results. Further analysis reveals that our approach can effectively enhance the weak syntactic label memory cognition of LLMs by combining the advantages of both implicit multi-task fine-tuning and explicit label bank guiding. Our code and dependency label banks are released at https://github.com/Flamelunar/memory_dep.

## introduction dependency parsing employs hierarchical tree structures to exhibit syntactic and grammatical relationships between words. as shown in figure 1, the tree includes an arc from the headword "小 说 (fiction)" to the dependent word "新的 (new)" with the label "amod", indicating adjectival modification. these hierarchical structures are widely applied in multiple natural language processing (nlp) tasks, including machine translation @xcite , question answering @xcite , grammatical error correction @xcite , and text classification @xcite . recently, researchers focus on improving the syntax understanding of large language models (llms) using dependency trees @xcite @xcite . advances in language models have markedly improved supervised dependency parsing for highresource languages @xcite @xcite . however, language model-enhanced parsers are highly dependent on the scale and quality of training data, and their performances drop sharply when they are directly transferred to low-resource languages due to semantic interference @xcite @xcite . therefore, cross-lingual dependency parsing has emerged as a promising direction, aiming to transfer effective knowledge from high-resource languages to low-resource ones @xcite @xcite . existing approaches fall broadly into two categories, i.e., traditional and llm-based methods. traditional methods mainly rely on syntactic feature projection or transformation @xcite @xcite . @xcite incorporate the source and target linguistic typological knowledge into a multi-task learning framework to enhance cross-lingual knowledge transfer. in contrast, llms ( chatgpt foot_0 , llama foot_1 , qwen foot_2 , and deepseek foot_3 ) exhibit remarkable generalization across a wide range of nlp tasks, benefiting from massive pre-trained corpora and highly optimized architectures. moreover, their capabilities can be further strengthened by useful prompt learning @xcite , task-specific parameterefficient fine-tuning @xcite , and retrieval augmented generation @xcite . however, llms struggle in low-resource languages' dependency parsing due to memory forgetting @xcite . the main reason is that normal llms are prone to memorizing the semantic preferences of highresource languages while their capability in lowresource languages is obstructed @xcite . as illustrated in figure 1 , we can see that llms show strong parsing ability in the high-resource language (chi-nese) with numerous training data, achieving a 85.72% accuracy. in contrast, the parsing accuracy of vietnamese is only 57.14%. concretely, although vietnamese and chinese share a subject-verb-object structure, they diverge in modifier placement such as vietnamese favors postmodifiers, whereas chinese employs pre-modifiers. even though there is linguistic structural variation in real scenarios, the relative structure between the dependency label and pos tags is constant. for example, both chinese and vietnamese have a dependent word with pos tag "adj" modifies the head word with pos tag "noun", owning the same dependency label "amod". hence, dependency relations (head-dependent patterns) often remain consistent across languages, these crosslinguistic syntactic similarities can be leveraged to improve parsing performance of low-resource languages @xcite . to alleviate this drawback, we propose a deep hierarchical syntax-aware approach to enhance the semantic memory capability of llms. first, we employ a multi-task joint fine-tuning strategy to implicitly align llms' syntactic knowledge across different languages. meanwhile, fine-tuned llms are utilized to yield the initial parsing trees of the target language data. then, we construct multilingual dependency label banks by extracting statistical patterns from the universal dependency treebanks. next, each label's memory strength is estimated through structural analysis of the initial parsing trees and its distribution in the label bank. finally, memory strength is used to guide llms in capturing cross-lingual syntactic commonalities, thereby reinforcing the memory capability of weak dependency labels. experiments on four benchmark datasets demonstrate substantial performance gains in low-resource scenarios, achieving prior state-of-the-art results. further analysis indicates that our approach can effectively strengthen the weak syntactic label memory strength of llms by integrating the advantages of both implicit multitask fine-tuning and explicit dependency label bank guiding. ## our approach in this work, we propose a deep hierarchical syntax understanding approach to strengthen crosslingual semantic memory in llms. first, we jointly employ cross-lingual part-of-speech (pos) tagging and dependency parsing tasks to fine-tune parameters of llms, thus implicitly aligning linguistic knowledge between source and target languages. meanwhile, we utilize fine-tuned llms to generate initial parsing trees for target language test sentences. second, we build multilingual dependency label banks by extracting statistical syntactic patterns from universal dependency corpora, which explicitly exhibit the relationship between common dependency labels and fine-grained pos tags. then, we analyse each label's correct rate in initial parsing trees and the distribution frequency in finetuning training data to identify its memory strength. finally, memory strength is further exploited to guide llms to learn the linguistic commonalities from multilingual dependency label banks, yielding more accurate final parsing trees. figure 2 shows the overall architecture with three components, i.e., multi-task joint fine-tuning, dependency label bank construction, hierarchical memory enhancement. ## conclusion we propose a novel deep hierarchical syntax understanding method to enhance the weak dependency label memory capability in large language models. concretely, we exploit implicit multi-task fine-tuning and explicit dependency label bank guiding to boost llms to absorb crosslingual syntactic commonalities. experiments on four benchmark datasets show substantial accuracy gains across all baseline models, achieving stateof-the-art performance. analysis reveals that both multi-task joint fine-tuning and extra dependency label bank can extract useful syntactic knowledge from the source language to enhance the target language parsing accuracy. moreover, in-depth comparison demonstrates that our method can alleviate semantic interference across languages and improve the memory strength of most dependency labels, thus further improving the parsing performance.