LLM Dependency Parsing with In-Context Rules (2025)

Abstract: We study whether incorporating rules (in various formats) can aid large language models to perform dependency parsing. We consider a paradigm in which LLMs first produce symbolic rules given fully labeled examples, and the rules are then provided in a subsequent call that performs the actual parsing. In addition, we experiment with providing human-created annotation guidelines in-context to the LLMs. We test on eight low-resource languages from Universal Dependencies, finding that while both methods for rule incorporation improve zero-shot performance, the benefit disappears with a few labeled in-context examples.

## introduction dependency parsing is a classic task in natural language processing, requiring systems to parse complex linguistic structures. standard approaches to dependency parsing train neural models on large amounts of labeled data (human-created parses), which either cast parsing as a word-by-word classification task @xcite , a sequence-tosequence generative task @xcite , or a graph-based structure prediction task @xcite . one limitation of neural parsing methods is the requirement for large amounts of labeled data, which is often unavailable for low-resource languages. while universal dependencies (de @xcite currently includes over 150 languages, many of these languages have less than one thousand labeled tokens worth of data. crosslingual transfer has been proposed as a method to overcome these limitations @xcite , but faces challenges due to differences in linguistic structure across languages @xcite . recent work has investigated the use of large language models (llms) as an approach for parsing tasks @xcite @xcite @xcite . 1 llms are pretrained on huge multilingual corpora, and can potentially leverage cross-lingual information for effective parsing, even on rarer languages. generally, this research has found that llms are effective zero-shot parsers on common languages such as english and chinese and can also be effective on rare languages through in-context learning. however, in-context learning quickly grows inefficient and expensive with many examples. we explore an alternate paradigm for dependency parsing on low-resource languages with llms. in our system, the llm first acts as a descriptive linguist, observing labeled examples and producing linguistic rules. then, the rules are provided in another llm call where parsing is performed. we consider several techniques for providing context to the llm during rule generation. we hypothesized that explicitly producing symbolic rules could help improve the robustness of llm-based parsing. we find that incorporating the llm-generated rules offers clear improvements over the zero-shot setting, but worse performance than few-shot prediction. furthermore, our best llm-based setting underperforms state-of-the-art approaches across languages. we explore a number of failure cases and suggest future methods that could be used to address them. our code and full results are available on github. foot_1 ## related work recent work has explored the potential of large language models for syntactic parsing. some work has finetuned language models on dependency parsing as a sequence-to-sequence task @xcite or proposed statistical methods to automatically extract dependencies from language models @xcite . the most similar work explores prompting-based methods for llm parsing, that do not require training. @xcite evaluates llms on zero-shot parsing for english and chinese. @xcite proposes similar strategies for constituency parsing in english. @xcite benchmarks dependency parsing performance across four languages and many llms. however, using llms to parse rare languages remains unexplored. ## data we use data from universal dependencies (ud) @xcite for eight languages with differing geographic regions, linguistic features, and resource availability. we use the pre-defined train/eval/test splits from ud when available, and otherwise produce our own splits which are reused across experiments. we also remove examples with non-projective dependencies, which can pose issues for transition-based parsers. we summarize our languages and splits in ## experimental results we report our results on the development set, using gpt-4o, in figure 1 . we average uas and las scores over languages; full results are available in our github repo. generally, we observe a clear trend where providing symbolic information helps in the zero-shot setting, but the benefit decreases with increased in-context examples. at the five-shot setting, any benefits from symbolic knowledge are effectively nullified. next, we analyze the effects of the various settings. we also perform additional variational experiments on the use of labels and chain-of-thought prompting in appendix d. ## conclusion we studied methods for performing dependency parsing on low-resource languages with large language models (llms) that incorporate (symbolic) rules. we compared using llm-written rules, extracting contexts that words appear in, and providing human-readable annotation guidelines. overall, we found that these methods provide benefits in the zero-shot setting, but with sufficient in-context examples, their benefit was minimal. we evaluated several llms against state-of-the-art baselines, finding that the llms were unable to beat the best prior models.