Better Benchmarking LLM s for Zero-Shot Dependency Parsing (2025)

Abstract: While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs.

## introduction autoregressive large language models (llms) and instruction-based variants @xcite @xcite are known for their zero-shot and few-shot abilities @xcite . in practical terms, they can serve as versatile systems whose behavior is easily adapted through prompting. beyond what we experience as everyday users, documented examples in the context of natural language processing (nlp) include question answering @xcite , summarization @xcite , machine translation @xcite @xcite and information retrieval @xcite @xcite , among many other tasks. related, syntactic parsing has long explored few-shot learning approaches. prior to the development of current llms, various methods were studied to perform zero-shot or few-shot parsing, and many of these approaches achieved com-petitive results. these methods focused on factors such as the quality and quantity of annotations @xcite , crosslingual learning @xcite , multilingual pre-training @xcite , and treebank difficulty @xcite . however, the effectiveness of zeroshot parsing with llms remains a topic of debate. recent work has showed how state-of-the-art llms exhibit low performance in syntactic parsing @xcite , even when designing manual specific prompts @xcite . nonetheless, these results have been deemed sufficient to categorize llms as potential zero-shot parsers. while some studies @xcite suggest that multi-stage complex approaches can yield a competitive zero-shot performance, in this work we will focus on single prompt approaches -similar to what works for other nlp tasks -to evaluate to what extent llms can perform the task on their own without externally-provided planning. studies covering these approaches leave a substantial gap in evaluating llms on low-resource setups and often omit comparison with uninformed baselines, which are essential for determining whether llms achieve accuracy levels meaningfully above chance. contribution we address the lack of comparison against uninformed baselines, and include some that have not been proposed before but offer a higher standard than traditional blind baselines, such as left-or right-branching trees. these baselines provide more robust benchmarks and offer a fairer evaluation of llms' potential as zeroshot parsers. we prioritize depth over breadth by evaluating a wide range of llms to identify any substantial differences across them -a contribution that, to our knowledge, has not been thoroughly explored in previous work. 1 2 zero-shot dependency parsing next, we review the notation, benchmarks, uninformed baselines, and introduce the llms used. dependency parsing is the task of obtaining the syntactic structure of a sentence as a set of labeled directed relations (dependencies) between words. in zero-shot parsing (whether relying on llms or other models), the core idea is to perform the dependency parsing task without using taskspecific labeled data during either the pre-training or fine-tuning steps. this approach contrasts with the standard setup for training dependency parsers, where task-specific labeled data is integral to explicitly teaching models syntactic structures in a supervised learning framework. instead, zero-shot parsing leverages the model's general pre-trained knowledge to infer syntactic relationships in unseen data. before the emergence of large autoregressive generative models, pre-training largely avoided task-specific annotations, aligning closely with the zero-shot paradigm. however, given the extensive and diverse nature of the data these models are trained on, it is plausible that some exposure to annotated dependency parsing examples has occurred. this possibility will be examined further in subsequent sections. notation let @xmath0, ..., w n ) be a sentence, a dependency graph is defined as @xmath1, a), where w is the set of nodes and a the set of arcs. each arc in a is a tuple (h, d), where @xmath2, n] is the position of the head node, and d ̸ = @xmath3, n] the position of the dependent node. foot_1 g is a tree t iff (i) is a connected acyclic graph, (ii) each word w i has only one head, so @xmath4, d) : @xmath5...n}, and (iii) there is only one arc of the form (0, d), and w d is designated as the root of the sentence. this work only studies trees. ## experiments we conduct an in-depth evaluation of llms as zero-shot dependency parsers by generating outputs in conll format and comparing them to uninformed baselines. unlike lin et al. ( 2023 ), who evaluated only chatgpt-3.5 due to limited system availability, our work expands the analysis to a broader set of models across a select few languages, albeit on a smaller subset of treebanks. this approach, while time-intensive due to the extensive input and output token requirements, offers a more comprehensive understanding of model performance across different llms. ## conclusion we revisited the potential of autoregressive llms as zero-shot dependency parsers. taking a more conservative approach than previous studies, we compared several llms with simple baselines to establish minimal performance benchmarks. our results show that most llms performed on par with uninformed baselines, indicating comparable performance to toy approaches that operate without any access to the input sentence. figure 1 showed the original tree.