Step-by-step Instructions and a Simple Tabular Output Format Improve the Dependency Parsing Accuracy of LLM s (2025)

Abstract: Recent advances in large language models (LLMs) have enabled impressive performance in various tasks. However, standard prompting often struggles to produce structurally valid and accurate outputs, especially in dependency parsing. We propose a novel step-by-step instruction strategy, where universal part-of-speech tagging precedes the prediction of syntactic heads and dependency labels, and a simplified CoNLL-U like output format, our method achieves state-of-the-art accuracy on Universal Dependencies datasets across 17 languages without hallucination or contamination. We further show that multilingual fine-tuning simultaneously improves cross-language generalization performance. Our results highlight the effectiveness of explicit reasoning steps in LLM-based parsing and offer a scalable, format-consistent alternative to bracket-based approaches.

## introduction recent advances in large language models (llms) have dramatically reshaped the landscape of natural language processing; however, their potential for syntactic analysis -particularly dependency parsing -remains underexplored. furthermore, it is desirable to systematically investigate prompting and fine-tuning techniques that enhance the performance of llm-based dependency parsing. in this work, we examine how fine-tuned llms can be effectively guided to perform accurate dependency parsing using simple, structured instruction prompts. specifically, we design a single-turn supervised fine-tuning setup where the input sentence is accompanied by a tabular output format based on a minimal subset of the conll-u 1 , which is the standard format of universal dependencies (ud) treebanks @xcite as in figure 1 . this table-based representation not only improves 1 https://universaldependencies.org/format.html format validity and readability, but also facilitates learning non-projective structures. the results of our preliminary experiments using ud_english-ewt foot_0 are summarized in despite using a very simple prompt, we observed fairly high parsing accuracy, prompting us to investigate the possibility of data contamination (refer appendix b for details). based on our analysis, we found no evidence of contamination in token recall upos uas las udpipe 2.0 100.0 97.5 93.4 91.5 gpt-4o-mini-2024-07-18: chain-of-thought steps 1 (upos+head+deprel) 98.5 98.4 93.3 91.4 2 (upos; head+deprel) 99.1 98.6 94.8 93.1 3 (upos; head; deprel) 99.6 98.6 95.1 93.5 the prediction of syntactic heads, and dependency relations by the models used in this study for the test set of ud_english-ewt r2.15. however, we suspect that the part-of-speech tagging may have been exposed to the models during its pre-and mid-training 3 phases. ## related work linearization techniques are essential for both constituency parsing @xcite and dependency parsing @xcite using sequence-to-sequence model with bracket-based representations, illustrated in in generative parsing using bracket-based representations, the tree structure in the output text is often invalid, which is one of the factors that reduces the accuracy of parsing, resulting in additional recovery procedure @xcite , or even redesign the topology of neural networks to ensure the output validity @xcite ). ## https://vintagedata.org/blog/posts/what-is-mid-training approach in this section, we describe a table-based representation of dependency structures, similar to the conll-u format, and explain how to construct instruction prompts for dependency parsing. ## experiments we conducted both supervised fine-tuning (sft) with low-rank adaptation (lora) @xcite and inference experiments for open llms on a high-performance cloud service 5,6 . for ope5 experiments were conducted on a google cloud a2 ultra instance with 8 × nvidia a100 gpus (80gb each), 96-core intel xeon cpus @ 2.20ghz, 1,360gb ram, and 5tb of ssd storage. the software environment included: ubuntu 22.04, cuda 12.1, python 3.11.9, pytorch 2.5.1, transformers 4.49.0, trl 0.15.2, peft 0.14.0, openai 1.68.2, unsloth 2025.3.18, and vllm 0.7.2. 6 the implementation used in the experiments is available on github. https://github.com/megagonlabs/llmpp nai models, sft was performed via the official web console foot_2 . we explored sft hyper-parameters foot_3 on the ud_english-ewt r2.15 development set and applied them to all experiments. we used simple tsv recovery process only restores the id and form on a row-by-row basis. ## conclusions we proposed a novel step-by-step prompting strategy for llm-based dependency parsing using a simple tabular format, achieving improved output validity and parsing accuracy across 17 languages. multilingual sft often outperformed monolingual models and generalized well to unseen languages.