The T hai U niversal D ependency Treebank (2025)

Abstract: Automatic dependency parsing of Thai sentences has been underexplored, as evidenced by the lack of large Thai dependency treebanks with complete dependency structures and the lack of a published evaluation of state-of-the-art models, especially transformer-based parsers. In this work, we addressed these gaps by introducing the Thai Universal Dependency Treebank (TUD), a new Thai treebank consisting of 3,627 trees annotated according to the Universal Dependencies (UD) framework. We then benchmarked 92 dependency parsing models that incorporate pretrained transformers on Thai-PUD and our TUD, achieving state-of-the-art results and shedding light on the optimal model components for Thai dependency parsing. Our error analysis of the models also reveals that polyfunctional words, serial verb construction, and lack of rich morphosyntactic features present main challenges for Thai dependency parsing.

## introduction dependency parsing is the task of identifying the dependencies between words in a sentence according to the syntax of its language. many downstream nlp tasks rely on the support of automatic grammatical analysis of sentences, such as event extraction @xcite and relation extraction @xcite . to allow multilingual approaches to automatic syntactic analysis despite cross-lingual variation, universal dependencies * corresponding author. dependency parsing for the thai language is currently not viably accurate due to the lack of large annotated datasets. we found only two publicly available dependency treebanks for thai: thai-pud and blackboard treebank @xcite . thai-pud contains 1,000 sentences of complete dependency annotation, which might not be enough for training dependency parsers. blackboard treebank contains 38,588 sentences, but it is only annotated with arcs and does not comply with the standard of ud. the lack of large and reliable training data presents a major obstacle in the field of thai dependency parsing. in addition, we found no previous studies on the application and evaluation of state-of-the-art transformer-based models for thai dependency parsing, while transformer-based parsers have achieved state-of-the-art performance in many languages @xcite @xcite . in this work, we create the thai universal dependency treebank (tud), a large corpus annotated with dependency relations according to the ud framework. our first challenge of this annotation project is that thai has no word or sentence boundaries. we first had to segment raw text into words and then group them into sentences using the boundaries implied by the annotated dependencies. our second challenge is the adaptation of the ud framework to fit thai syntactic idiosyncrasies. we provide guidelines for annotating thai serial verb constructions and grammatical particles, which are prevalent in the language but have not been explicitly covered in previous works or in the ud guideline itself. tud is publicly released on github foot_0 along with the source code and raw data. our main contributions in this work can be summarized as follows: ## related work previously, two thai dependency treebanks were released, namely, thai-pud foot_1 and blackboard treebank. 3 thai-pud is a small treebank consisting of 1,000 sentences translated from various european languages and annotated according to the ud framework. the size of this dataset is small compared to other ud treebanks of high-resource languages. it was created as part of the conll 2017 and 2018 shared tasks @xcite but the organizer decided to exclude it from the former due to ''consistency issues,'' the details of which were not clarified. blackboard treebank is a large non-ud treebank consisting of 130,561 thai clauses. this dataset does not provide a complete annotation. subordinate and relative clauses are detached and analyzed as their own trees separate from the main clauses. moreover, blackboard treebank only provides dependency arcs and not dependency types. traditional dependency parsers are either transition-based or graph-based. transition-based models attach dependencies incrementally based on a sequence of actions called ''transitions'' while graph-based models assign scores to all possible dependencies and then extract a valid tree with the best score @xcite . recent works have proposed novel transition systems @xcite , improved postprocessing of traditionally parsed results @xcite , incorporated higher-order information @xcite , and employed joint training with related tasks @xcite to boost performance. so far, no thai dependency parsers have achieved performance competitive with state-ofthe-art models in high-resource languages such as english, for which a result as high as 97.42 uas and 96.26 labeled attachment score (las) has been reported @xcite . @xcite utilize convolutional neural networks and long short-term memory models as encoders for transition-based and graph-based parsers, achieving 78.48 uas (las not reported) on thai-pud. @xcite formulates the task as sequence labeling for thai dependency parsing and utilizes a pretrained transformer, achieving 77.53 las (uas not reported) on thai-pud. language-agnostic models, capable of parsing any structures in any languages, have also been tested on thai but tend to perform poorly. @xcite relies only on static embeddings and achieves only 0.88 uas and 0.65 las on thai-pud. @xcite leverage multilingual bert @xcite but still achieve only 49.05 uas and 26.06 las on thai-pud. ## annotation guidelines for thai universal dependency treebank we follow the ud annotation guidelines for the most part. however, we need to clarify parts of the guidelines to accommodate the idiosyncrasies of the thai language. ## thai universal dependency treebank creation process our text sources cover a wider range of document types and more diverse domains than thai-pud. the raw text for tud was taken from two sources: the thai national corpus @xcite and the november 2020 dump of thai wikipedia. we randomly sampled 5,000 paragraphs from selected documents while making sure the documents were from a wide range of document types, including news articles, wikipedia articles, essays, advertisement, interviews, and stories. the selected documents covered diverse topics, such as politics, crime, entertainment, sport, history, religion, culture, and science. then each paragraph was automatically tokenized using the dictionary-based (newmm) tokenizer from the library pythainlp @xcite . annotation was done on datasaur's labeling platform. 7 we initially recruited a total of 10 annotators, all of whom either completed at least a bachelor's degree education in linguistics or were in the process of getting one, and one annotation manager/instructor specialized in thai linguistics. each annotator had four tasks: 1) correct tokenization errors resulting from the preprocessing step, 2) label each token's universal pos (upos), 3) identify dependency arcs, and 4) label each arc's universal dependency relation (deprel) without a subtype. the lemma field of conll-u format was not annotated because the thai language does not have inflectional morphology. to train the annotators, they were first instructed to study the manual and review the concepts of head-dependent relationship, syntactic distribution, and polyfunctional words. the instructor also provided some parsed examples from the real data. after that, the annotators were assigned to annotate a set of pilot sentences specifically designed to test their understanding. once pilot sentences were completely annotated, the instructor would identify common errors in the pilot annotation and re-emphasize what the correct annotation should be. however, after a manual quality review of each annotator's work, only two were identified as producing the highest-quality annotations. consequently, we chose these two annotators to complete the remainder of the dataset. to assess annotation consistency after training, we randomly sampled 20 sentences, containing 399 tokens, from the treebank and had them fully annotated separately by the two annotators. for upos and dependency relations, the cohen's kappa scores @xcite were 0.92 and 0.84, respectively. for dependency arcs, we could not use cohen's kappa since the arcs can be arbitrarily long. instead, we calculated the ''uas'' and ''las'' scores by treating one annotator as a ''gold-standard'' annotation. our uas was 0.85 and our las was 0.78. we considered these four scores to be reasonably high given the complex nature of dependency annotation. finally, the annotators went on to annotate real sentences prepared in the previous step. during annotation, any perceived errors or ambiguities not covered by the annotation manual were recorded and adjudicated on a case-by-case basis. after all paragraphs had been annotated, the data was exported into conll-u format and then automatically split into separate trees based on the annotated dependency links, i.e., all connected tokens were grouped as individual trees. any resulting trees that either 1) contained only a single token, 2) were not completely labeled, or 3) were not valid dependency trees such as trees that the final treebank consists of 3,627 valid and completely labeled trees. we randomly split tud into three splits, train, dev, and test, using an 8:1:1 ratio. the most common upos tags by far are noun and verb, representing main content words, while the least common tag is sym ( ## thai dependency parsing models the models to be evaluated in our experiment can be categorized into two types: 1) baseline models and 2) strong open-source models. we used the baseline models to identify the effects of individual components and methods on thai dependency parsing performance. we tested the strong open-source models to test the quality of our dataset and find out the accuracy rates that could be expected. all models used the gold-standard tokenization that comes with the treebank. ## results and discussion in running these series of experiments, we want to know 1) which parsing method is better for thai? 2) is phayathaibert a better thai encoder we repeated this analysis with las as a target variable ( which parsing architecture is better for thai? our results suggest that transition-based models perform significantly better than graph-based models in uas (@xmath0.05) but perform similarly in las (@xmath1.937). the overall best model is stanza, although it is a graph-based model. this is because stanza also employs additional techniques not present in other models, namely, 1) augmenting each token's representation with fast-text's static pretrained word embeddings @xcite , and 2) including terms that explicitly model the probability of each link between a head and a dependent based on their distance and linear order @xcite . nevertheless, note that many of our transition-based models achieve very competitive results with stanza. in future work, it would be interesting to see if transition-based models could outperform it once additional techniques are implemented. is phayathaibert a better thai encoder than wangchanberta? our results show that phayathaibert significantly leads to better performance than wangchanberta in dependency parsing (@xmath2.05), supporting the claim in its paper @xcite that phayathaibert is an improved version of wangchanberta. our models outperformed thai dependency parsers in previous works, which do not use these pretrained language models @xcite @xcite , highlighting the importance of large, contextualized, language-specific, pretrained encoders. do sentence embeddings and super-token embeddings help? contrary to altıntas ¸and tantȗg (2023), we found that augmenting token embeddings with sentence embeddings and super-token embeddings does not result in significant improvement (uas @xmath3.174; las @xmath4.195). nevertheless, many instances of improvement over non-augmented models are observed even in transition-based models, which are not experimented with in the original work. this suggests that the method itself has some potential, although it does not improve all the models consistently enough to be statistically significant. more experimentation is therefore needed to find the conditions in which the method works best. do gold-standard upos tags play an important role? gold-standard upos tags clearly lead to significantly superior performance (@xmath5.05). the same cannot be said for automatically tagged upos, however, as using them does not result in significant improvement over not using them at all (uas @xmath6.259; las @xmath7.227), and in some instances, it even leads to slight degradation ( ## error analysis in this section, we will identify some challenges unique to thai dependency parsing, which should give us valuable insights that could help close the gap in performance between thai and high-resource languages. the full test split prediction of every model is available on tud's github repository for later, more extensive investigation. the vast majority of upos confusions were caused by polyfunctional words. polyfunctional words are words that can function in more than one role. the most common mistakes in this category were caused by the fact that thai has a large number of deverbal grammaticalizations. among the most versatile, and thus the most confused ( distinction between common nouns and proper nouns in thai is unclear. in many languages, proper nouns are capitalized and cannot be accompanied by articles. these features do not exist in thai so proper nouns have to be identified either from context or from world knowledge, causing confusion between the noun and propn tags. this tends to happen with foreign proper names, such as 'dispatch' and 'matrix' ( the roles of thai relative pronouns are difficult to determine without world knowledge. in thai, like in many languages, the relative pronouns , , ('that/which') are fronted in relative clauses. this fronting eliminates word order as a clue for identifying their syntactic roles. combining with the pro-drop nature of thai and the absence of subject-verb agreement, even less information is available. tree (11) illustrates this ambiguity. only by using the world knowledge that rice/meal is inanimate and thus cannot eat anything can we determine that the correct annotation for ? is obj and not nsubj. parsers cannot easily distinguish between compounds and syntactic phrases. determining whether a sequence of words constitutes a compound and not a phrase in thai relies on multiple factors and the two classes can be described to exist on a spectrum @xcite . this distinction is notoriously difficult because the difference is quite subtle @xcite . tree (12) shows some compounds that were mistaken as syntactic phrases by the parsers. means a villager (compound noun) and not a person who stays in a house (noun phrase). means a servant (compound noun) and not a user (noun phrase). the correct dependencies depend on the meaning of these phrases in a larger context of a sentence. verbal and verbal-like dependents were often confused with one another. without explicit linking words, structures involving verbal dependents @xcite and verbal-like dependents (advmod, aux), which frequently share word forms with verbs, can all appear like sequences of two verb phrases, which leads to ambiguity. this confusion was also exhibited by our annotators during pilot annotation since distinguishing between them often requires performing non-trivial linguistic tests. tree (13) shows some examples. despite looking superficially similar, the sentence on the left describes simultaneous actions (lying down and sleeping as opposed to lying down and doing something else) while the sentence on the right usually does not because the word cannot be interpreted as a verb despite having the same form as the word meaning 'to come.' it is rather an adverb indicating that the action has already taken place and has some implication affecting the present. although in uncommon context, it could still be interpreted as a verb, e.g., when arriving at a place while lying down. many mistakes were correlated with ''chain dependencies.'' some relations, most notably nmod and compound (and acl in thai-pud), allow two or more tokens of the same upos to be chained together into one structure. this creates ambiguity when subsequent dependents need to be attached to one of the tokens in the chain. to illustrate, tree (14) shows part of a real tree from tud. the text that follows the above structure is a relative clause 'which takes people's lives,' which can potentially be attached to any of the four nouns in the chain. lacking world knowledge that flood kills people, many parsers attached the relative clause to 'news' or 'event' instead of 'great flood.' we perform further analysis by investigating the relative position of the true heads and the heads incorrectly predicted by the parsers and discover that the two tokens often form structures involving the aforementioned relations ( ## conclusion in this work, we introduce the thai universal dependency treebank (tud), a new thai treebank consisting of 3,627 trees annotated according to the ud framework. we found that strong open-source graph-based parsing system with the state-of-the-art plm achieves the best results on our dataset. we also found that an accurate pos tagger is crucial to good parsing results, but the current automatic pos tagger is not accurate enough to improve parsing performance. our error analysis shows that polyfunctional words and ambiguities created by the lack of inflectional morphology present the main challenges for dependency parsing in thai.