Dependency Parser for B ulgarian (2024)

Abstract: This paper delves into the implementation of a Biaffine Attention Model, a sophisticated neural network architecture employed for dependency parsing tasks. Proposed by Dozat and Manning, this model is applied to Bulgarian language processing. The model’s training and evaluation are conducted using the Bulgarian Universal Dependencies dataset. The paper offers a comprehensive explanation of the model’s architecture and the data preparation process, aiming to demonstrate that for highly inflected languages, the inclusion of two additional input layers - lemmas and language-specific morphological information - is beneficial. The results of the experiments are subsequently presented and discussed. The paper concludes with a reflection on the model’s performance and suggestions for potential future work.

## introduction the paper presents an implementation of a neural network-based dependency parser using tensorflow and keras 1 . the parser is trained and evaluated on the bulgarian universal dependencies dataset. the article introduces a model in its intermediate stage of development, with the current focus entirely on dependency analysis. at this stage, it does not aim to predict parts of speech, lemmas, etc. (these are expected to be added to the model later). for this reason, only the results for labeled 1 the code, written in python 3.11, has been tested on ubuntu 22.04.3 lts running on wsl2. it is accessible at https://github.com/nassoo/dependecy_parser . the necessary packages are enumerated in the requirements.txt file. additionally, an environment.yml file is provided for convenience. note that the virtual environment includes packages not directly used in the notebook, such as pytorch and unlabeled attachment scores will be presented. ## data preparation the ud treebanks come in various formats, including conll-u (a format used for linguistic treebanks in the conference on natural language learning), tensorflow datasets (tfds), and huggingface datasets. given that this project uses tensorflow, tfds is the most convenient format. the treebanks are always split into training, development, and test sets. the bulgarian ud dataset includes 8,907 training sentences, 1,115 development sentences, and 1,116 test sentences. this distribution, which approximates an 80-10-10 split for training, validation, and testing respectively, is typical for ud treebanks. given its suitability for developing a neural network parser, no additional splitting of the data is required. in dozat and manning's architecture, only specific parts of the data are utilized. the tokens column, which contains the tokenized sentence, and the upos column, which includes universal part-of-speech tags, are used to train the parser. the deprel column, with dependency relations, and the head column, indicating word head indices, serve as the parser's targets. given the focus on using the parser for bulgarian, as highlighted in the experiments and results section, two additional columns are incorporated into the input layers: lemmas (containing the lemmatized forms of the words) and xpos (providing more detailed morphological language-specific information, although these tags are not consistent across languages). after loading the data, the next step is to construct vocabularies. these vocabularies consist of unique words, lemmas, universal part-of-speech tags, language-specific tags, and dependency relations from the training set, all of which are converted into numerical representations for model processing. the vocabularies facilitate the conversion of words and tags into these representations, while the dependency relations are used to construct the target vectors for the parser. the unique values from the dataset are extracted and stored in tensorflow's hash tables for more efficient tensor handling. special tokens are added to equalize input sequence lengths, represent the root of the dependency parse tree (the dummy token that governs the main verb of the sentence), and denote unknown words, tags, or relations. these tables are utilized to convert between numerical ids and their corresponding labels during both training and prediction. a configuration management module was implemented to streamline experimentation and reproducibility. this module incorporates hyperparameter loading from a json file and manages essential data structures, such as hash tables, necessary for both model training and data preprocessing. by centralizing configuration parameters, the development process was optimized, facilitating efficient exploration of the hyperparameter space. the vectorization process is a critical step in preparing the data for the neural network. this process transforms the dataset into a format where each element is a tuple of inputs and outputs. the inputs consist of tokenized sentences, universal part-of-speech tags, lemmas, languagespecific tags, and sentence lengths. the outputs are word head indices and dependency relations. word heads are cast to numbers (specifically, tf.int32 to meet the requirements of tf.lookup.statichashtable), while the remaining elements are encoded as integers using their respective hash tables. to ensure uniformity, sequences are either padded or truncated to a predetermined length, as specified in the configuration. each sequence begins with a dummy value, and an additional token is included in the sentence length to represent the root of the dependency parse tree. the vectorized inputs and outputs are then used to generate a new dataset. the data preparation process includes batching and shuffling steps to optimize the training of the neural network. in the batching step, the dataset is divided into smaller groups or batches. each batch contains a certain number of examples that the model will process simultaneously. the size of these batches is a configurable parameter and can be adjusted based on the computational resources available. the shuffling step randomizes the order of the examples in the dataset. this is done to ensure that the model does not learn any unintended patterns from the order of the examples, which could lead to overfitting. this step is performed before each epoch, i.e., each pass through the entire dataset, to ensure that the model is exposed to a different order of examples in each epoch. this dataset, vectorized and batched, can be used for model training or prediction. ## the model this chapter introduces the primary neural network model, designed for dependency parsing. built with tensorflow and the keras api, the model incorporates several components. these include embedding layers for words, lemmas, pos, and language-specific tags, bilstm layers for sentence encoding, and mlp layers for transforming the sentence encodings into a form suitable for predicting arcs and relations between words. the model also features biaffine layers, which take these transformed outputs and make the actual predictions of arcs and relations between words. the model is equipped with a custom loss function and utilizes the adam optimizer for training. it also includes metrics for monitoring the loss and accuracy during both training and evaluation. ## experiments and results the model underwent testing on the bulgarian ud treebank with varying hyperparameters, and the results were evaluated using the unlabeled attachment score (uas) and labeled attachment score (las). the uas measures the proportion of words correctly attached to their head, while the las measures the proportion of words correctly attached to their head with the correct dependency relation. optimal performance was achieved with a higher dropout rate of 0.5, as opposed to the 0.33 reported by dozat and manning. this can be attributed to the smaller size of the bulgarian ud treebank compared to the treebanks used in their study, necessitating stronger regularization to prevent overfitting and enhance generalization. a substantial improvement (0.86% for uas and 1.31% for las) was observed upon the inclusion of lemmas and language-specific morphological information as input layers. this enhancement is anticipated given the complexity of the bulgarian language and the significance of morphological information in parsing it. another enhancement involved replacing the traditional embedding layer for input tokens with roberta embeddings. this change leverages the pre-trained model @xcite to generate contextualized embeddings, which capture richer semantic and syntactic information. since roberta is used in the current version of the parser only for token vectorization, the improvement is not particularly large (0.21% for uas and 0.49% for las). however, it still demonstrates the advantage of using large language models. ## future work the model performs well on the bulgarian ud treebank, with its results approaching those of state-of-the-art parsers. however, further improvements are necessary. potential areas for enhancement include: while the focus of this study was specifically on bulgarian, the model should be evaluated with other highly inflected languages to determine if the inclusion of lemmas and morphological tags improves performance for these languages as well. ## conclusion the implementation of the neural networkbased dependency parser, utilizing tensorflow and keras, gave near state-of-the-art results (uas: 93.32, las: 89.73). the parser underwent training and evaluation on the bulgarian universal dependencies dataset, yielding competitive results and thereby demonstrating the efficacy of the proposed architecture. although the original model by dozat and manning was evaluated on considerably larger datasets and languages with simpler morphological structures, the results are comparable. for instance, the parser's results for bulgarian outperforms the original model's scores for chinese and czech, the latter possessing one of the largest treebanks. the parser can predict the dependency syntax structure of bulgarian sentences, and the displacy module from the spacy library can visualize these predictions. the parser's performance can be further enhanced by optimizing its hyperparameters and employing superior word embeddings.