A Simple and Effective Dependency Parser for T elugu (2020)

Abstract: We present a simple and effective dependency parser for Telugu, a morphologically rich, free word order language. We propose to replace the rich linguistic feature templates used in the past approaches with a minimal feature function using contextual vector representations. We train a BERT model on the Telugu Wikipedia data and use vector representations from this model to train the parser. Each sentence token is associated with a vector representing the token in the context of that sentence and the feature vectors are constructed by concatenating two token representations from the stack and one from the buffer. We put the feature representations through a feedforward network and train with a greedy transition based approach. The resulting parser has a very simple architecture with minimal feature engineering and achieves state-of-the-art results for Telugu.

## introduction dependency parsing is extremely useful for many downstream tasks. however, robust dependency parsers are not available for several indian languages. one reason is the unavailability of annotated treebanks. another reason is that most of the existing dependency parsers for indian languages use hand-crafted features using linguistic information like part-of-speech and morphology @xcite @xcite which are very expensive to annotate. telugu is a low resource language and there hasn't been much recent work done on parsing. most of the previous work on telugu dependency parsing has been focused on rule based systems or datadriven transition based systems. this paper focuses on improving feature representations for a low resource, agglutinative language like telugu using the latest developments in the field of nlp such as contextual vector representations. contextual word representations @xcite @xcite are derived from a language model and each word can be uniquely represented based on its context. one such model is bert @xcite . bert vectors are deep bidirectional representations pre-trained by jointly conditioning on both left and right context of a word and have been shown to perform better on variety of nlp tasks. in this paper, we use bert representations for parsing telugu. we replace the rich hand-crafted linguistic features with a minimal feature function using a small number of contextual word representations. we show that for a morphologically rich, agglutinative language like telugu, just three word features with good quality vector representations can effectively capture the information required for parsing. we put the feature representations through a feed forward network and train using a greedy transition based parser @xcite . past work on telugu dependency parsing has only been focused on predicting inter-chunk dependency relations @xcite @xcite @xcite . in this paper, we also report parser accuracies on intra-chunk annotated telugu treebank for the first time. ## related work extensive work has been done on dependency parsing in the last decade, especially due to the conll shared tasks on dependency parsing. creation of universal dependencies @xcite led to an increased focus on common approaches to parsing several different languages. there were new transition based approaches making use of more robust input representations @xcite and improved network architectures @xcite . graph based approaches to dependency parsing have also become more common over the last few years @xcite . however, there hasn't been much recent work on parsing indian languages and much less on telugu. most of the previous work on telugu dependency parsing has been focused on rule based systems @xcite or data-driven transition based systems @xcite using malt parser @xcite . the malt parser uses a classifier to predict the transition operations taking a feature template as input. the feature templates used in telugu parsers commonly consist of several hand-crafted features like words, their partof-speech tags, gender, number and other morphological features @xcite . there has been some work done on representing these linguistic features using dense vector representations in a neural network based parser @xcite . recent developments in the field of nlp led to the arrival of contextual word vectors @xcite @xcite and their extensive use in downstream nlp tasks, from pos tagging @xcite to more complex tasks like question answering and natural language inference tasks @xcite . in this paper, we train a bert-baseline model @xcite on telugu wikipedia data and use these vector representations to improve telugu dependency parsing. ## telugu dependency treebank we use the telugu treebank made available for icon 2010 tools contest. we extend this treebank by another 900 sentences from the hcu telugu treebank. the size of the combined treebank is around 2400 sentences. the treebank is annotated using computational paninian grammar @xcite proposed for indian languages. the treebank is annotated at interchunk level @xcite in ssf @xcite format. only chunk heads in a sentence are annotated with dependency labels. we automatically annotate the intra-chunk dependencies (bhat, 2017) using a shift-reduce parser based on context free grammar rules within a chunk, written for telugu foot_0 . annotating the intrachunk dependencies provides a complete parse tree for each sentence. the treebank is converted from ssf to conll-x format @xcite foot_1 . ## our approach we propose to replace the rich hand-crafted feature templates used in malt parser systems with a minimally defined feature set which uses automatically learned word representations from bert. we do not make use of any additional pipeline features like pos or morphological information assuming this information is captured within the vectors. we train a bert baseline model @xcite on the telugu wikipedia data, which comprises 71289 articles. we use the ilmt tokenizer included in the telugu shallow parser foot_2 to segment the data into sentences. the sentence segmented data consists of approximately 2.6m sentences. we convert all of the data from utf to wx foot_3 notation for faster processing. we use byte-pair encoding @xcite to tokenize the data and generate a vocabulary file. we pass this vocabulary file to bert 5 for pre-training. after pre-training, we extract contextual token representations for all the sentences in the treebank from the pre-trained bert model. in case a single word is split into multiple tokens, we treat these tokens as continuous bag of words and add the representations of all the tokens in a word to obtain the word representation. we find that this approach works better than considering only the first word-piece vector @xcite . we use these word representations as input features to the parser. our feature function is a concatenation of a small number of bert vectors and we integrate it into a transition based parser. the specific details are mentioned in section 4.2 ## experiments and results the telugu dependency treebank is quite small in size consisting of only 2400 sentences. we also observe that the sentence length and quality of annotation in the treebank is not uniform and has a high amount of variation. we therefore evaluate our parser on the treebank using ten-fold crossvalidation. we report the cross-validation accuracies on both inter-chunk ( we compare these results with a baseline using only word2vec word representations and subsequently adding part-of-speech (pos) and suffix representations described in @xcite . we also try to reproduce @xcite experiments on both inter-chunk and intra-chunk annotated treebanks. @xcite report their best results for telugu on the inter-chunk annotated treebank using word, pos and suffix representations. their results are reported on a test set and since their exact dataset is not available, we report average 10-fold cross validation accuracies. the reproduced results are listed in implementation details: the parser comprises of simple feed forward neural network with one hidden layer consisting of 1024 hidden units and a relu activation function and a dropout layer with dropout probability of 0.2. we use xavier uniform initialization @xcite to initialize the network parameters and adam optimizer (diederik p. kingma, 2015) with default momentum and learning rates provided by pytorch. we use bert baseline model for pre-training and each bert token representation is of dimension 768. arc-standard vs arc-eager: we experiment with both arc-standard @xcite and arc-eager @xcite transition systems and find that arc-standard works better in our case ( ## error analysis in this section we look at some of the most common errors made by this parser and try to understand why those errors might be occurring. we evaluate the parser on a test-set of 240 sentences. the most frequently occurring errors are k1(agent/subject) and k2(object/patient) mismatch, k1 is labeled as k2 and vice versa. k1 and k2 are the most fre-quently occurring labels after root . 78% sentences in the test-set contain k1 dependency and 50% sentences contain k2 dependency. k1 is labeled as k2 15% of the time and k2 is labeled as k1 18% of the time. these errors are usually seen when the words occur without case-markers. in these cases, k1 and k2 can be distinguished by looking at the verb agreement. fixing these two errors would greatly improve the parser. other frequently occurring errors are confusion between k2 and k4(recipient) since they sometimes take the same case-markers, nmod and nmod adj, vmod and adv , sent adv labels. the label vmod is ambiguous in general and can be easily confused with adverbs. ## conclusion and future work we present a simple yet effective dependency parser for telugu using contextual word representations. we demonstrate that even with vectors trained on a small corpus of 2.6m sentences, we can reduce the need for explicit linguistic features in deep learning based models. we show based on the results of the parser that bert vectors effectively capture much of the linguistic information required for parsing. we also show that with good vector representations, a small feature set is more effective for a morphologically rich, agglutinative language like telugu. future work could include finding a way to incorporate other linguistic features like case-markers, gender, number, person, tense, aspect and verb agreement information into the parser.