Efficient EUD Parsing (2020)

Abstract: We present the system submission from the FASTPARSE team for the EUD Shared Task at IWPT 2020. We engaged with the task by focusing on efficiency. For this we considered training costs and inference efficiency. Our models are a combination of distilled neural dependency parsers and a rule-based system that projects UD trees into EUD graphs. We obtained an average ELAS of 74.04 for our official submission, ranking 4th overall.

## introduction latterly, the environmental impact of ai and nlp's dependency on deep neural networks has come under scrutiny @xcite . this has coincided with a renewed push for efficiency in nlp so as to make systems more easily used in different contexts, be it in hardware impaired conditions, large web-scale applications, or a host of other considerations @xcite @xcite . here we describe our contribution to the enhanced universal dependencies (eud) shared task at iwpt 2020 @xcite , where we have considered efficiency as well as bare accuracy performance. we combine linguistics and machine learning to develop efficient parsers, both with respect to training and inference. first we curtail the amount of training data we use, second we try distillation to create smaller networks for dependency parsers while maintaining accuracy, and third we develop a rule-based system to cast universal dependency (ud) trees as eud graphs. ## unravelling trees with shrewd rules rule-based systems are intrinsically efficient with respect to training time (barely a flash in the pan) and inference time (there is practically none). so we developed a simple rule-based system to enhance the existing dependency tree and reveal hidden dependencies in a cross-lingual setting using as few language specific rules as possible. beyond the basic enhancement of the original dependencies, there are four main phenomena that create new dependencies: relative clauses, control, conjunction and ellipsis. since our pipeline does not predict empty nodes, we decided to ignore ellipsis in this system. to deal with each of these phenomena, our algorithm needs to make a number of passes over each sentence. pass one -relative clauses and controls: the first pass of the algorithm iterates through each word in the sentence and creates enhanced relations according to the type of the original dependency. when necessary, it adds lemma and case information. if the current word is a relative pronoun/adverb, its antecedent is found by following its path to the root until an acl:relcl relation is met. then a ref edge is created between the word and its antecedent and an edge between the antecedent and the governor of the relativiser with the same relation type as the original relation (if the relative pronoun is the object of a verb then the antecedent becomes the object of that verb). if the word is the dependent of an xcomp relation, the algorithm looks for a subject amongst its controlling predicate's arguments. if a subject is found, it creates an edge between the subject and the current word of type nsubj(:xsubj) (or csubj in the case of a clausal argument). if no subject is available, the current word is stored in a separate list for later processing. if the word is the dependent of a conj relation, it too is stored in a separate list along with all other conjuncts. whenever we encounter an argument of the type subject, object or oblique, this information is kept for resolving subjects of controlled predicates. pass two -resolving conjunctions: we have two general functions, one for dependent level conjunctions and one for governor level conjunctions, and a few special cases. the dependent level function propagates the conjunction head's original relation to its conjuncts adapting it if necessary, for example in coordinated nmod with different adposition or case. the governor level function propagates the conjunction head's dependents to its conjuncts in the absence of similar dependents and according to morphological agreement. we have a special function that handles subjects of conjuncts because subjects are more diverse than other syntactic functions. in ud at least three relations can mark subjects, namely nsubj for nominal subjects, csubj for clausal subjects and expl used amongst other for syntactic subjects in non prodrop languages (e.g. "it rains"). subject edges also embed information about their governor, notably information about the voice as :pass when relevant. and, subjects can be absent altogether in prodrop languages, so we rely on morphological information to decide to propagate a given subject in these languages. pass three and onwards -sweeping up controls: once conjunctions have been resolved and more predicates have their arguments stored, the algorithm iterates over controlled predicates that do not have a subject after the first sentence traversal. several such iterations may be necessary since the number of times a predicate may be coordinated with a controlled verb itself already coordinated to another controlled verb is not bounded. like in the sentence "sam stood up and wanted to scream and start running." but in practice one iteration solves the vast majority of missing subjects. ## results and discussion despite focusing on efficiency, our official submission obtained an average elas of 74.04 which was the fourth best system (out of 9 full submissions). our improved score after training distilled models to convergence (or closer to convergence) obtained an average score of 76.14. the full breakdown of these results are shown in our system is competitive mainly by the grace of our rule-based system which obtains an average 98.20 elas when used on the gold development treebanks. and for the most part its performance echoes the quality of the predicted dependencies and tags used by the system as is seen in we also demonstrated that smaller networks can be competitive, even if in this context distillation does not perform as well as previously observed for ud parsing. and beyond that, we show that it is possible to train competitive models with less data and by doing so lowering the energy cost of training parsers. one potentially interesting result is that tamil performs noticeably better with distillation than either the full baseline model or the small model of the same size trained normally. it has the smallest training treebank out of all the treebanks used in the shared task. the other smaller treebanks also perform better with distillation, e.g the next three smallest treebanks french, lithuanian, and swedish all follow this trend but the increase in performance is less pronounced. perhaps smaller treebanks benefit from what is essentially ensemble training as it tempers a network's penchant for over-fitting. for our implementation there are two probability distributions as we are using a biaffine parser, one for head predictions and one for label predictions. 2 the student is also trained directly on the gold heads and labels using a categorical cross entropy loss, e.g. for the loss on head predictions: the total loss is therefore the combination of the kullback-leibler loss between the probability distributions of the teacher and the student for both head and label predictions with the cross entropy loss between the student predictions and the gold data: