Statistical Deep Parsing for S panish Using Neural Networks (2020)

Abstract: This paper presents the development of a deep parser for Spanish that uses a HPSG grammar and returns trees that contain both syntactic and semantic information. The parsing process uses a top-down approach implemented using LSTM neural networks, and achieves good performance results in terms of syntactic constituency and dependency metrics, and also SRL. We describe the grammar, corpus and implementation of the parser. Our process outperforms a CKY baseline and other Spanish parsers in terms of global metrics and also for some specific Spanish phenomena, such as clitics reduplication and relative referents.

## introduction the syntactic analysis of sentences is one of the key activities within the natural language processing pipeline, and is often seen as a necessary step that must be performed before extracting deeper information such as semantics. two main paradigms have historically dominated the work on syntactic analysis: constituency parsing, where the sentence is structured as a tree of linguistically motivated segments called constituents; and dependency parsing, where the sentence is structured as a set of bi-lexical dependencies with each word associated to another word that acts as its head. throughout the years there have been attempts to create deeper syntactic models that try to combine both syntactic and semantic notions in the same analysis, for example ccg @xcite , hpsg @xcite or tag @xcite . the process of analyzing a sentence in one of these formalisms is often called deep parsing. over the last years the use of deep neural networks has advanced the state of the art in many nlp tasks, and though some work has been done for performing deep parsing using these architectures (mainly in english and for ccg), few works have focused on the application of these techniques to other formalisms or other languages. in this work, we present a neural network architecture for deep parsing spanish sentences using the hpsg formalism. the rest of the paper is structured as follows: section 2 describes the grammar and corpus we use. section 3 introduces our parsing architecture and the baseline we compare it to. section 4 shows an analysis of results. section 5 describes relevant related work. finally, section 6 gives some conclusions and future perspectives. ## grammar and corpus we use a hpsg style grammar @xcite adapted to spanish. hpsg grammars are unification grammars that operate on feature structures. each word is defined by a feature structure that indicates the combinatorial properties of the word: how it can be combined to other words in order to form phrases. because of this, these grammars tend to have few rules, which are very generic. when applying a rule, the process of unification validates that both the conditions of the rule and the constraints imposed by the expressions being combined are met, and the resulting phrases inherit some of the features of their children in order to define their own combinatorial properties. ## parser development this section describes the lstm top-down parsing architecture we use in this work and the baselines we compare it to. ## experimental results in this section we present the experimental results for our lstm top-down neural network architecture and the baselines in terms of syntactic parsing and semantic role labeling. we also present an analysis of execution time for the different approaches. we compare our method to the cky baseline defined before and six well established baselines for spanish parsing. three of the baselines are the parsers contained in the freeling library @xcite , which include both dependency parsing and srl: txala @xcite , treeler foot_0 and a lstm implementation. the other baselines are the pretrained spanish models spanish of spacy foot_1 (models es core news sm and es core news md) and udpipe @xcite (model spanish-ancora), which only include dependency parsing but no srl. we show the perfor-mance of the (unrealistic) cky parser using gold supertags as an upper bound for the cky performance, but also the more realistic cky using the tags predicted by the supertagger. ## related work most of the work over the last years on deep parsing using neural network architectures has been done for the ccg formalism and for english language. for example @xcite uses a recurrent neural network for improving the supertagging and parsing accuracy for ccg, while @xcite describes a neural networks architecture that performs ccg parsing. for hpsg, the work by @xcite is very relevant as they try to derive a hpsg grammar from the penn treebanks in english and chinese and use an self-attention based mechanism followed by a cky decoder to parse them, obtaining very good results. in our work, however, we focus in hpsg for the spanish language and define a different architecture for parsing: a top-down approach with lstms. our approach to hpsg development is similar to another relevant statistical parser for hpsg in english, the enju parser @xcite , which was created by transforming the english penn treebank through a set of rules into hpsg format and used this transformed corpus to train a statistical model. another relevant precedent for hpsg is the lkb @xcite platform together with the pet @xcite parser, used to define hpsg grammars in english and other languages (in particular the spanish resource grammar @xcite ), although they do not use neural networks for the parsing process. we differ from the spanish resource grammar in that we derive the feature structures from a large corpus instead of building them manually, with the aim of building a purely statistical parser that could, for example, handle slightly ungrammatical sentences or out of vocabulary concepts better. the cky with supertagging method we compare to in this work follows a similar approach to the parsing methods used for deep syntactic grammars such as ccg @xcite , hpsg @xcite @xcite and tag @xcite . much more work on applying neural network architectures to parsing has been done for the two more classical syntactic paradigms: constituency and dependency parsing. for constituency parsing, @xcite defines a class of recurrent neural networks that combine pairs of words and builds a tree bottom-up, with the aim of improving sentiment analysis in english sentences. on the other hand, @xcite frames the parsing process as a translation between a sentence in natural language and the bracketed representation of its parse tree. other authors train neural models to predict the actions to be performed in a transition based shift-reduce constituency parser @xcite , combining it with a sequence-to-sequence modeling @xcite , or encoding the parsing stack using a recurrent network @xcite . in @xcite a transition based constituency parser gets good results for english and french using lstms for representing word spans instead of partially derived trees. another approach that focuses on determining the word spans in the tree is used in @xcite , which describes a top-down parser that greedily splits a sentence in constituents and assigns labels to them, processing the text spans with lstms in order to generate an intermediate representation. this approach is the most similar we found to the one we use, the main differences are that they work for french, they use a standard constituency grammar while ours is a strictly binarized hpsg grammar, and we add a further step for predicting the argument structure of predicates for the generated trees. related approaches for english include: @xcite , that calculates the score and label for each sentence span then uses cky to find the optimal tree; @xcite , that predicts the syntactic distances for words and builds the tree top-down using these distances. compared to english, there are few works that focus on spanish parsing. for constituency parsing, @xcite tries two approaches to improve standard pcfg parsers: including morphological information in the probabilistic model, and a reranking method with max-margin criterion trained over a set of global features from the parse trees. their evaluation against the cast3lb corpus, a subset of ancora, achieves a constituent f1 of 83.6 and 85.1 respectively. @xcite experiments with spanish parsing using a pcfg with latent annotations with a simplified tagset, achieving 85.47 f1 over the cast3lb corpus. there is considerably more work done for dependency parsing and srl in spanish, beginning in the conll-x @xcite and @xcite shared tasks. the best las achieved for spanish were 82.3 (max span tree approach) and 81.3 (transition based approach). later on, @xcite describes a dependency grammar and a rule-based dependency parser for spanish (one of the freeling parsers @xcite ) transforming the result of a shallow parser, achieving 81.13 uas and 73.88 las. in @xcite they describe a set of experiments using malt-parser @xcite to determine how much corpus size, sentence length or other factors contribute to the dependency spanish parsing performance. the latest efforts in dependency parsing have generally focused on using the universal dependencies @xcite framework. for example in conll 2017 @xcite and @xcite shared tasks on multilingual parsing from raw text to universal dependencies the best parsers achieve las of 87.29 @xcite and 90.93 @xcite respectively for spanish. ## conclusions we presented a statistical deep parser for spanish that outputs trees in the hpsg formalism. the parser uses a top-down approach for building the tree followed by a second step for calculating the arguments, both steps are implemented using lstm neural networks. the parser gets good results for performance compared to a cky baseline and to other parsing baselines in spanish, achieving 87.57% unlabeled and 82.06% labeled constituency f1, and 91.32% unlabeled and 88.96% labeled dependency accuracy. it also achieves good performance for srl, getting 87.68% unlabeled and 80.66% labeled f1, and beats the baselines for some particular spanish phenomena we analyzed. although the results improve over the baseline methods, there is still room for improvement, especially in terms of execution time. we plan to build a unified architecture that could perform both the syntactic and the arguments step at the same time, which could help lower the execution times and it might also help the network generalize better in order to improve the prediction of labels in srl. it would be also very interesting to try this approach to other languages such as english, and also languages that share some of the characteristics we analyzed such as italian or french.