Revisiting Supertagging for faster HPSG parsing (2024)

Abstract: We present new supertaggers trained on English HPSG-based treebanks and test the effects of the best tagger on parsing speed and accuracy. HPSG treebanks are produced automatically by large manually built grammars and feature high-quality annotation based on a well-developed linguistic theory. The English Resource Grammar treebanks include diverse and challenging test datasets, beyond the usual WSJ section 23 and Wikipedia data. HPSG supertagging has previously relied on MaxEnt-based models. We use SVM and neural CRF- and BERT-based methods and show that both SVM and neural supertaggers achieve considerably higher accuracy compared to the baseline and lead to an increase not only in the parsing speed but also the parser accuracy with respect to gold dependency structures. Our fine-tuned BERT-based tagger achieves 97.26% accuracy on 950 sentences from WSJ23 and 93.88% on the out-of-domain technical essay The Cathedral and the Bazaar. We present experiments with integrating the best supertagger into an HPSG parser and observe a speedup of a factor of 3 with respect to the system which uses no tagging at all, as well as large recall gains and an overall precision gain. We also compare our system to an existing integrated tagger and show that although the well-integrated tagger remains the fastest, our experimental system can be more accurate. Finally, we hope that the diverse and difficult datasets we used for evaluation will gain more popularity in the field: we show that results can differ depending on the dataset, even if it is an in-domain one. We contribute the complete datasets reformatted for Huggingface token classification.

## introduction we present new supertaggers for english and use them to improve parsing efficiency for head-driven phrase structure grammars (hpsg). grammars have been gaining relevance in the natural language processing (nlp) landscape @xcite , since it is hard to interpret and evaluate the output of nlp systems without robust theories. head-driven phrase structure grammar @xcite is a theory of syntax that has been applied in computational linguistic research (see @xcite . at the core of such research are precision grammars which encode a strict notion of grammaticalitytheir purpose is to cover and generate only grammatical structures. they include a relatively small set of phrase-structure rules and a large lexicon where lexical entries contain information about the word's syntactic behavior. hpsg treebanks (and the grammars that produce them) encode not only constituency but also dependency and semantic relations and have proven useful in natural language processing, e.g. in grammar coaching @xcite @xcite , natural language generation @xcite , and as training data for high precision semantic parsers @xcite @xcite . assuming a good parse ranking model, a treebank is produced automatically by parsing text with the grammar, and any updates are encoded systematically in the grammar, with no need of manual treebank annotation. 1 hpsg parsing, which is typically bottom-up chart parsing, is both relatively slow and ramhungry. often, more than a second is required to parse a sentence (see here we contribute to the second line of research, aka supertagging, a technique to discard unlikely interpretations of tokens. @xcite and @xcite used maximum entropy-based models trained on a combination of gold and automatically labeled data from english, requiring large-scale computation. they report an efficiency improvement of a factor of 3 for the parser they worked with @xcite and accuracy improvements with respect to the parseval metric. we present new models for hpsg supertagging, an svm-based one, a neural crf-based one, and a fine-tuned-bert one, and compare their tagging accuracy with a maxent baseline. we now have more english gold training data thanks to the hpsg grammar engineering consortium's treebanking efforts @xcite @xcite . 2 it makes sense to train modern models on this wealth of gold data. then we use the supertags to filter the parse chart at the lexical analysis stage, so that the parser has fewer possibilities to consider. we report the results of parsing all of the test data associated with the english hpsg treebanks @xcite in comparison with parsing the same data with the same parsing algorithm but with no tagging at all, as well as with the integrated memm-based tagger. if we use the tagger with some exceptions, our system is the most accurate one (using the partial dependency match metric). it is not faster that the memm-based tagger integrated into the parser for production mode, although it is of course much faster than parsing without tagging (by a factor of 3). the paper is organized as follows. in §2, we give the background necessary for understanding the provenance of our training data. §3 presents the methodology, starting from previous work ( §3.1). we then describe our training and evaluation data ( §3.2), and finally how we trained the new supertag-gers ( §3.3). in §4, we present the results: first for the accuracy of the supertagger ( §4.1) and then for the parsing experiments, including parsing speed and parsing accuracy ( §4.2). we trained the neural models with nvidia geforce rtx 2080 gpu, cuda version 11.2. the svm model and the maxent baseline were trained using intel core i7-9700k 3,60hz cpu. the parser was run on the same cpu. the code and configurations for the reported results as well as the datasets are online. 3 the original data we used is publicly available. 2 further details can be found in the appendix. ## background below we explain hpsg lexical types ( §2.1), which serve as the tags that we predict, and in §2.2, we give the background on the english treebanks which served as our training and evaluation data. ## methodology supertagging @xcite reduces the parser search space by discarding the less likely interpretations of an orthography. for example, the word bark in english can be a verb or a noun, and in the dog barks it is a lot less likely to be a noun than a verb (see also figure 1 ). in principle, there are at least two possible interpretations of the sentence the dog barks, as can be seen in figure 2 . with supertagging, the pragmatically unlikely second interpretation would be discarded by discarding the noun lexical type (mass-count noun in figure 1 ) possibility for the word barks. in hpsg, there are fine-grained lexical types within the pos class (e.g. subtypes of common nouns or wh-words), so the search space can be reduced further. in precision grammars, supertagging comes at a cost to coverage and accuracy; selecting a wrong lexical type even for one word means the entire sentence will likely not be parsed correctly. thus the accuracy of the tagger is crucial. related to this is the matter of how many possibilities to consider for supertags: the more are considered, the slower the parsing, but the higher the accuracy. in this paper, we experiment with a single, highest-scored tag for each token. however, we combine this strategy (which prioritizes parsing speed) with a list of tokens exempt from supertagging (which increases accuracy). ## conclusion and future work we used the advancements in hpsg treebanking to train more accurate supertaggers. the erg is a major project in syntactic theory and an important resource for creating high quality semantic treebanks. it has the potential to contribute to nlp tasks that require high precision and/or interpretability including probing of the llms, and thus making hpsg parsing faster is strategic for nlp. we tested the new supertagging models with the state-of-theart hpsg parser and saw improvements in parsing speed as well as accuracy. we consider the results on multiple domains, well beyond the wsj section 23. we show promising results but also confirm that domain remains important, and purely statistical systems are brittle and often require rule-based additions in real-life scenarios. we contribute the erg datasets converted to huggingface transformers format intended for token classification, along with the code which can be adapted for other purposes.