Character-level Representations Improve DRS -based Semantic Parsing Even in the Age of BERT (2020)

Abstract: We combine character-level and contextual language model representations to improve performance on Discourse Representation Structure parsing. Character representations can easily be added in a sequence-to-sequence model in either one encoder or as a fully separate encoder, with improvements that are robust to different language models, languages and data sets. For English, these improvements are larger than adding individual sources of linguistic information or adding non-contextual embeddings. A new method of analysis based on semantic tags demonstrates that the character-level representations improve performance across a subset of selected semantic phenomena.

## introduction character-level models have obtained impressive performance on a number of nlp tasks, ranging from the classic pos-tagging @xcite to complex tasks such as discourse representation structure (drs) parsing @xcite . however, this was before the large pretrained language models @xcite took over the field, with the consequence that for most nlp tasks, state-ofthe-art performance is now obtained by fine-tuning on one of these models (e.g., @xcite . does this mean that, despite a long tradition of being used in language-related tasks (see section 2.1), character-level representations are no longer useful? we try to answer this question by looking at semantic parsing, specifically drs parsing @xcite . we aim to answer the following research questions: why semantic parsing? semantic parsing is the task of automatically mapping natural language utterances to interpretable meaning representations. the produced meaning representations can then potentially be used to improve downstream nlp applications (e.g., @xcite @xcite , though the introduction of large pretrained language models has shown that explicit formal meaning representations might not be a necessary component to achieve high accuracy. however, it is now known that these models lack reasoning capabilities, often simply exploiting statistical artifacts in the data sets, instead of actually understanding language @xcite . moreover, @xcite found that the popular bert model @xcite completely failed to acquire a general understanding of negation. related, @xcite contend that meaning cannot be learned from form alone, and argue for approaches that focus on grounding the language (communication) in the real world. we believe formal meaning representations therefore have an important role to play in future semantic applications, as semantic parsers produce an explicit model of a real-world interpretation. ## results lms vs char-level models drs parsing is no exception to the general trend in nlp: it is indeed the case that the pretrained language models outperform the char-only model ( ## conclusion we performed a range of experiments on discourse representation structure parsing using neural sequence-to-sequence models, in which we vary the neural representation of the input documents. we show that, not surprisingly, using pretrained contextual language models is better than simply using characters as input (rq1). however, characters can still be used to improve performance, in both a single encoder and two encoders (rq2). the improvements are larger than using individual sources of linguistic information, and performance still improves in combination with these sources (rq3). the improvements are also robust across different languages models, languages and data sets (rq4) and improve performance across a range of semantic phenomena (rq5). these methods should be applicable to other semantic parsing and perhaps other natural language analysis tasks.