Experiments in training transformer sequence-to-sequence DRS parsers (2023)

Abstract: This work experiments with various configurations of transformer-based sequence-to-sequence neural networks in training a Discourse Representation Structure (DRS) parser, and presents the results along with the code to reproduce our experiments for use by the community working on DRS parsing. These are configurations that have not been tested in prior work on this task. The Parallel Meaning Bank (PMB) English data sets are used to train the models. The results are evaluated on the PMB test sets using Counter, the standard Evaluation tool for DRSs. We show that the performance improves upon the previous state of the art by 0.5 (F1 %) for PMB 2.2.0 and 1.02 (F1 %) for PMB 3.0.0 test sets. We also present results on PMB 4.0.0, which has not been evaluated using Counter in previous research.

## introduction discourse representation structures (drss) are a way of representing meaning based on discourse representation theory @xcite . in addition to predicateargument structures, drss express temporal relations, anaphora, modals, negation, and presuppositions, and can be further employed by other automatic processes to understand natural language. the task of mapping sentences to their drs meaning representations is called drs parsing. there now exists a large dataset with drss for corpus examples, the groningen parallel meaning bank (pmb, @xcite , which makes it possible to train deep neural networks of the kinds that provide state-of-the-art performance on a variety of nlp tasks these days. recent work has explored a variety of neural network architectures for this task, but curiously, little work has been done using the otherwise widely utilized transformer-based encoder-decoder archi-tecture. in this paper, we report on such experiments using wordpiece @xcite to tokenize the input and output, and train a sequence-tosequence model where the encoder is a pre-trained bert model @xcite and the decoder consists of randomly initialized transformer layers with cross attention. we experiment with different hyperparameter settings and achieve higher performance than in previous work. in the remainder of this paper, we briefly introduce drss and the pmb dataset in section 2. we then survey previous work on drs parsing in section 3. section 4 provides the machine learning configurations we used. section 5 presents the results and a comparison with prior work. section 6 remarks on our overall takeaways from this work. ## data historically, drss are represented in a box notation designed for human readability. the left-hand side of figure 1 shows the representation of dvořák was not aware of it. the negated content was not aware of it is represented as a separate embedded box labeled b6. moreover, the sentence contains three presuppositions that must be resolved: these are the boxes b2, b4, b7 (shown inside a presupposition operator ∂), corresponding to the referents t1 (time at which the sentence holds), x5 (the referent of the proper name dvořák), and x6 (the entity to which it refers). the latter two referents appear inside the negated box, because they are syntactically in the scope of negation, but they must in fact be interpreted in a wider context (i.e. the text entails that there exists a reference for it and a time at which the state of dvořák not being aware of it held). for more details about drss, we refer to @xcite . the release of the pmb offered for the first time relatively large amounts of text annotated with deep there are several releases of the pmb, differing in size and also in some choices of representation. previous work has focused on version 2.2.0, which contains 5929 drss for english sentences, and version 3.0.0, which has 8403 english drss. the latest release, version 4.0.0, has 10715 english drss. all versions also have data in dutch, german, and italian, which we ignore here. each release has various data files available at the website (parallel meaning bank, 2020), but also provides a separate download that contains only the data relevant for experiments in semantic parsing ("exp data"). the annotations are done automatically and then manually corrected. the representations are labeled with bronze, silver, or gold status. bronze sentences have no manual correction, silver sentences have a partial manual correction and gold sentences have a full manual correction. the dev, test, and eval datasets consist of gold sentences only. ## related work before the advancement of machine learning systems, rule-based approaches were proposed as a solution for the drs parsing task. work within this research track mainly tried to resolve anaphora @xcite , scope ambiguities, and presuppositions @xcite ) on short english text. later, the boxer software @xcite used syntactic parses from a combinatory categorial grammar @xcite to produce drss. in another line of work, drss were represented as graphs obtained from dependency structures of sentences @xcite and ranked according to their probabilities of representing the sentence where the probabilities are obtained from a corpus by computing word-to-word alignments using an external tool @xcite . with the advent of language models and a data set like the pmb which is large enough for finetuning such models, it became possible to employ neural nets for drs parsing. all systems in the recent shared task on drs parsing @xcite used neural architectures, as shown in most systems used a variety of a sequence-to-sequence lstm @xcite , though @xcite used a transformer model @xcite . the system input was either character-level representations or word embeddings obtained from one of the widely utilized bert language models. later, van noord et al. (2020) combined these two inputs to their lstm sequence-to-sequence system, which also used an attention mechanism @xcite , arguing that this improved results even when added to the rich bert embeddings. they also report results using a transformer model but were unable to beat the lstm sequence-to-sequence model in this way. their work reported state-of-the-art results for pmb 2.2.0 and pmb 3.0.0 english datasets. later, @xcite used bert word embeddings and position embeddings as input and expression of drss as trees as output to train a transformer sequence-to-sequence model. they reported a slight improvement (0.4%) upon the state of the art for pmb 2.2.0 dataset. as far as we know these are the only attempts at using the transformers architecture which is the default approach across many nlp tasks today. ## machine learning configurations we use sequence-to-sequence modeling with two main components: an encoder and a decoder. huggingface transformers library @xcite provides the class encoderdecodermodel to configure such models. the models are trained with various configurations of this class to test the performance. 2 for the encoder side, 7 configurations are tested. the first two options test different sizes of random initialization (no-pt). one configuration is 6 layers of 768 hidden layer size (no-pt, 6x768), and the other is 8 layers of 512 hidden layer size (no-pt, 8x512). the rest of the encoders are pre-trained models: bert base cased, bert base uncased, bert large cased, and bert large uncased. for the decoder side, we use the size of 6x768 with the 6x768 sized no-pt encoder, and 8x512 with both a no-pt encoder setup and the pre-trained encoders. all decoder side weights are randomly initialized. the 12x768 networks have 12 and 8x512 networks have 8 attention heads per layer. for the 6-layer setups, two configurations are used: 6 and 12 attention heads per layer. all decoders include cross2 the replication code is published under github: https: //github.com/textlab/seq2seqdrsparser attention layers as it is effective in sequence-tosequence training @xcite . the work by @xcite reports that updating pre-trained encoder weights always resulted in poor performance. therefore, a similar approach is followed and the encoder side weights are frozen whenever we use the pre-trained encoders. when the 12x768 decoder is used with no-pt encoders, the number of parameters to be trained gets too high and a model cannot be trained. thus, the 12x768 decoder configuration is only used together with the frozen pre-trained encoders. our configurations get inputs as sub-word tokens derived from the widely utilized wordpiece tokenizer @xcite . with the pre-trained encoders, the tokenizer used to train that pre-trained model is used as the input tokenizer. for no-pt encoders and for the decoder side output, we train custom wordpiece tokenizers for each dataset. since the output of drs parsing is a drs, the serializations of drss are tokenized using the relative clause notation introduced in van noord (2021). all custom tokenizers are trained with: a vocabulary size of 25000, the minimum frequency for consideration of a token is set to 3, and the maximum tokenization length (maximum number of tokens for one sentence) is set to 512 tokens. to test the effect of using different parameters introduced in this section, the other hyperparameters are fixed such as the optimizer, learning rate, and loss function. we use the adam optimizer @xcite with a learning rate of 0.0001, and use the negative log-likelihood loss @xcite to compute the loss in each batch between the model output and the expected output. we set the batch size to 16 sentences as this is the amount the graphic cards could handle. for any other parameter, the default value defined by version 4.17.0 of the transformers library is used for the objects of types bertconfig, encoderdecodermodel, en-coderdecoderconfig, and bertmodel. we use four nvidia v100 32gb gpus to train the models. the training time depends on the number of parameters and the number of attention heads. for one configuration, training for pmb 2.2.0 sets takes around one day, and training for pmb 3.0.0 and pmb 4.0.0 sets takes around 2 days. when four gpus are used, it takes around one week to train all models in all configurations. we train for each configuration only once. ## results the performance scores are computed for dev, test, and eval 3 sets for each dataset. to compute the scores, we used the counter tool provided by van noord (2022). to make the results comparable with the previous work, the version of counter with the same version tag for each release of the datasets is used. for the 4.0.0 release of the datasets, we use the latest version of the code as 4.0.0 is the newest release. the models are trained for at least 80 epochs for all datasets and stopped if there is no increase in performance for the last five epochs. ## conclusions our work presents the effect of using various sizes of transformer-based encoders and decoders in sequence-to-sequence neural networks with the our results provide a new state-of-the-art of what can be achieved in a vanilla setup of transformer networks with raw text input and clause format drs output. while it is likely that the results can be improved with better language models, or by fine-tuning strategies similar to those of @xcite (prioritizing gold data over silver and bronze), we think more substantial improvements can come from working on the input and output representations. on the output side, we plan to experiment with other ways of expressing drss such as the format introduced by @xcite . on the input side, we believe that syntactic dependency parses contain much information that is useful to drs parsing, such as predicate argument structures. we are currently experimenting with rule-based extraction of relevant information from ud trees and ways of adding this information to the input.