The long and the short of it: DRASTIC , a semantically annotated dataset containing sentences of more natural length (2023)

Abstract: This paper presents a new dataset with Discourse Representation Structures (DRSs) annotated over naturally-occurring sentences. Importantly, these sentences are more varied in length and on average longer than those in the existing gold-standard DRS dataset, the Parallel Meaning Bank, and we show that they are therefore much harder for parsers. We argue, though, that this provides a more realistic assessment of the difficulties of DRS parsing.

## motivation corpora with deep, logic-based semantic annotations are quite rare because they are so hard to annotate. the arrival of the groningen meaning bank @xcite and the parallel meaning bank (pmb; @xcite changed this situation by offering full discourse representation structures (drss; @xcite for substantial amounts of text in dutch, english, german, and italian. the current release, 4.0.0, contains more than 10,000 sentences in english and between 1,400 and 2,800 sentences in other languages. however, the dataset contains both bronze (automatic), silver (partial manual disambiguation), and gold (full manual disambiguation) data, and the gold sentences are consistently very short (@xmath0. since the dev, test, and eval sets contain only gold data, this means that drs parsers are tested only on very short sentences, yielding an overly optimistic assessment of results in this area. in this paper, we improve on the situation by offering a gold standard dataset containing drss with a more realistic sentence length distribution. we call this dataset drastic, for 'discourse representation annotation with sentence texts of increased complexity'. 1 an additional strength of drastic is that the texts it contains -three contiguous documents plus a selection of medium-length sentences -are from the gum corpus @xcite , allowing users to explore connections between the drs annotation and the rich annotation available in gum: beside morphosyntactic annotation following the universal dependencies (ud) scheme @xcite , this also includes entity recognition, coreference, discourse structure and more. 2 the current size of our dataset is small, at 157 sentences with full manual disambiguation, but around 1,000 more sentences have received a first manual annotation by student annotators and will subsequently be integrated into the dataset. drs parsing gets harder as sentences grow longer (cf. @xcite . this is natural, but some peculiarities of the pmb annotation are especially hard to capture, and contribute only little extra information. cases in point are recursive presuppositions, strict separation of different presuppositions of a single sentence, and the use of discourse relations with relatively bland content such as continuation. as the sentence grows in length, these result in a complex network of embedded drss. in such cases, parser output that is (more or less) logically equivalent to the gold can still get a low score. to avoid this, we simplify the annotation of such structures (see section 2.2). since our corpus is small and does not include training data, we provide a script that flattens pmb-style annotations to our format. this can be used to flatten pmb data before training a parser, or alternatively to flatten the output of a parser trained on the pmb. the structure of the paper is as follows. in section 2, we introduce discourse representation theory (drt), as well as the pmb annotation and our simplifications of it. in section 3, we describe our corpus, and section 4 studies the effects of sen-tence length on drs parsing and offers baseline modelling results on our data. ## the format: discourse representation structures in discourse representation theory @xcite @xcite the meaning of a sentence is analysed as its contribution to the existing semantic representation of the discourse context, called a discourse representation structure (drs). this means that drt belongs to the family of theories called dynamic semantics, although drt treats only the process of interpretation as dynamic, not the notion of meaning itself. 3 drss are traditionally represented as boxes divided into two: a universe of discourse at the top, containing a number of discourse referents, which can then be referred to by the set of conditions in the lower part of the box. drs conditions are by and large simply formulae of some predicate logic, but can also contain complex conditions relating multiple drss via logical operators like negation, implication, and disjunction, or modal operators like possibility and necessity. by way of illustration, figure 1 gives a drs for the sentence jadzia thought that miles or julian had been hurt. drt is compatible with many different specific theoretical approaches to semantics; in figure 1 , as in our corpus, we use a neo-davidsonian event semantics where events and states (collectively called eventualities) are treated as first-class entities in the ontology, and semantic dependents are related to their heads via thematic role predicates such as agent, patient, etc. (on event semantics see e.g. @xcite . a basic representation of tense is also given, by including the relation time between an eventuality and its time, and relating that time to the constant 'now' (referring to the time of utterance) or to other times. aside from the rich body of theoretical work in drt exploring various knotty semantic phenomena such as anaphora @xcite , tense @xcite , rhetorical structure @xcite , propositional attitudes @xcite , and others, one other good reason for using drss as our semantic representations is the existence of the parallel meaning bank (pmb; @xcite , a multilingual corpus of drs-annotated texts in english, dutch, italian, and german, to which we aim to contribute. ## summary we have presented a new dataset, the drastic corpus, which contains pmb-style drss annotated over sentences with more realistic lengths than the original pmb dataset, and which is, accordingly, much more of a challenge for state-of-the-art parsers. we hope that this will lead both to a more realistic assessment of the difficulty of drs parsing and, in the longer term, to the development of more robust models.