Comparing PTB and UD information for PDTB discourseconnective identification (2020)

Abstract: Our work on the automatic detection of English discourse connectives in the Penn Discourse Treebank (PDTB) shows that syntactic information from the Universal Dependencies (UD) framework is a viable alternative to that from the Penn Treebank (PTB) framework. In fact, we found minor increases when comparing between the use of gold standard PTB part-of-speech (POS) tag information and automatically parsed UD information. The former has traditionally been used for the task but there are now much more UD corpora and in many more languages than that available in the PTB framework. As such, this finding is promising for areas in discourse parsing such as in multilingual as well as under production settings, where gold standard PTB information may be scarce.

## introduction discourse analysis is about identifying the semantico-pragmatic links between parts of a document in order to reveal a structure that organizes a given document. this enables inferences to be made about the content of the document. within a document, each unit (termed a 'discourse unit') is a span of text ; and its meaning depends on the meaning of its surrounding units, as well as the relation that holds between them. the presence of different types of relations are frequently marked by a specific set of wordforms (termed as 'discourse connectives'). for instance, 'because' is one of the markers for an expansion-reason relation, where one discourse unit serves to explain the cause for the other unit it is linked to. in natural language processing, discourse parsing corresponds to several tasks, the very first one being the identification of such discourse connectives. therefore errors in it will cascade to later tasks and impact the overall performance of a discourse parser. identifying discourse connectives in a text is more complex than a simple search and find. this is because a connective may be lexicalized by different wordforms 1 , as well as could have a nondiscourse reading. take the following sentences with the same wordform 'when' in each of them : in sentence (1-a), 'when' is a conjunction between two verb phrases (vps) and serves as a discourse connective ; it marks a relation of temporal succession -from the second vp to the first vp. however, in sentence (1-b), 'when' is not serving as a discourse connective. syntactic information can be useful to distinguish between such instances of discourse and non-discourse usage ; although 'when' has a 'whadvp' syntactic category in both sentences, it is only in the first example that it links two vps. our work focuses on detecting discourse connectives automatically, since improving and avoiding errors at this step is crucial for a parser's performance ; we leave for future work the study of the other parts of the pipeline. this task is generally done on the penn discourse treebank (pdtb) @xcite , the largest corpus annotated for discourse relations in english, and has typically been solved by training a classifier using lexical, morpho-syntactic as well as syntactic information @xcite . in particular, part-of-speech (pos) tags and syntactic trees from the penn treebank (ptb) @xcite have been used. however, the recent release of the universal dependencies (ud, @xcite ) framework is seeing much more corpora and pos taggers made available for ud than there is available for the ptb ; they are also available in comparably many more languages in ud now. it is thus crucial to understand whether the information captured within the ud framework is sufficient for the task of automatically detecting connectives, as it would enable the development of discourse parsing systems for new languages, especially those that are not currently served by ptb-styled corpora and tools. in our work, we seek to establish the effects of using the pos tagset from ud for the task, instead of those from the ptb. there are however, important differences between the ud and ptb frameworks. firstly, the ud pos tagset is coarser-grained compared to the one in the ptb -the ptb has 48 syntactic categories, compared to the 17 categories in ud -and could miss important distinctions necessary for the task. our first results, focused on english, suggest that coarser-grained syntactic annotation is sufficient, and can in fact lead to performance improvements on the task. future work could include demonstrating the same on other languages such as french and chinese, for which moderate-sized pdtb-style corpora have been annotated @xcite . ## related work the pdtb consists of one million words contained in 40,600 articles obtained from the wall street journal (wsj) @xcite . the annotation approach in the pdtb focuses on identifying the local elements making up a coherent text. it identifies relations between two adjacent discourse units, which are linked by a relation that may be marked by a discourse connective. approaches based on the pdtb and similar corpora, which focus on identifying local units of coherence in a text are referred to as shallow discourse parsing (sdp). this is in contrast with 'deep' approaches, using corpora such as the rhetorical structure theory (rst) discourse treebank @xcite , which seeks to identify relations between discourse units extending across an entire document as well as hierarchically between groups of discourse units in the form of structured trees. a hundred connective types and their lexicalization variants are annotated throughout the pdtb. these connective types fall in three syntactic classes, namely : subordinating and coordinating conjunctions such as because and when ; as well as and, as well as or respectively, and discourse adverbials such as for example and instead. on top of these, the spans of each connective's arguments, as well as the relation between them are also annotated. the relations marked by connectives fall within four broad classes (termed as 'senses') -temporal, contingency, expansion, and comparison, which are further categorized into finer types and sub-types @xcite . a pdtb parser typically addresses the identification of connectives first and it is only after this that the classification of the connectives' relations, and the spans of text they cover, are sequentially handled. such modular approaches broadly adhere to the instructions in the annotation manual foot_0 used in the annotation process for the pdtb version 2.0 @xcite . importantly, this is possible due to the local coherence approach taken by the pdtb, which limits but does not preclude the direct application of an automatic connective identification module on parsers for other corpora that focus on more global levels of coherence foot_1 . the task of connective identification attracted attention when the pdtb was first released. pitler & @xcite proposed the use of simple lexical and morpho-syntactic 4 features in a binary classifier. they showed that simply using the connective string already leads to high performance (85.86% in accuracy), meaning that many forms are very often in discourse use. adding morphosyntactic information leads to a very high accuracy of 96.26%. this led to assumptions that the task was solved. however, later work, especially @xcite and @xcite , demonstrate that the task was in fact easy mainly for (1) a few connectives that occur highly frequent and (2) within a fully gold setting ; when considering a realistic setting with predicted (ptb) syntactic information, a performance drop of 4.5 percentage points in macro f1 was observed. they also observed that connective classifiers typically struggle to predict accurately connective strings that occur less frequently ; for instance, lin et al. (2014)'s classifier could only reach a 43.2% f1 score for the connective 'ultimately', which is in the fiftieth percentile amongst the 100 connectives in terms of frequency @xcite . additionally, neural network approaches, leveraging word vectors, have also taken hold in recent years @xcite . with regards the latter, efforts have also been made to produce an end-to-end approach to parsing the pdtb dataset (covering connective identification as well as 'downstream' discourse parsing tasks such as argument identification and sense labeling) @xcite . regarding studies of the impact of syntactic information in discourse parsing, @xcite analyzed the role of syntax in the discourse parsing tasks. they found that using ud syntactic information led to a loss in performance compared to when gold standard ptb syntactic information is used ; albeit this was for a different task of sentence boundary identification on a different discourse corpora, the rst discourse treebank (rst-dt). they observed that ptb pos tags including 'wdt' (wh-determiner) and 'wp$' (possessive wh-pronoun) collapse into one single pos tag, 'det' (determiner) in ud, and the decreased granularity led to an ambiguous signal that is is a source of increased error when using ud information for their task. finally, recent work on learning representations for sentences have found relevance in using explicit discourse connectives identification as a task to guide the learning of such representations. nie et al. (2019) and sileo et al. (2019) trained sentence encoders using large corpora of sentence pairs with discourse markers between them and achieved state-of-the-art results on their approaches. the former worked with a corpora containing 15 of the most frequently occurring discourse connectives, whereas the latter relied on a heuristic to identify discourse connectives candidates, of which some of them have not been annotated in any dataset such as the pdtb. ## approach pitler & nenkova (2009) approached the automatic connective identification task by training a binary classifier using a set of features that includes, and combines, lexical and syntactic information of the connective candidate. they observed that discourse connectives occur in "specific syntactical contexts" ; many connectives take a subordinate clause as one of its arguments (for example, in the sentence "after i went to the store, i went home") and the ptb pos tag 'sbar' marks such subordinate clauses. it was observed that such syntactic information are indicators of a connective candidate being in discourse usage. however, a single pos tag alone may not be sufficient to disambiguate between whether a connective candidate is in discourse or non-discourse usage 6 , and pitler & @xcite found that extending the set of features to include syntactic information from a wider context around a connective candidate improves the performance of a classifier. their work have become seminal for the task and is cited by subsequent researchers working on connective identification. to study whether ud information is sufficient for the task, we adopt their general approach as the basis for our experiments. we also include the work of lin et al. (2014) whose added syntactic features (see section 2) meaningfully improved on the performance of pitler & @xcite , as well as @xcite , who reported the highest f1 score on the task with the pdtb test set during the conll 2016 shared task @xcite . in practical terms, we reproduced the pre-processing and feature engineering pipelines of these three authors as well as obtained approximately gold-standard ud information for the pdtb, which is not available. this allowed us to (1) validate our reconstruction of their connective identification pipelines, (2) isolate the impact of differences in our classifier and hyperparameter settings with these authors', and (3) have a broad-based set of experimental set-ups to study the impact of using ud 6. for example, the two words 'instead' as well as 'and' are discourse connective candidates in the sentence "nasa won't attempt a rescue ; instead, it will try to predict whether any of the rubble will smash to the ground and where." @xcite , though only the former is being used as a discourse connective. this is despite the syntactic category (the pos tag immediately encapsulating a connective candidate) for 'and' being 'sbar' too ; and is cited by them as demonstrating that syntactic information from a wider context is necessary for connective disambiguation. ## settings in this section we outline the settings of our experiments. our experiments were conducted with version 2.0 of the pdtb, which were made available by organisers of the conll 2015 shared task 15 . we kept to the train-development-test split that were prescribed by the pdtb creators 16 . @xcite and @xcite used implementations of maxent classifiers in two nlp machine learning packages 17 . both of these are java-based machine learning packages with specific requirements on the format of the training data. we chose instead to implement the feature extraction and classification pipeline in python, using a popular machine learning package scikit-learn @xcite . we used the latter's logisticregressioncv classifier under a multinomial setting which makes it equivalent to a maxent classifier 18 . to allow the results between each set of experiments to be as directly comparable as possible, we did not implement any hyperparameter optimization procedures.