Modal Sense Classification At Large: Paraphrase-Driven Sense Projection, Semantically Enriched Classification Models and Cross-Genre Evaluations (2016)

Abstract: Modal verbs have different interpretations depending on their context. Their sense categories – epistemic, deontic and dynamic – provide important dimensions of meaning for the interpretation of discourse. Previous work on modal sense classification achieved relatively high performance using shallow lexical and syntactic features drawn from small-size annotated corpora. Due to the restricted empirical basis, it is difficult to assess the particular difficulties of modal sense classification and the generalization capacity of the proposed models. In this work we create large-scale, high-quality annotated corpora for modal sense classification using an automatic paraphrase-driven projection approach. Using the acquired corpora, we investigate the modal sense classification task from different perspectives.

## introduction factuality recognition (saurí, 2008, de marneffe et al., 2011, 2012, saurí and pustejovsky, 2012) is an important subtask in information extraction. beyond the bare filtering aspects of veridicality recognition, classification of modal senses plays an important role in text understanding, plan recognition, and the emerging field of argumentation mining. communication often revolves about hypothetical, planned, apprehended or desired states of affairs. such "extra-propositional meanings" (morante and sporleder, 2012), or intensional contexts are often linguistically marked using modal verbs, adverbs, or attitude verbs, foot_0 as in (1) for hypothetical situations, or (2) for expression of desires or requests. in the present work, we focus on modal verbs and their epistemic and non-epistemic meaning distinctions. following kratzer (1991)'s seminal work in formal semantics, recent computational approaches such as ruppenhofer and rehbein (2012) distinguish different modal 'senses', most prominently, epistemic (3.a), deontic/bouletic (3.b) and circumstantial/dynamic (3.c) modality. modal sense tagging is typically framed as a supervised classification task, as in ruppenhofer and rehbein (2012). they manually annotated the modal verbs must, may, can, could, shall and should in the mpqa corpus of wiebe et al. (2005). the obtained data set comprises 1330 instances. individual lexical classifiers trained on this data set yield accuracies ranging from 68.7 to 93.5. while these accuracies seem high, there is a strong distributional bias in the training data. due to the small data set size and its skewed distribution of senses, classifiers seem to overfit and hardly beat the majority baseline. it is thus an open question whether the obtained models have sufficient generalization capacity when applied to novel data with different sense distributions. in this paper we reexamine the prior work on modal sense classification and show that indeed specific sense distinctions are difficult for state-of-the art models. this effect was obscured in previous work by the skewed distribution of modal senses in the small data sets. in our work we aim to provide improved models for modal sense classification (i) that are based on larger and re-proportioned training data sets, (ii) that are linguistically insightful and (iii) that are generalizable across genres and robust against variations in sense distribution. the main contributions and insights of our work are as follows: i. paraphrase-driven modal sense projection. we address the manual annotation bottleneck for modal sense labeling by devising a paraphrase-driven cross-lingual modal sense projection approach. we demonstrate that using this method we are able to create large volumes of sense-annotated data of very high quality with minimal effort. ii. semantically enriched models for modal sense classification. using larger volumes of heuristically sense-annotated data, we demonstrate that specific modal sense distinctions are difficult to discriminate for state-of-the-art models. examples of difficult sense distinctions are dynamic vs. deontic readings of can (4.a), epistemic vs. dynamic readings of could (4.b) or epistemic vs. deontic readings of should (4.c). august 2016 (4) a. you can do this, if you want. ability (dynamic) vs. permission (deontic) b. he could have arrived in time. possibility (epistemic) vs. ability (dynamic) c. he should be aware of the issue. possibility (epistemic) vs. obligation (deontic) we investigate the effect of a semantically enriched feature space for modal sense classification and show that specific sense distinctions greatly benefit from semantic information relating to lexical, proposition-level and discourse-level aspects of meaning. overview. the structure of this article is as follows. we review related work in section 2. in section 3 we formulate our research questions. section 4 outlines our paraphrase-driven approach for modal sense projection using parallel corpora and evaluates the quality of the induced annotations, after which, in section 5, we present and analyze the data sets we are using as a basis for our empirical analyses: (i.) the manually annotated mpqa corpus used in ruppenhofer and rehbein (2012)'s work, (ii.) a large, automatically sense-tagged corpus we obtain from cross-lingual modal sense projection, and finally, (iii.) a manually labeled subcorpus of masc, subdivided by different genres. section 6 motivates and defines a semantic feature set for modal sense classification. section 7 performs systematic classification experiments on the basis of the mpqa and the automatically constructed data set. we investigate various classification models, using different training data sets and feature inventories, evaluating in particular the impact of semantic features on difficult sense distinctions in balanced and unbalanced training settings. section 8 analyzes the occurrence of modal verbs and senses in different genres, and investigates the performance and robustness of different classifier models across genres. in section 9 we summarize and discuss our findings and conclude with an outlook on future work. ## research questions and overview our leading hypothesis is that modal sense classification can be improved by semantically enriched feature sets. to explore this hypothesis, we aim to find automated methods for extending the size of the currently available data sets, to reduce sparsity and distributional bias. an open question is also whether modal sense classification models are able to generalize across different genres. we explore these questions in the following order: q1: how can we resolve the manual annotation bottleneck for modal sense classification, in view of acquiring large-size training corpora from different textual genres? q2: can a semantically grounded feature space for modal sense classification improve on prior work in overall performance and robustness against variations in sense distribution? q3: are there genre differences in the distribution of modal senses, and to what extent are they mirrored in the performance of different classification models? in order to investigate modal sense classification from these multiple perspectives, we make use of existing annotated corpora and create novel annotated data sets. all data sets are described in section 5. our models will be trained and evaluated on the primary existing data set annotated for modality, the mpqa corpus (cf. subsection 5.1). this benchmark corpus allows us to explore the impact of semantic features on modal sense classification in a standard setting and facilitates direct comparison to previous work. the existing mpqa corpus is small and shows a biased distribution of senses, so we need more annotated data to determine how well the alternative classification models scale. in order to create large training data sets without extensive manual annotation effort, we create an automatically sensetagged data set, epos (cf. subsection 5.2). this dataset is created via cross-lingual projection (described in section 4) using unambiguous indicators of modal sense in a source language that is projected to our target language, english. the core modeling choice we address is the use of semantic features, both on their own and in combination with more standard features (cf. section 6). with the new expanded datasets, we are also able to investigate to what extent different classification models can profit from being trained on balanced as opposed to unbalanced data. the results of these classification experiments are described in section 7. finally, we investigate the applicability of the classification models built on mpqa and epos to corpus data stemming from various genres of text. to this end, we manually annotate portions of the english multi-genre corpus, masc (cf. subsection 5.3) and evaluate our models on this new, multi-genre dataset. we analyze to what extent distributions of modal verbs and modal senses vary across genres and how these differences affect different classification models. these investigations are described in section 8. ## paraphrase-driven modal sense projection to address our first research question q1, we propose a method for cross-lingual sense projection that will allow us to automatically generate high-quality sense-labeled data for different textual genres in large quantities, and thus to alleviate the manual annotation bottleneck. our approach exploits the paraphrasing behaviour of modal senses, which holds across modal verbs, modal adverbs and certain attitude verbs. as illustrated in (5) and ( 6 ), this paraphrasing behaviour can also be found across languages. maybe is he already at home. ## experimental data sets in this section we describe in more detail the various data sets we use for experimentation. as motivated in section 4.1, we adopt r&r's annotation scheme, with minimal adaptations. @xcite for annotation, the guidelines remained stable across annotation tasks, with only small refinements made for later annotation efforts on masc. ## semantic features for modal sense classification taking up our research question q2 from section 5, we aim to develop a semantically grounded feature space for modal sense classification. our aim is to develop a classifier with high performance that is robust against variations in sense distribution. our hypothesis is that a semantically grounded feature space is able to generalize from surface or domain-specific lexical properties found with different textual genres. we thus expand the feature inventory used in prior work to incorporate semantic factors at various levels. in the following we motivate and describe our semantic feature set, which we organize in seven groups. an overview of the proposed features is given in august 2016 building on the dependency parser foot_15 output, we access the lemma of the embedded verb and its part-of-speech tag in the sentence. we also extract whether the verb has a particle (e.g. the plane could take off ), and if yes, which. in most cases, the relation between a modal verb and the embedded verb is given through the auxiliary (aux) dependency. some other constructions that we capture through dependencies are the copula (cop) in examples like (11.a) and an open clausal complement (xcomp) in (11.b). analysing misparses we noticed that some of them pass through an adverbial modifier clause (advcl) dependency. in case the embedded verb cannot be found, we return the value none. the relation between the embedded verb and its particle is given by the phrasal verb particle dependency (compound:prt). we extract person and number of the subject and the noun_type (common, proper, pronoun). person is identified via personal pronoun features; the remaining features are extracted from pos tags. the countability of the subject head is obtained from the celex database (baayen et al., 1996). we make use of the following columns from the database: lemma (word), countability (c_n) and uncountability (unc_n) of a noun. @xcite since nouns can appear more than once with the same or different column values, we convert the output, so that every noun appears only once with a single countability value from embedded verb vb lemma lemma of head part-of-speech pos of head particle up, off, on,... tva passive true / false progressive true / false perfect true / false voice active / passive neg negation true / false wnv wn sense [0 -2] wn senses (head+hypernyms) wn sensetop top sense in hypernym hierarchy subject noun phrase sbj number sg, pl person 1, 2, 3 countability count, uncount, ambig, none noun type common, proper, pronoun wn sense [0 -2] wn senses (head+hypernyms) wn sensetop top sense in hypernym hierarchy wn lexname active person, artifact, event, ... wn lexname passive person, artifact, event, ... sentence structure s conjunct clause true / false adjunct clause true / false relative clause true / false temporal mod. true / false subject and verb semantic features wn all wn features of sbj group and wnv features august 2016 {countable, uncountable, ambiguous, none}. if the surface form of the subject head can be found in the above described countability database, countability of the surface form is used. otherwise, the subject head is lemmatized and the countability of the lemma is assigned to it. since entries in the celex database are case-sensitive, the subject is lower cased if it is the first word in the sentence. lexical semantic features for the subject head are extracted from wordnet (fellbaum, 1999) using nltk. @xcite following reiter and frank (2010), we take the most frequent sense of the common or proper noun in wordnet (subject_sense0), add the direct hypernym of this sense, the direct hypernym of that hypernym, etc., resulting in features we also extract the topmost hypernym of subject_sense0 in word-net as subject_sense_top (e.g. entity). finally, the name of the lexicographer file in wordnet, containing subject_sense0, is retrieved using two features, lexname_active and lexname_passive. if the subject appears in an active construction, the wordnet lexical filename is assigned to lexname_active and lexname_passive is set to "none", and vice versa for the passive subject. pos is retrieved if there is no lexical filename accessible for the subject in wordnet. using two features for lexical filename, with respect to the subject being in a passive or an active construction, captures the additional distinction of whether the subject is a deep subject or a deep object. this can be beneficial for sense distinctions as in ( 14 ). tva: tense/voice/grammatical aspect features. these features capture voice and grammatical aspect of the embedded verb complex. these characteristics of the embedded verb are important factors for modal sense disambiguation. (15.a) clearly favors an epistemic reading, as the event is in perfective aspect, and thus already completed, whereas deontic sense is favored with future events in indicative mood as in (15.b). we capture both grammatical aspect and voice features using sequences of pos tags of the verbal complex, following loaiciga et al. (2014). the boolean features perfect and progressive indicate the respective grammatical aspect; voice indicates active or passive voice. neg: negation. negation is a semantic feature at the proposition level that can have reflections in modal sense selection. should, e.g., seems to favor a deontic meaning when negated in (16.a). also, negation can interact with disambiguation of epistemic vs. deontic readings depending on specific propositional or discourse context. in (16.b), the favored reading is deontic in the negative sentence. the negation feature captures the presence or absence of negation in the modal construction. we use the dependency label neg to identify negation. along with negation (not), this modifier dependency also captures negation adverbials, such as (never). wnv: lexical semantic features of the embedded verb. this feature group encourages semantic generalization for lexical features of the embedded verb. it can play a role in interaction with other features, such as grammatical aspect, proposition-level features like negation, or the combined lexical semantic features described below (wn). the features in this group are parallel to the wordnet features described for the sbj feature group above (minus lexname_active and lexname_passive), but apply to the embedded verb instead of the subject. if the embedded verb has a particle, then the wordnet sense of the phrase is extracted. s: features of sentence structure. when modals appear as part of a complex sentence, certain structural configurations can reflect rhetorical or temporal relations between the proposition modified by the modal and dependent clauses. an example are telic clauses that can favor specific modal sense readings, as in ( 17 ). we extract features from the constituent tree to capture such effects: whether the modal clause is conjoined to the main clause (embedded conjunctsentence), whether it embeds adjunct clauses (and if so, the conjunction) (adjunctsentence), and whether it is in a relative clause (relativesentence). finally, has_tmod indicates the presence of a temporal modifier. it is easily extracted through the temporal modifier dependency (nmod:tmod). wn: all wordnet features. this feature group aims to capture aspects of proposition-level semantics by combining semantic features of the subject with those of the embedded verb. this feature group simply includes both the wordnet features described in sbj and those in wnv. while this feature is indeed redundant when combined with other feature groups, we include it in the set of features in order to investigate the impact of combined subject-predicate semantic information in the ablation studies we describe later. the intuition is that certain subject-predicate combinations may have a preference for certain modal senses. in (18), for example, can appears with a proposition that is subject to specific prescriptions or "laws": soldiers are subject to restrictions with respect to consuming alcohol. ## classification experiments and results having addressed our research question q1, the automated induction of high-quality modal sense tagged data in section 4, and with the design of a semantically grounded feature space for modal sense classification in place (see section 5), we are now in a position to experimentally investigate our research question q2: can a semantically grounded feature set improve the performance of modal sense classification, and to what extent does a semantic feature space generalize in view of distribution differences? in order to investigate this question, we construct contrasting classifier models with different feature sets and different compositions of training data. our hypothesis is that classifiers trained only on the highly unbalanced mpqa data set will have difficulty separating the effect of distributional bias in the training data from the predictive force of their feature set. a classifier that follows the majority class in the training data will tend to neutralize the potential impact of its feature set. in order to separate the predictive force of different feature sets and the effect of different training data distributions, and also to alleviate sparsity inherent in the data, we evaluate different classifier models obtained from different feature sets and different training data compositions. ## modal sense classification across genres we now address our third research question (q3): are there genre differences in the distribution of modal senses, and to what extent are they mirrored in the performance of classification models? to address this question, we first examine differences in modal sense distributions across genres, and then investigate the performance of different classifier models applied to data from various genres. ## conclusions modality is an important aspect of discourse analysis relating to argumentation, planning, and reasoning in intensional contexts. in this work, we focused on the classification of modal verb senses that carry different types of intensional meaning: epistemic, deontic and dynamic modality. prior work in ruppenhofer and rehbein (2012) established a wellfounded annotation scheme for modal senses, provided manually annotated resources and induced lexical modal sense classifiers. yet, due to the small data set and strong distributional bias in the data, the classifiers could hardly beat the most frequent sense baseline, and it was unclear whether the model generalized to variations in meaning distribution. in our work, we address this issue, and provide answers to three research questions relating to (i.) overcoming the sparsity of annotated resources, (ii.) the design of a semantic classification model that achieves significant performance improvements and proves to be robust against variation in sense distributions, and (iii.) gaining insights into the variability of modal sense distributions across different genres, via annotation of a considerable portion of the masc corpus, followed by investigation of the extent to which distributional differences may affect classifier models in an out-of-domain evaluation experiment. in response to these questions, we have made various contributions and obtained a number of insights. we applied a paraphrase-driven projection method for the automatic induction of sense-labeled training data using parallel corpora. the senses of modal verbs bear an ideal level of granularity for sense projection using a small set of paraphrases. using carefully selected paraphrase seeds, the induced annotations yield high accuracy of 0.92. this method can be applied on a broad scale and for many languages, given that bitexts are available in large quantities. recent work in marasović and frank (2016) applied the same method to automatically acquire a large dataset for modal sense classification for german. we designed a linguistically motivated semantic feature space for modal sense classification that is effective in reducing misclassifications.