Selective Annotation of Modal Readings: Delving into the Difficult Data (2016)

Abstract: Modal auxiliaries have different readings, depending on the context in which they occur (Kratzer, 1981). Several projects have attempted to classify uses of modal auxiliaries in corpora according to their reading using supervised machine learning techniques (e.g., Rubinstein et al., 2013, Ruppenhofer & Rehbein, 2012). In each study, traditional taxonomic labels, such as ‘epistemic’ and ‘deontic’ are used by human annotators to label instances of modal auxiliaries in a corpus. In order to achieve higher agreement among annotators, results in these previous studies are reported after collapsing some of the initial categories. The results show that human annotators have fairly good agreement on some of the categories, such as whether or not a use is epistemic, but poor agreement on others. They also show that annotators agree more on modals such as might than on modals such as could. In this study, we used traditional taxonomic categories on sentences containing modal auxiliary verbs that were randomly extracted from the English Gigaword 4th edition corpus (Parker et al., 2009). The lowest inner-annotator agreement using traditional taxonomic labels occurred with uses of could, with raw agreements of 42%−48% (κ = 0.196−0.259), compared to might, for instance, with raw agreement of 98%. In response to the low numbers, rather than collapsing traditional categories, we tried a new method of classifying uses of could with respect to where the reading situates the eventuality being described relative to the speech time. For example, the sentence ‘Jess could swim.’ is about a swimming eventuality in the past leading up to the time of speech, if it is read as being an ability. The sentence is about a swimming eventuality in the future, if it is read as being a statement about a possibility. The classification labels we propose are crucial in separating uses of could that have actuality inferences (Bhatt, 1999, Hacquard, 2006) from uses that do not. For the temporal location of the event described by a use of could, using four category labels, we achieved 73%−90% raw agreement (κ = 0.614−0.744). Sequence of tense contexts (Abusch, 1997) present a major factor in the difficulty of determining the temporal properties present in uses of could. Among three annotators, we achieved raw agreement scores of 89%−96%(κ =0.779−0.919%) on identification of sequence of tense contexts. We discuss the role of our findings with respect to textual entailment.

## introduction modal auxiliary verbs such as can, could, would, should, might, must, and may have different readings depending on the context in which they occur. sentence (1) below provides an illustration: on one reading of sentence (1), the author is talking about a past ability the president had to (possibly repeatedly) stop members of congress from expressing their opinions. this reading could occur in a disambiguating context such as that in sentence (2) below: the reading of could is expressing an ability of the president in the past. one can infer that it is possible that there were frequent or at least multiple instances of presidents stopping members of congress from expressing their opinions in the 80's. on another reading, the author is surmising that there is a possibility of the president stopping members of congress from expressing their opinions in the future relative to the speech time. a disambiguating embedding is given in sentence (3) below: for the possibility reading, there is no inference invited that there was even one past instance of the president stopping congress members from expressing their opinions. it seems rather more likely that the president has not stopped them up until the time of speech and that he will stop them in the future. readings of modal auxiliary verbs have been of much interest theoretically in linguistics and philosophy due to the nuances of meaning they convey and the difficulties in representing their meaning in formal models. formal semantic models of modal auxiliary meaning distinguish very fine-grained differences among the taxonomic categories. those differences will only be alluded to here as they are not the focus of this paper, but they include topics such as which kinds of possible worlds a modal proposition's truth value is evaluated relative to (e.g., kratzer, 1981,1991; lewis, 1973), what type of ranking orders the accessible worlds given a modal reading (e.g., kratzer, 1981; schulz, 2007; lassiter, 2011), and the perspective from which the modal is evaluated (e.g., egan et al., 2005; macfarlane, 2011; lasersohn, forthcomming). along with the semantic properties of modal auxiliaries, there are interactions among modal auxiliaries, tense, grammatical aspect, lexical aspect, and temporal phrases. these interactions have also been the topic of many studies (e.g., condoravdi, 2002; iatridou, 2000; bhatt, 1999; bhatt & pancheva, 2006; hacquard, 2006). the fact that modal auxiliary verbs have multiple readings poses several problems for applied areas of research, such as textual entailment and text interpretation. when working on an applied task, however, there must be a balance between accurate representation and broad coverage. therefore, not everything about modal auxiliary meaning that is of interest can be represented at once. rather, it is important to focus on the parts of modal auxiliary meaning that most directly impact an automated learner. one major issue in representing modal auxiliary meaning is regarding the status of the event that is being described by the modality. the event, such as the 'stop-members-of-congress-from-expressingtheir-opinions' event referred to in the example sentences above, can be an event that the speaker knows or believes to have happened in the actual world or an event that the speaker predicts in the future or, in the case of counterfactuals, considers not to have happened in the actual world in the past. in automated tasks, such as question and answering systems, it is important to distinguish among modal auxiliary readings. given a text that includes sentence (1), an automated system would need to provide a different answer to questions like 'what did the president do?' or august 2016 'did the president ever stop members of congress from expressing their opinions?' based on the reading of the modal auxiliary verb. when working on problems in natural language processing (nlp) that require considerable semantic knowledge, it is common to use supervised learning methods, either by having human annotators label a sub-set of the data as training data or by carefully choosing seed set data that reflect patterns typical of the desired classes. when using human annotators, it is important to ensure that what the human annotators are labeling with class labels accurately reflects the concept that the automated system is intended to learn. many decisions are involved in word-sense disambiguation tasks. the noun 'bat', for example, involves two distinct classes: the flying mammal called a 'bat' and the instrument for hitting balls called a 'bat'. the words occur in different positions with respect to their semantic roles: the mammal is likely to be the subject or object of a verb, but the instrument is likely to occur in prepositional phrases. the types of words in the text with each use differs. when the text has words like 'baseball' and 'stadium', the instrument is more likely than when the text contains words like 'cave'. the words sense disambiguation problem posed by modal auxiliary verbs, however, poses additional challenges. modal auxiliary verbs are flexible regarding with which verbs they can occur. just as one could go to a baseball game, one could go to a cave. the readings are not limited by their position in the sentence. two distinct readings can occur in identical strings, as sentence (1) above illustrates. this paper is about the method we designed to train annotators to label the modal auxiliary could according to the temporal properties of the sentence in which it occurs. we focused on could for two reasons. the first reason is because results of inter-annotator agreement (iaa) in previous studies show lower agreement on the reading of examples with could than on other modal auxiliaries. the second reason is because could is the modal auxiliary verb which is most commonly associated with readings allowing actuality inferences. section 2 reviews previous studies that led us to isolate uses of could from other modal auxiliaries, providing results of our preliminary study. section 3 presents an overview of the current study. in section 4, our methods are presented. section 5 presents the results of our annotation project. section 6 discusses what the results mean and describes directions for future research. ## previous work on modal auxiliary sense disambiguation the distribution of modal auxiliary verbs in corpora has been studied for various applications including corpus linguistics (coates, 1983), natural language processing (baker et al., 2012, ruppenhofer & rehbein, 2012, rubinstein et al., 2013), and english language education (romer, 2004). based on previous studies, it is clear that, although each modal auxiliary has multiple readings, not all modal auxiliaries present the same degree of challenges in terms of word-sense disambiguation (wsd) tasks. for example, the modal auxiliary might can be read as expressing a possibility, as in sentence (4) below: in some dialects of english, the modal might can also be used to request permission as in sentence (5) below: although the modal auxiliary might has at least two senses, one of its senses is disproportionately more common than the others. in the distributional corpus study by romer (2004), the modal auxiliary might is found to have the possibility sense in 95% of the data observed. similarly, as shown in the chart in 1, the modal auxiliary may has a possibility sense in 83% of the samples she labeled. in contrast to the modal auxiliaries might and may, romer (2004) shows the modal auxiliaries can, could, and would to display a more even distribution of senses. for example, could has an ability reading in 34% of the labeled data and a possibility reading in 41.5% of the data, other significant readings include hypothetical (14.5%) and 6.5% of the cases are classified as unclear. coates (1983) hand-labeled sentences containing modal auxiliary verbs found in both spoken and written corpora. she presents a breakdown of modal auxiliaries by sense, illustrated in the histogram in figure 2 below. modal auxiliaries are positioned relative to each other in the chart given their overall frequency in the text combined with a particular sense. the histogram in figure 2 shows that the most frequent modal auxiliary in the text is a 'prediction' reading of will. in order to see, in the histogram, how one reading of a particular modal auxiliary compares to another, it is necessary to find the different instances of the modal has semantic category labels that reflect traditional labels for modal auxiliary readings. although we expect north american english written newswire to be considerably different than spoken british english, it provides generalizations that were born out in other studies. (figure from romer, 2004). the authors combined could and can into a single category. their annotation of can and could is higher than the baseline of the most frequent reading, which they report as 52.17 (2012, page 1545). but their results show that the iaa still is significantly lower for can and could than for other modal auxiliaries in terms of classification results. item κ %-agreement may, might 195 0.621 0.89 must 183 0.848 0.98 shall, should 182 0.602 0.96 can 598 0.614 0.77 the report in rubinstein et al. (2013) is not broken down by modal auxiliary but rather by reading. all modal auxiliaries are grouped into a single report. therefore, it is not possible to directly compare our results with could to their results without additional information on their classification results, broken down by modal auxiliary. the authors report that they achieved higher results on iaa upon collapsing their semantic categories, but how the collapse alters the results specifically for could is not reported. baker et al. (2012) provide the results for a single annotator who tagged not only modal auxiliaries but also non-modal auxiliary verbs that were determined by the annotator to be in the scope of the modal and negation. the tags are reported to be 92% correct, however, it is not clear if the annotator accepted one or more than one label for each sentence in determining whether or not the label was correct. classification of uses of modal auxiliaries according to semantic taxonomies has been a difficult problem (baker et al., 2012; rupenhofer & rehbein, 2012; rubinstein et al., 2013). high inter-annotator agreement and automated results have been reported by authors who use only the classes of 'epistemic' and 'non-epistemic' (or 'root' ) (hacquard & wellwood (2012) for only must and have to, rubenstein et al. (2013) after collapsing more fine-grained categories). ## overview of the present study in order to improve the classification of uses of could, we propose a coarse grouping of senses by the temporal properties associated with them. we claim that these groupings, in the case of could, are more tractable than fine-grained semantic categories such as 'ability', 'possibility', and 'circumstantial'. in addition to being more tractable, the temporal categories subsume the finer grained semantic categories. some modal auxiliary verbs express past or non-past through the morphology of the modal auxiliary verb and other ones do not express temporal morphology on the modal auxiliary, but appear to express the past through a combination with the perfect form of the main verb. one type of modal reading that has a past and non-past form is the ability use of can and could : one type of modal reading that does not have a past and non-past form morphologically indicated on the modal auxiliary verb is the epistemic use of might: the modal auxiliary might does not change to reflect whether the eventuality is in the past or not. it appears that the past is represented by the use of the perfect form 'have come' in contrast to 'come'. the modal auxiliary verbs that indicate a difference in temporal meaning through the morphology of the modal itself will be called paradigm a modal auxiliaries. those that do not exhibit a morphological change on the modal auxiliary verb and describe past scenarios via the use of perfect on the main verb will be called paradigm b modal auxiliaries. the modal auxiliary could is claimed to have both a paradigm a sense and a paradigm b sense. @xcite paradigm a uses are those in which the present tense of could is can. the use of could is used either to express a past tense use of can (such as an ability or circumstantial use) or to match the grammatical past tense of an embedding verb. the semantic readings that fall under paradigm a are circumstantial, ability, and deontic. we categorized these readings as past a if they were about past eventualities and grammatical past paradigm a, labeled grampast a, if they were present tense paradigm a uses displaying sot effects. the distinction between these two sub-categories depends on whether the reported speech makes more sense with can or could. whenever a use of could occurred and was embedded under a past-tense verb, we tested whether the reported speech made more sense with can or with could, or if the reading made sense with both and was thus ambiguous. if the reading is best with can, then the sentence is treated as an instance of grammatical past. an example would be as below: in sentence (26), it is most likely that john said, 'i can make a cake' because, in the context of promising to bring something to a party, a firm offer is more felicitous (this is a tendency, not absolute rule). in this case, the past tense of can is merely an instance of grammatical paradigm b present past inductive possibility could epistemic could have metaphysical hypothetical could counterfactual could have the semantic category is that of an ability, and it is in the past, so it is a past paradigm a reading. the second broad category, paradigm b, encompasses the uses where past eventualities are referred to by could have plus the perfect form of the main verb (e.g., gone, come, been going). the subcategories of paradigm b include: counterfactual, hypothetical, epistemic, and possibility. the semantic categories covered by paradigm b are shown in the readings involve incomplete information on the part of the person making the statement. possibility is speculation, inductive reasoning about the future based on what knowledge an individual has regarding actual states of affairs. epistemic reasoning is about states of affairs that are either true or not true in the actual word at the time of utterance, but of which the person making the statement does not know the truth status. the metaphysical readings involve using one's imagination to discuss states of affairs that are not necessarily believed to be true and can be known or believed to be false in the actual world. a hypothetical statement reasons about how things could be now or in the future if some premise held (that the speaker knows or expects not to hold). the counterfactual readings are about how things would have been if a premise held (which the speaker knows does not hold). an example of a hypothetical statement is given in sentence ( 28): (28) if i had a car, i could get home easier. a counterfactual expresses a past state of affairs as in sentence (29): in both cases, there is an inference that the speaker does not or did not have a car at the time referenced by the if -clause. the if -clause does not have to be present. it can be contextual, as sentence (30) below, said in a context where the grandmother is dead and people are thinking of her: it should be noted, that because we are using news corpora, the distribution of kinds of readings will be skewed. note the following example in sentence (31) below: sentence ( 31 ) above is ambiguous. one reading is the reading associated with a historical narrative, in which the author is describing events before speech time. on the narrative reading, the referent of 'he' was possibly facing war crime charges in the past of the readers present, for example, at a time that the narrative has not yet covered. historical narrative of this sort is uncommon in news corpora, the source from which these sentences have been pulled. the second reading, is the most likely reading, given the source of the text. this reading is a non-past reading such that, at the time when the sentence was authored, the possibility of the referent of 'he' facing war crime charges was something which loomed in the future. 7 ## methods three annotators participated over the course of the project: the primary researcher and two upper-level undergraduate students in linguistics. our methods for preprocessing data were the same as in the previous study. samples were taken from the afp_eng_199609 file of the english gigaword 4 th edition corpus (parker et al., 2009). documents in the corpus were stripped of all html code as well as their document identification numbers. the extracted sentences were randomized using a randomization script in python. preprocessed sentences with could extracted from the corpus were put into spreadsheets with drop-down choices among labels. the dropdown choices were new in this round of annotation and helped avoid typographical errors, as well as simplifying the annotation process. an example of the format that annotators saw is shown in figure 3 . annotators were given spreadsheets with 25-50 sentences. a total of 600 distinct sentences were annotated and adjudicated for a gold standard annotated corpus with our labeling. the third column was only present in this round to facilitate annotators with checking their grammatical and temporal interpretation against possible readings. although annotators seemed to have stronger intuitions about semantic categories like 'epistemic' or 'deontic', there were very few examples that annotators both found intuitive and assigned the same label. with each traditional taxonomic label, the temporal label to which it belonged was listed, as shown in the display pull-down menu in figure 3 . when annotators viewed the pull-down menu, for instance, to see 'ability' they would see the choice 'ability (past_a or gram_past)'. this added an additional check-point to annotation accuracy. if an annotator thought a reading was an ability reading, but had chosen the 'non_past_b' label in column two, then she was aware that she had mislabeled one column. annotators read each sentence and were encouraged to fill out whichever of the three columns was most immediately obvious to them and then proceed to the other columns and check that all were consistent with each other, consistent with the sentence, and consistent with the decision-tree algorithm explained in the next subsection. ## conclusions our work demonstrated a novel way to classify uses of could. we showed that could presents a number of difficulties due to its high number of readings, lack of one single highly dominant sense, and susceptibility to sot effects. sub-groupings of modal auxiliaries that are based on temporal properties are at the heart of the most difficult data in current classifiers. the sub-groupings divide up data in a way that may prove more informative for automated learning. our temporal semantic categories constitute an instance of reducing labels, but in an intrinsically useful way. in the work reported in this paper, we demonstrated that grammatical features help to determine the temporal placement of a modal auxiliary and reduce the number of possible readings that a modal auxiliary has. this work suggests that using a semantically informed divide-andconquer approach can increase the success of classifying modal auxil- @xcite for more details on the nature of actuality inferences associated with ability can and could, see the discussion in moon (forthcoming).