Quirk or Palmer: A Comparative Study of Modal Verb Frameworks with Annotated Datasets (2023)

Abstract: Modal verbs, such as can, may, and must, are commonly used in daily communication to convey the speaker’s perspective related to the likelihood and/or mode of the proposition. They can differ greatly in meaning depending on how they’re used and the context of a sentence (e.g. “They must help each other out.” vs. “They must have helped each other out.”). Despite their practical importance in natural language understanding, linguists have yet to agree on a single, prominent framework for the categorization of modal verb senses. This lack of agreement stems from high degrees of flexibility and polysemy from the modal verbs, making it more difficult for researchers to incorporate insights from this family of words into their work. As a tool to help navigate this issue, this work presents MoVerb, a dataset consisting of 27,240 annotations of modal verb senses over 4,540 utterances containing one or more sentences from social conversations. Each utterance is annotated by three annotators using two different theoretical frameworks (i.e., Quirk and Palmer) of modal verb senses. We observe that both frameworks have similar inter-annotator agreements, despite having a different number of sense labels (eight for Quirk and three for Palmer). With RoBERTa-based classifiers fine-tuned on MoVerb, we achieve F1 scores of 82.2 and 78.3 on Quirk and Palmer, respectively, showing that modal verb sense disambiguation is not a trivial task.

## introduction modal verbs (also referred to as modal operators, modals, or modal auxiliaries @xcite ) convey important semantic information about a situation being described or the speaker's perspective related to the likelihood and/or mode of the proposition @xcite . because of the widespread use of modal verbs in our daily lives, an accurate modeling of modal verb senses from context is essential for semantic understanding. for example, as modal verbs are often used with verbs that express one's personal state or stance, such as admit, imagine, and resist @xcite , we can utilize them for better speaker intention identification or sentiment analysis. in both linguistics and nlp, however, there is no unifying consensus on how to organize these words ( two commonly used frameworks come from @xcite and @xcite . to compare these frameworks, we present a new dataset, moverb, containing 4540 annotated conversational english utterances with their modal verb categories. we chose the conversational domain since spoken, casual text is more flexible and nuanced compared to language from other domains and therefore could reap the most benefits from better modal verb classifications. to the best of our knowledge, this study provides the first empirical comparison of two modal verb frameworks with annotated datasets, evaluating the practicality of these different theoretical frameworks. our study shows a clear inclination towards one of the two frameworks and quantitatively shows how humans struggle with the task. our main contributions are as follows: ## related work there are numerous linguistic studies about modal verbs and their categorization @xcite @xcite @xcite @xcite . however, despite attempts to reconcile them @xcite , widespread variation makes it unclear which framework would work best for specific nlp tasks. a dataset using multiple modal verb frameworks would help researchers experiment, but that dataset is yet to be built. to the best of our knowledge, there is no english dataset dedicated to the comparison of modal verb labeling. framework consistency is not the only thing lacking in modality datasets. sources of modality can vary as well. in a multilingual corpus focusing on modality as a whole, nissim et al. manually tag words and phrases representing modality. due to the lack of emphasis on modal verbs, this dataset contains only 32 instances over 7 modal verbs: will, might, can, may, would, could, and should @xcite . we argue that a dataset focusing on modal verbs is also necessary because of the ample complexities of modal verbs on their own. even datasets that do focus on modal verbs are not guaranteed to study the same set of words @xcite . modal verbs in different domains, namely conversational and academic, have quite dissimilar distributions @xcite . in our crossdomain analysis, we utilized a dataset for subjectivity analysis in opinions and speculations from the news domain @xcite . ruppenhofer and rehbein do not include would and will in their annotations, making their dataset challenging for analyzing conversational english. would and will are 1st and 3rd when we rank modal verbs by their frequencies in spoken english @xcite . we note that there is a slight difference in our annotation frameworks. ruppenhofer and rehbein create a schema of their own, building off of work by @xcite and @xcite . we do ## potential applications with modal verbs there is some debate as to whether we should focus on modality as a whole since it can be expressed in other ways not limited to modal verbs @xcite . however, we argue that modal verbs alone offer enough complexity. there is untapped potential in improving the categorization of modal verbs, which could greatly enhance the performance of various downstream natural language processing (nlp) tasks. difficulty with modal verb understanding can cause confusion in semantic similarity tasks. using a roberta hugging face model @xcite pretrained on the microsoft research paraphrase corpus (mrpc) subset of the general language understanding evaluation (glue) dataset 7 , we saw that the model was not able to reliably identify the unlikely interpretations for given sentences. for example, given the sentence, "my parents said i can go", the model would flag all following three as semantically equivalent by a score of at least 0.73: "my parents said i have the ability to go.', "my parents said i might go.", and "my parents said 7 textattack/roberta-base-mrpc i have permission to go". 8 as another example, we generated paraphrases for the empathetic dialogues dataset @xcite using the t5 parrot paraphraser @xcite in the hugging face library. 9 this revealed that 1951 out of 2490 (78.35%) paraphrases created for 865 sentences foot_8 kept their original modal verbs. this suggests that being able to correctly identify and paraphrase the sense of a modal verb can greatly increase variety in paraphrasing. ## theoretical frameworks we use two labeling frameworks in our dataset annotations that we refer to as quirk's categories and palmer's categories. or event by giving permission, expressing an obligation, or making a promise or threat. an epistemic one is concerned with matters of knowledge or belief and with the possibility of something being true. lastly, dynamic modal verbs are related to the volition or ability of the speaker or subject, in other words, some circumstantial possibility involving an individual (figures 7 and 8 in appendix a.1). @xcite ## moverb: annotated modal verb dataset we use the eight core modal verbs in our study: can, could, may, might, must, will, would, and should. shall is also another core modal verb but is excluded from our work since there are too few instances of it in our conversational dataset. 11 we chose the empathetic dialogues dataset @xcite for our annotation task because of its variety of utterances in the conversational domain and wide usage in social dialogue studies. an utterance is defined as a speaker's output in a single turn and can potentially be one or more sentences. we extracted utterances containing only one modal verb as detected using spacy's pos tagger and lemmatizer @xcite . we focused on utterances containing one modal verb for simplicity, but this excluded very little from the original dataset since only 2.4% of the utterances had more than one modal verb. 12 we included utterances containing more than one sentence (as long as they used only one modal verb) in order to retain as much context as possible. in this way, we separated out the first 4540 utterances containing single modal verbs, except for may and might, which we collected and used all of due to scarcity ( after finalizing which utterances to annotate, we utilized amazon mechanical turk (mturk) to gather crowd-sourced labels for each modal verb. three annotations were collected for each of the 4540 utterances, and we assigned final labels based on majority voting (figures 1b and 1c ). we refer to the annotations for quirk's categories as moverb-quirk and those for palmer's categories as moverb-palmer. our hit (human intelligence tasks) form, containing definitions and examples for the annotators, is included in appendix a.1 (figures 410 ). we limited our mturk pool to master workers (high-performing workers) residing in the us with approval rates @xmath0. each worker was allowed to annotate as many hits as they wanted and were allowed to submit annotations for both frameworks. they were prevented from participating any further if we saw that their annotations for quirk's categories seemed random (appendix a.2). we did not apply the same filter for palmer's categories because of the less stringent restrictions on which modal verbs each category could be attributed to. however, 95% of our annotators had submitted at least one hit for each framework, so we were able to apply our criteria to the vast majority of them. ## classification tasks we answer the following questions using the collected moverb dataset: (1) how well moverb can be used to train transformer models for a modal verb sense prediction task and (2) how transferable that knowledge (trained on the conversational domain) is to other domains, namely the news opinion domain. experiment design in the following experiments, we exclude data where all three annotators disagreed with each other (appendix c). this is to enable our use of pre-trained models and to share available insights. for our first classification experiment, we split our datasets into cross-validation train-test ratios of 90/10. for the second experiment focusing on transferability, we bring in a third dataset, which we refer to as ruppenhofer and rehbein. we use one dataset for training and the other for testing and vice versa. when comparing moverb-palmer and ruppenhofer and rehbein, we only consider the overlapping labels since the majority of ruppenhofer and rehbein's labels come from palmer. we conducted this on the setup where both moverb-palmer was the training set and ruppenhofer and rehbein was the test set and vice versa. additionally, since we initially hypothesized that the lack of will/would examples in the ruppenhofer and rehbein dataset would cause issues, we conducted the same experiment with those modal verbs removed from moverb-palmer ( for all experiments, we ran 10-fold crossvalidations and used an early stopping callback that would get triggered once the f1 value stopped increasing by at least 0.01. for learning rates, we tested among 5e -6, 1e -5, and 2e -5, and used the weighted f1 score for evaluation. we used the pytorch lightning library to train and evaluate a transformer model with an adam epsilon of 1e-8, and a batch size of 32. additionally, our trainer used gpu acceleration with a geforce rtx 3090 using the distributeddataparallel strategy. we fine-tuned six transformer-based models @xcite from huggingface transformers @xcite : albert base @xcite , bert (both base and large) @xcite , roberta (both base and large) @xcite , and distilbert base @xcite . in all runs, the roberta models showed the best test f1 scores (tables 14 and 15 in ap-dataset val. f1 test f1 (base) moverb-quirk 78.98 82.22 (29.9) moverb-quirk (w/o w 2 ) 83.56 84.31 (38.3) moverb-palmer 77.08 78.36 (53.7) moverb-palmer (w/o w 2 ) 80.62 80.89 (52.1) ruppenhofer and rehbein 83.31 85.60 (52.0) ## conclusions and future work modal verb categorization is a difficult task even for humans, making supervised datasets a vital part of computational analyses. in this study we presented moverb, a new modal verb dataset that consists of 4540 conversational utterances with crowd-sourced annotations for the modal verb categories presented by palmer and quirk. we show that within moverb, annotators struggled less with quirk's categories. fewer disagreement relative to the number of labels led to less noise, which translated to better performance on our models, both intra and cross-domain. additionally, moverb-quirk gave us a more precise study of modal verb patterns due to more specific labels. therefore, barring cases where there is a specific reason to dataset utterance prediction label moverb-quirk we do not have a fence but i know my dog will stay in the yard volition (49.36) prediction (3/3) moverb-palmer that stinks! try not to be jealous though. something else will come your way. dynamic (49.87) epistemic (3/3) ruppenhofer and rehbein (r&r) "a government in which the president controls the supreme court, the national assembly and the armed forces can not be called a democracy, " soto charged. deontic (65.7) dynamic (n/a) moverb-@xmath1, and their diet ranges from bagels and cream cheese to rice and beans -all eaten with plastic utensils -after which the prisoners may clean their teeth with specially shortened brushes. this work presents several opportunities for further development. an immediate next step would be to incorporate more modality frameworks into the existing dataset. potential additional work would be to use the dataset for specific nlp tasks, such as paraphrasing and inference. one way in which modal verbs could be used in inference is to focus on permission and obligation to see social power dynamics in a text (who seems to be receiving/giving permission more than average or who seems to be controlled by more social obligations). additionally, one could investigate the annotations with complete disagreements to determine the causes and exhibit high degrees of natural 14 however, this disagreement subset is not used in our experiments. language understanding.