Probing BERT ’s ability to encode sentence modality and modal verb sense across varieties of E nglish (2023)

Abstract: In this research, we investigate whether BERT can differentiate between modal verb senses and sentence modalities and whether it performs equally well on different varieties of English. We fit probing classifiers under two conditions: contextualised embeddings of modal verbs and sentence embeddings. We also investigate BERT’s ability to predict masked modal verbs. Additionally, we classify separately for each modal verb to investigate whether BERT encodes different representations of senses for each individual verb. Lastly, we employ classifiers on data from different varieties of English to determine whether non-American English data is an additional hurdle. Results indicate that BERT has different representations for distinct senses for each modal verb, but does not represent modal sense independently from modal verbs. We also show that performance in different varieties of English is not equal, pointing to a necessary shift in the way we train large language models towards more linguistic diversity. We make our annotated dataset of modal sense in different varieties of English available at https://github.com/wagner-jonas/VEM.

## data we use two existing datasets to test whether bert can differentiate modal verb senses and construct a new one, taking data from the international corpus of english (ice). the first is a portion of the multi-perspective question answering (mpqa) dataset that has been annotated for modal sense @xcite . this consists of 1,201 sentences taken from news articles dated june 2001 to may 2002 @xcite . the second is the heuristically tagged epos-e dataset @xcite , based on the europarl and opensubtitles-english datasets, consisting of data from the european parliament and film subtitles, comprising 2,453 sentences. modal sense is annotated for each sentence in both datasets. for comparability, we do not report results for ought, as it is not evaluated in previous publications either @xcite . we also remove might and shall from our results due to their low frequency. the main difference between the two datasets (besides the annotation methodology) is size, with epos-e being almost twice the size of mpqa. they also draw their data from different sources, which is important given the genre effects found by @xcite . mpqa includes more senses than epos-e, which we discard in our analysis to maintain comparability. the balancing for the different senses for each modal verb also varies between them: the most common sense for must makes up 92% of instances in mpqa, but only 60% in epos-e; for may, this is 74% and 87%; for can 67% and 84%; for could 65% and 43%; and for should 92% and 94%. this, naturally, may impact classification results. as in previous research, we only investigate the modal verbs that are annotated in the dataset in cases where there is more than one modal verb per sentence. for the last experiment, in which we test a classifier trained on epos-e on data from different varieties of english, we use the written components of eight sub-corpora from the international corpus of english (ice; https://www.ice-corpora.uzh.ch/en. html ), a comparative corpus of varieties of english. for each variety, the same kinds of documents (like student writings or fiction) are used to compile subcorpora of about 400,000 written tokens for each variety (see http://ice-corpora.net/ice/index.html for more information). we investigate philippine (ph), canadian (ca), irish (ir), hong kong (hk), sri lankan (sl), jamaican (ja), nigerian (ni), and indian (in) english. for each modal verb in each variety, we randomly extract 20 sample sentences that contain the modal verb for a total of 800 sample sentences. three annotators independently annotate these. we discard all instances where no two annotators agree on one sense, where the sense is unclear (e.g. due to missing context), and false positives (e.g. must as a noun instead of a modal verb). this leaves 782 sentences for analysis. agreement between the first and second annotator is highest (83.75%), followed by agreement between the second and third annotator (79.88%), and between the first and third annotator (78.00%). we use the majority labels as gold labels. we call this corpus vem -the varieties of english modal sense corpus -and make it available at https://github.com/wagner-jonas/vem . ## conclusion and outlook our experiments have demonstrated that bert does not appear to have any representations of modal sense as its own category. classification did not show satisfactory results for either modal verb sense the embeddings of modal verbs or modality in the [cls] token. however, bert showed some ability to predict masked modal verbs, though its success depends greatly on which modal verb has been masked, making it unclear whether this is truly an ability to predict specific modal verbs or rather prediction of any modal verb. modality does not appear to be encoded in the [cls] token at all, calling into question whether sentence-level encodings of modality exist in bert. however, different classifiers may yield different results, and representations of sentence meaning other than the [cls] token (such as summing up embeddings) may yet encode modality. further research is thus necessary to come to a complete conclusion. classification was most successful when done separately for each individual modal verb. this indicates that, while bert may not have representations of modal verb sense as its own category, it does appear to encode sense differences for each modal verb. thus, it can differentiate between must in sentences like "you must complete all tasks for course credit" and "you must be tired after the long journey", but it also views the deontic modal verbs must and should in a sentence like "you must/should do your homework" as different. this has some intuitive appeal -clearly, the actual meanings of the sentence change quite considerably with the strength of deontic obligation expressed by must and should, respectively. the results of the last experiment demonstrate that the difference in modal verb sense use across different varieties of english may negatively impact this performance. some varieties (canadian, sri lankan, indian, and jamaican english) reach comparatively good performance, while modal verb sense classification in irish english proves difficult. it is clear that more focus must be put on linguistic diversity for language models to be more useful for such (often marginalised) varieties. further research into bert's representations of modal sense may focus on non-categorical representations of modal sense. those who have annotated modal sense can attest that it is not always very clear-cut, and often, more than one interpretation of modal sense can be perceived as valid. as bert embeddings are continuous -only our classification forces them into categories -researchers may want to investigate whether non-continuous bert embeddings of modal verbs also match human annotators' certainties or disagreements.