Modality annotation for P ortuguese: from manual annotation to automatic labeling (2016)

Abstract: We investigate modality in Portuguese and we combine a linguistic perspective with an application-oriented perspective on modality. We design an annotation scheme reflecting theoretical linguistic concepts and apply this schema to a small corpus sample to show how the scheme deals with real world language usage. We present two schemas for Portuguese, one for spoken Brazilian Portuguese and one for written European Portuguese. Furthermore, we use the annotated data not only to study the linguistic phenomena of modality, but also to train a practical text mining tool to detect modality in text automatically. The modality tagger uses a machine learning classifier trained on automatically extracted features from a syntactic parser. As we only have a small annotated sample available, the tagger was evaluated on 11 modal verbs that are frequent in our corpus and that denote more than one modal meaning. Finally, we discuss several valuable insights into the complexity of the semantic concept of modality that derive from the process of manual annotation of the corpus and from the analysis of the results of the automatic labeling: ambiguity and the semantic and syntactic properties typically associated to one modal meaning in context, and also the interaction of modality with negation and focus. The knowledge gained from the manual annotation task leads us to propose a new unified scheme for modality that applies to the two Portuguese varieties and covers both written and spoken data.

## introduction there has been a growing interest in text mining applications that can automatically detect opinions, facts and sentiments in texts. many of the current opinion or sentiment mining applications use a crude division between negative, neutral and positive sentiments. modality, defined from a linguistic perspective as the speaker's attitude towards the proposition in the text @xcite offers a theoretical framework to make more fine-grained distinctions between different attitudes. for example, one can detect whether the speaker expresses his or someone else's commitment, hope, belief or knowledge about a certain proposition. in this paper we aim to combine a linguistic perspective with a practical and application-oriented perspective on modality. on the one hand we design an annotation scheme reflecting theoretical linguistic concepts and apply this schema to corpus data to fit with real world language usage. on the other hand we use the annotated data not only to study the linguistic phenomena of modality, but to train a practical text mining tool to detect modality in text automatically. the concept of modality is related to factivity, evidentiality, hedging, and is consequently grounded on notions of subjectivity and attitude. the study of modality relates to the question of how subjective language use is. how many of our utterances and sentences are modalized? how is subjectivity and attitude grammaticalized in the sentence through modal markers? are certain genres more prone to subjectivity? and how successful can we be in dealing with modality in automatic processing systems? these questions depart from a purely formal approach to language and move into a cognitively grounded discourse perspective. while certain linguistic aspects such as morphology, lexicography and syntax have been addressed in interdisciplinary studies combining theoretical and computational approaches that have produced a body of available resources, other aspects such as semantics and discourse are still lacking the same level of attention. although modality has been intensively explored from a theoretical linguistic perspective, there are few theoretical studies relying on corpus data and no concerted effort to create a resource covering this topic. in this paper we aim to contribute to filling this gap and provide an annotation schema for modality based on linguistic theory but firmly grounded by applying and revising the schema based on corpus annotation. this schema is also evaluated from a computational perspective as we use the annotated data to train an automatic modality tagger intended to be part of a text mining application. moreover, many of the existing efforts in this domain are applied to written texts, while we discuss here the annotation of both written and spoken data. furthermore, while most work on human and automatic annotation of modality is focused on english, we provide feedback on another global language, portuguese, and address language diversity by covering two national varieties of portuguese (brazilian and european). corpus annotation does not solely provide the basis for text mining applications, but sheds new light on linguistic issues. indeed, we believe that corpus annotation triggers research questions left untouched when exclusively assessing constructed samples and allow to redefine categories after assessing the contexts. first, the specific task of labeling each modal marker in context challenges the usual theoretical perspectives that center on specific lexical items, mainly semi-auxiliary verbs, and leads us to reflect on the diversity of modal markers, providing the necessary data for an approach at the crossroad with typological linguistics. second, this has implications over the modal types traditionally described in linguistics. it is necessary to address the complexity of corpus data and keep a balance between achieving accuracy and preventing the proliferation of labels. this inevitably requires an investigation of the role played by modal marker's ambiguity. third, the analysis of corpus data will also provide empirical insights into theoretical issues to modality: modal logic @xcite and discourse-focused modality @xcite ; propositional modality and event modality @xcite . finally, embedding the annotation in actual corpus data reveals the interaction of modal markers with systems such as negation and focus, which falls outside modality but may affect the modal interpretation. computationally, the automatic identification and interpretation of modalized statements is a prime concern in a large number of applications, especially with the recent attention to opinion mining and social networks. as the vast amount of digitally available data keeps growing, so does the demand to automatically extract relevant information. we see a clear trend in information extraction applications to go beyond the extraction of pure facts, to focus on personal opinions in sentiment analysis and opinion mining, and to distinguish between factual and probable information @xcite , to detect uncertainty, speculation and negation, especially in biomedical text mining @xcite @xcite . modality detection is therefore also clearly linked to the current trend in nlp on sentiment analysis and opinion mining. we will report on the two modality annotation schemes designed for portuguese, each addressing a specific national variety and a specific text type. the european portuguese (ep) scheme was envisaged as genre independent and tested on written data, while the brazilian portuguese (bp) scheme was designed for spontaneous speech, and modality is taken as the evaluation or the point of view of a conceptualizer towards the locutory material in a given utterance in a communicative act. both schemes cover different grammatical categories that convey modal values, such as adjectives, adverbs, nouns and verbs which we will denote as modal trigger in the rest of this paper. the ep scheme has been manually applied to a small written sample of 160k tokens @xcite while the bp scheme was manually applied to a spoken sample of the coral-brasil spoken corpus @xcite . the ep annotated sample was used as the basis for the development of a software tool that can automatically detect modal triggers and their scope in text. such automatic tool has many practical applications in the area of automatic document understanding or information extraction. the first experiment has been to automatically label a subset of 11 highly frequent and ambiguous modal verbs. for instance, the portuguese verb poder is highly ambiguous and can express multiple modal meanings: "to be possible", "to be able" or "to give permission". this polysemy increases the level of difficulty of the automatic annotation task. to create the modality tagger, we first automatically assign lemmas, pos and syntactic tags, we then automatically identify modal triggers and apply a machine learning approach to attribute a modal value to the triggers, comparing the results with our gold (manually annotated) labeling. constructing an automatic modality tagger requires a data set with labeled examples to train and evaluate the tagger. as we are currently in need of a suitable data set, one of the goals of the current experiments is to develop an automatic modality tagger on a small manually labeled sample that can later be applied (semi-automatically) to generate a larger data set. our paper first discusses related work on modality, its annotation and automatic labeling in section 2. the two schemes are presented in section 3. the automatic labeling tool trained over the ep corpus is presented in section 4. in section 5 we discuss several valuable insights into the complexity of the semantic concept of modality that the process of manual annotation of the corpus, its linguistic exploration and the analysis of the results of the automatic labeling have provided us. in the last two sections we discuss the next steps that we aim to take such as using a unified approach to modality annotation presented in section 6 and we conclude in section 7. ## related work modality is traditionally grounded in epistemic modality, which is concerned with the truth-value of the proposition: whether the proposition is considered by the speaker (or another entity) as certain, uncertain, possible, probable etc. in languages with no modality-exclusive morphological system, such as english or portuguese, research has mainly focused on semi-auxiliary modal verbs, e.g., can, may, must. for instance one can have different perspectives on the proposition (ana buy a house), such as: ana may buy a house, ana must buy a house, ana can buy a house. other modal types besides epistemic modality have been proposed that reflect some degree of subjectivity towards the proposition (or towards the event). although terminologies vary considerably, the concepts at hand are relatively constant: they include notions of obligation and permission, of capacity and necessity, of evidentiality (what is the source of information), wish and evaluation. this broader scope of modal types is reflected in the growing attention to a large set of linguistic categories that may convey modal values, such as tense affixes, mood, auxiliary verbs, main verbs, adjectives, adverbs and multiword expressions. in fact, several lexical items conveying modality may occur in the same context, and, furthermore, many of these items are ambiguous between one or more modal values. the complexity of the phenomenon does not undermine its importance in language to convey attitudinal information, such as belief, uncertainty, factuality and evidentiality. most of the research on modality has been based on constructed examples, although the importance of looking at modal items in context is increasingly acknowledged, on a par with a clear trend in corpus linguistics and natural language processing (nlp) to go beyond the part-of-speech (pos) and syntactic levels and to include semantics, pragmatics and supra-sentential information. this interest gave rise to some proposals for the annotation of modality in corpora, mainly for the english language. the annotation schemes covering modality differ greatly in their objectives and in the nature of the concepts that are labeled @xcite . according to august 2016 @xcite , computationally, the automatic identification and interpretation of modalized statements is a prime concern in a large number of applications, especially with the recent attention to opinion mining and social networks (pp. vii) but the computational linguistics community is still far from having developed working, shared standards for converting modality-related issues into annotation categories. (pp. vii) modality may be one aspect of the semantic information encoded in the properties of events (cf. @xcite @xcite @xcite ) or it can be the core of the scheme. for instance @xcite use a restricted notion of modality and establish conditions for an expression to be considered modal, such as the requirement for a propositional argument. in other cases, modal values are included in annotation schemes that cover both factuality and modality, as in the work of @xcite that distinguish between factual and probable information, while @xcite specifically address factuality, rated on a scale of -3 (certainly did not happen) to 3 (certainly did). different annotation schemes also diverge in what textual elements they annotate: the modal value may be attributed globally to the sentence/event or it can be encoded on specific lexical items. the applied nature of these studies leads to detailed description of the textual elements involved in the expression of modality and the roles they have: most schemes identify a modal trigger, the subject of the modality (source) and the elements in the scope of the modal trigger (target/scope/focus). the availability of large sets of data annotated with modality provides important insights on the interaction between modality and other linguistic systems, such as negation @xcite and focus @xcite . another application of these data sets are experiments in the automatic annotation of modal values for information extraction and data mining. in bionlp, @xcite annotated pre-recognized events with the epistemic value "level of certainty" and attain f-measures of 74,9 for "low confidence" and 66,5 for "high but not complete confidence". the factuality-oriented scheme presented by @xcite has been applied in an experiment of automatic identification of events and their modal features in text, and attains 97.04 accuracy with the evita tool. a specific task for detecting uncertainty through the use of hedging clues was organized at conll2010 @xcite . contrary to these approaches, @xcite include both trigger and target in their experiments, and attain 86% precision with a structure-based tagger over 249 modality-tagged sentences from the english side of the nist 09 mteval training sentences. the approach of @xcite covers modality but is essentially geared towards the identification of belief. contrary to our own approach, the authors do not take into consideration the polysemy of the auxiliary verbs and only encode the epistemic value, although they do report that the verbs may be deontic in some contexts. this experiment has been extended to other modality values (ability, effort, intention, success and want) @xcite . the work of @xcite focuses on english modal verbs. the five english modal verbs (can/could, may/might, must, ought, shall/should) were first identified in texts and their modal values were predicted by training a maximum entropy classifier on features extracted from the training set. the classifier achieved an improvement of the most frequent sense baseline for all verbs but must, and accuracy numbers between 68.7 and 93.5. our experiments presented in section 4 are closely related to this type of automatic modal verb labeling. this brief overview of related work has shown the diversity of objectives, annotation choices and evaluation metrics, which makes it difficult to compare results. ## annotation schemes we present two separate modality annotation schemes for portuguese. although they apply to different varieties of portuguese, the general structure of the schemes is very similar. for instance, the range of phenomena that are not considered as modality, and are not annotated, is almost equivalent in both schemes: ## automatic labeling the ep corpus of 160k has been the source of data to train an automatic tagger for modality. many of the verbs in the data set, such as the high frequent modal verbs querer 'to want' and tentar 'to try', only have one modal value and for those, assigning the correct modal value becomes a trivial task. we focus instead on modal verbs with multiple modal meanings that each occur at least 5 times in the small annotated corpus sample. the labeling is done in several steps. first, the dataset is preprocessed with a syntactic parser and the parser output is used to detect the modal triggers. then, a machine learning approach is used to label each modal trigger with the appropriate modal value in context. the palavras parser @xcite was used for the syntactic analysis. it distinguishes between modal and non-modal uses of the verbs poder and dever by labeling the former as auxiliary verbs. by exploiting the parser's predictions, we were able to detect the modal usage of the two verbs with an f-score of 98%. ## corpus annotation as a source of linguistic insight on modality the manual annotation of the corpus, its linguistic exploration and the analysis of the results of the automatic labeling have provided valuable insight into the complexity of the semantic concept of modality, its intrinsic ambiguity, the interaction between embedded modal triggers and embedded sources, and the interaction between modality and other systems such as negation. the information gathered during the annotation itself and the error analysis of the tagger provided feedback to improve our modality scheme and is part of the motivation for the presentation of a unifying proposal in section 6. we discuss some aspects where our experiments have proven to shed some light on the concept of modality and its annotation. one such example is reproduced in (7): the trigger deveriam 'should' is preceded by the negation adverb and is followed by another trigger possibilitado 'made possible/enable'. the linguistic context of the trigger is misleading: although there is a negation marker, this operator doesn't have scope over deveriam but instead over its target (it should (not be possible to...)). example sentences ( 8 ) and ( 9 ) show two embedded triggers that both have negative polarity values. in sentence (8), the deontic verb poder 'should' has negative polarity and the embedded trigger conceder 'to concede' is preceded by the aspectual expression deixar de 'to cease' that carries intrinsic negative polarity. the double negation (should not cease to concede) is equivalent to a global positive polarity (should concede). in (9), the first trigger conveys an epistemic value and a negative polarity (não julgo '[i] don't believe'), while its target contains a second trigger that lexically expresses negative polarity (imprescindível 'indispensable') and denotes a deontic obligation value. the overall interpretation of the sentence is that the source believes that something is dispensable. in both cases, our annotation treats each trigger independently and does not express the overall positive polarity of the clause. the modal interpretation of an embedded trigger may also be affected by the preceding trigger. in sentence (10), the trigger pode 'can' influences the degree of certainty of the modal value evaluation of the second trigger difícil 'difficult'. august 2016 (9) não julgo imprescindível que se encontre desde já, a correr, o novo rosto. 'i don't think that it is indispensable that we find right now, immediately, the new face'. despite the challenge posed by these contexts, we believe that it is important to individually mark each element of the sentence that contributes to the modal values to be able to understand how each provides input to the overall semantic interpretation. the global interpretation of the sentence does not only depend on the modal information but is determined by an interplay of information at different levels including focus and negation. investigating these interactions at the global sentence level is beyond the scope of this current work but promises to be a fascinating future research direction. the annotation of corpus data also points to the fact that exclusive particles, such as só 'only', may affect and alter the modal meaning of the sentence. for instance, in contexts where the verb poder has an epistemic reading, adding the exclusive can restrict the set of possibilities to the possibility presented in the sentence, as illustrated in (11). in (11), by restricting the set of possible situations to one (x and only x), the adverb leads to an overall reading of the sentence that expresses epistemic necessity. this does not hold, however, in all such contexts: the interpretation requires that the target of the modal trigger is a state or a past event, and that the exclusive particle has scope over the full target of the modal trigger. compare, for instance, the modal readings of the corpus sentence (12) and the slightly adapted version in (13). in (12), the exclusive has scope over the temporal adjunct and denotes a condition over a possibility, while in (13) it has scope over the full target and the overall reading is one of epistemic necessity. the exclusive particle can also mark the possibility as weaker than expected. exclusives can downtone an alternative, by underlining the fact that this proposition "is not the strongest that in principle might have been the case", a function called mirative @xcite . in ( 14 ), the combination of the exclusive with the deontic marker doesn't express one obligation among a set of possibilities, but rather states that the alternative is much easier than one would expect. these meaning interactions are extremely challenging for any annotation scheme and these contexts tested the limits of what could be represented in our current schema. to represent and compute the meaning of such sentences, negation and focus (and any other element that affects modality) could either: (i) be integrated in the modality scheme, an approach that we follow partly by marking the polarity of trigger and target, and also by proposing to deal with exclusive particles as another modal marker in the modal set @xcite ; (ii) be the topic of separate layers of annotation with their own annotation scheme; (iii) be included as features of a global annotation scheme that would take the event as the tagging unit, instead of individual triggers. the role played by some of these attributes in the labeling of permitir is quite unexpected and was uncovered by the analysis of the results of the system. this was the case, for instance, of the attributes "no explicit subject" and "dative brother node in left context". ambiguity has typically no formal correlate in the nature and span of the components trigger and target of our annotation. for instance, in example (19), permitir has, on the one hand, an epistemic possibility reading, in the interpretation that the climate makes it possible for trees to grow. on the other hand, it expresses deontic permission if we consider that the climate is a necessary condition for the growth of the trees. in both cases, the subject np as condições climáticas will be marked as the source (of possibility or necessity) and the object will be the target. however, there might be a formal counterpart to semantic ambiguity. it is the case of contexts that are ambiguous between deontic obligation and participant-internal necessity with verbs dever and ter de, and epistemic possibility and participant-internal capacity with verb poder. in (20), two interpretations -and therefore two annotations -are possible. the epistemic possibility reading might be paraphrased as it is possible that [he is an excellent candidate], so that the target of the modal trigger is the whole proposition and includes the subject of the sentence ele@ser um excelente candidato foot_4 . the source of the modality will then be the speaker/writer. the internal capacity reading can be paraphrased as he has the capacity to be an excellent candidate: the source of the capacity is the subject and the target is the verbal phrase. currently the scheme has no way of capturing these structural ambiguities and only one structure, representing what the annotator considers to be the most salient meaning in the context, is expressed, although semantic ambiguity is marked in the ambiguity attribute of the trigger. accordingly, we expect our system to be able, in future experiments, to label these contexts with two possible modal meanings. the ambiguous modal triggers that we have discussed so far can denote two or more individual modal meanings and, in most cases, the context provides the necessary information to select the appropriate meaning (see @xcite and @xcite for different concepts of the polysemy vs. indeterminacy of modal verbs). in these cases, only one reading is available at a time and the interpretations belong to different modal categories (e.g. epistemic vs. deontic) @xcite . however, the verb conseguir 'to succeed' seems to behave differently in that the capacity reading and the success reading are not non-compatible discrete meanings, but rather intertwined aspects of the semantics of the verb. in fact, in most contexts of the verb, the value success is associated with an internal capacity of the subject. this type of semantic indeterminacy is related to the concept of gradience in the work of @xcite and would differ from cases of pure ambiguity. this is particularly important to assess, first, the relations that exist between the modal values included in the scheme to address any possible overlap in the case of some lexical triggers (not always foreseeable when establishing which modal values to include), and second the performance of the annotation tool. cases of gradience, where two or more values are simultaneously present in a single context, although one might be more salient than the other, will be especially difficult to tag by a manual annotator (the choice might be more prone to context influence, such as perfective tenses leaning to the success reading due to their association to factuality). ## future work: an unified scheme for portuguese and its application the proliferation of annotation schemes for modality is inevitable and the result of specific objectives of the different teams working on the topic. however, some attempt of standardization would be of interest to the field, making contrastive studies an attainable goal. in the case of the ep and bp schemes, the objectives are quite similar and the properties of both varieties do not differ in the components of the schemes, although the list of lexical triggers might be variety-specific to a certain extent. the preparation of a unified proposal is also an opportunity to refine both schemes according to our experience and the knowledge acquired during the manual and automatic labeling. as mentioned in section 3, the set of components is practically identical in each scheme. the differences arise essentially in the list of modal values, the target dependent component and the trigger and target attributes. let us start with the mismatches in modal values. ## conclusion we have presented two modality schemes for two varieties of portuguese, covering written and spoken registers, and an experiment to automatically annotate modality for 11 portuguese modal verbs, using a corpus sample of 160k tokens, manually tagged with modal values. the range of phenomena that are not considered as modality in ep and bp schemes are very similar, and so are the components of both modality schemes. differences are essentially genre-related since the bp scheme is applied to spontaneous speech and considers the scope of the modality to be the information unit. the ep scheme was applied to a sample from the ep written sub-part of the reference corpus of contemporary portuguese (crpc), while the bp scheme was applied over a sample from the c-oral-brasil. based on our annotation experience of these two corpus samples and on the automatic experiments on the ep corpus, we prepared a unified proposal for modality annotation in portuguese that applies to both written and spoken modality. for the automatic labeling of european portuguese, we selected verbs that had more than one modal meaning and occurred at least 5 times in the corpus sample. using the weka framework we conducted several experiments to study the effect of the usage of linguistic information to identify modal values. we compared those against the majority baseline and a bag-of-words approach and calculated precision, recall and f 1 measures. the system performed well above the majority baseline. we were able to get better results (precision and f1) with some settings for some verbs (permitir and saber), but most experiments, even with higher performance values, were not significantly different from the bag-of-words approach. we assume that this was mainly due to the small number of training examples. we studied a range of au-tomatically generated linguistic attributes that can be used to identify the modal values for portuguese verbs. we can conclude from these initial experiments that the use of information extracted from parse trees does not harm the performance of the automatic taggers and can, for some verbs and combinations, enhance it. considering that our training corpus was relatively small and that we selected challenging verbs in our experiment, we believe that our goal, of creating a larger corpus with modal information by a (semi) automatic tagging process, could lead to positive results in the future. as a next step we aim to run new experiments with the modality tagger using the unified scheme as proposed in section 6 and apply the tagger both to european written and brazilian spoken portuguese data. the current modality tagger uses output from the palavras parser to create its feature representation. the palavras parser has been recently adapted to run on the spoken corpus coral-brasil @xcite and we can expect the parser to work well on the two different textual genres. we plan to follow up this analysis with a detailed study identifying the individual role of the syntactic and semantic features that are used for the automatic attribution of the modal value in our system. more specifically, for each parse tree we intend to evaluate the relevance of the following features in the modal value classification: word, lemma, pos tag, syntactic tag, semantic information and role label. the analysis will be performed over the partial parse trees that include the modal verbs and their parents and grand-parents and also over the nodes in the path from the root of the sentence parse trees to the modal verbs. we will also make a comparative study of the relevant features for each of the studied modal verbs. in fact, from the analysis of the results that we obtained so far, we suspect that there are important differences between different modal verbs. the fact that, for some verbs, the parse tree input obtains better results and for others it is better to use the list of words of the sentences, suggests that syntactic and semantic features might not be equally relevant for all verbs. we also aim to compute a learning curve to estimate the amount of manually annotated examples that are needed to get a good performance from the modality tagger. furthermore, to be able to label new verbs that did not occur in the initial data set, we plan to train a general modal trigger classifier that is not dependent on the verb itself.