Crowdsourcing Veridicality Annotations in S panish: Can Speakers Actually Agree? (2023)

Abstract: In veridicality studies, an area of research of Natural Language Inference (NLI), the factuality of different contexts is evaluated. This task, known to be a difficult one since often it is not clear what the interpretation should be Uma et al. (2021), is key for building any Natural Language Understanding (NLU) system that aims at making the right inferences. Here the results of a study that analyzes the veridicality of mood alternation and specificity in Spanish, and whose labels are based on those of Saurí and Pustejovsky (2009) are presented. It has an inter-annotator agreement of AC2 = 0.114, considerably lower than that of de Marneffe et al. (2012) (κ = 0.53), a main reference to this work; and a couple of mood-related significant effects. Due to this strong lack of agreement, an analysis of what factors cause disagreement is presented together with a discussion based on the work of de Marneffe et al. (2012) and Pavlick and Kwiatkowski (2019) about the quality of the annotations gathered and whether other types of analysis like entropy distribution could better represent this corpus. The annotations collected are available at https://github.com/narhim/veridicality_spanish.

## introduction often when hearing an utterance we try to assess whether the information conveyed is likely to be truthful or not, that is, if it corresponds to actual situations in the real world @xcite . furthermore, as speakers or authors, we normally seek to convey what we know about the truthfulness or factuality of the events conveyed. for simplicity, here events are just considered as anything that happens or is like "being tall" or "having read a book". in the realm of linguistics, several features lead us to make the correct inferences about the factuality of an event, and comprehending them is key for building any system that aims at understanding human language. for example, the presence of the negation adverb "not" in "pedro has not done the laundry", leads us to infer that the event "pedro has done the laundry" did not happen, unless we know something about the speaker or pedro that makes us think otherwise. furthermore, in "pedro could have done the laundry" the modal verb "can" makes the event a possibility. this kind of analysis is what is called a veridicality study, more specifically, veridicality is an area of research within natural language inference (nli) and theoretical linguistics that studies the truth value of a proposition or event in a specific context @xcite . as to nli, it is a branch of natural language understanding (nlu) with its main task being entailment classification, that is, as it has been done above, to classify the relationship between two sentences, a premise, and a hypothesis, by picking a label from a usually small set of labels like {entailment, neutral, contradiction} @xcite or {yes, unknown, not}, depending on how the task is defined. so for example, we can classify the relationship between the premise "pedro has not done the laundry" and the hypothesis "pedro has done the laundry" as a contradiction or not. here a study of veridicality judgments in spanish is presented. specifically, the goal is to analyze how mood alternation, in other words, the possibility of using a verb either in indicative or subjunctive mood; and the specificity of the syntactic subject, that is, the identifiability of the referent in the discourse universe @xcite , affect factuality judgments about an event. this goal is realized in the following research questions: rq1.-in a complex sentence, how does the mood alternation of the embedded verb that occurs due to the negation of the main or matrix verb affect the factuality value of the embedded event? rq2.-how does an individual subject affect the factuality judgment of the event? to answer these questions a crowdsourcing experiment was run on toloka @xcite in which annotations from linguistically naive native spanish speakers were gathered for a corpus specifically designed. the corpus and the annotations are publicly available and their analysis is shown here. next, section 2 introduces the main concepts used here and presents the most important references to this work. then, section 3 explains how the corpus was designed and how the annotations were gathered. after that, section 4 presents the statistical and linguistic analysis of the annotations gathered, section 5 discusses the main issues seen throughout the study, and, finally, section 6 answers the research questions and proposes some lines of future work. ## corpus and annotation process the first step for creating the corpus was defining the experimental design. to do so, each of the research questions was set as one experimental condition: negation, individual, and collective. then the negation condition was divided into three categories: baseline, indicative, and subjunctive. the first refers to the case where there is no mood inductor, as in el presidente dijo que el país tenía problemas económicos (the president said that the country had economic problems). for both the indicative and the subjunctive categories we have the mood inductor no (not), but on the former the embedded verb is in the indicative mood, as in (2a), and on the latter, the verb is in the subjunctive mood, as in (2b). finally, these three negation categories were crossed with the specificity conditions, that is, with the individual and collective conditions. once the design was defined, the pairs were created. 30% of them were written manually and the rest were based on different corpora. specifically, possible premises in the indicative category were extracted with the help of linguakit @xcite ) from the following corpora: a section of the davie's corpus del español @xcite , the old news corpus for spanish @xcite , el quijote by miguel de cervantes (as found in dario (2017)), the xnli corpus, and the united nations corpus in spanish for the years @xcite @xcite @xcite . after that, some small modifications like reference resolution and reducing the number of words were done, hypotheses were extracted and pairs were modified according to the experimental design. finally, each pair was automatically annotated with additional infor-mation that could be used to later model the results. to annotate the corpus the labels displayed in figure 1 were used. these labels correspond to the set from @xcite minus certain but unknown output (ctu), as in de @xcite , and although they mapped them to the traditional square of opposition, here the labels are presented in an ordered linear scale of factuality. furthermore, given that the acceptability of mood alternation is not always certain, the label "not a sentence" (nas) was added. since does not fit within the scale, it is presented outside of it. consequently, the final set of labels is: certainly yes (ct+), probably yes (pr+), possibly yes (ps+), unknown or uncommitted (uu), possibly not (ps-), probably not (pr-), certainly not (ct-), not a sentence (nas). the experiment was run on the platform toloka @xcite . to select the workers two main criteria were used: language and country. they were required to have set up spanish as a language and their ip address from a country where spanish is an official language or an important minority one. in the beginning, all annotators under these criteria were eligible, but then this was reduced to the top 30% of annotators. they were paid $0.433 per set of pairs, which consisted of no more than 10 pairs. once all the annotations were gathered, pairs for which one worker or more used the label nas were removed, and if an annotator had labeled more than 1 pair within a single combination of experimental conditions, all his annotations in that combination were removed. this left a total of 477 pairs and 7 annotators per pair. the task was designed as in de @xcite : given a context (the premise), workers had to label the factuality of the event (hypothesis) by choosing one of the labels in figure 1 from a drop list. ## analysis of annotations overall distribution. as we can see in figure 2 , the distribution of label counts is negatively skewed, that is, there is a clear preference for the positive labels, even if more than half of the corpus sentences are negated. furthermore, the ## discussion the results of this study have shown that there is a tendency for annotators to use positive labels, even when most of the pairs of the corpora are negated. furthermore, the inter-annotator agreement score, ac 2 = 0.114, is rather low since it is barely within the range of slight agreement. therefore, it is important to discuss what could have caused such a lack of agreement. @xcite defines five sources of disagreement: errors and interface problems, annotation scheme, ambiguity, item difficulty, and subjectivity. although the first factor cannot be completely disregarded, given the persistence of some quite divided agreement patterns seen in figure 3 and that the interface was a simple drop list, the higher level of disagreement cannot entirely be blamed on this first factor. as to the annotation scheme, some improvements could most certainly be implemented. specifically, implementing training and testing. this was not done in fear of leading annotators towards concrete labels, but after encountering the work of @xcite , where they used carefully crafted training and testing that did not fully prevent disagreement, i understood it could be possible, even if not easy. also, given that the specific criteria for quality control, like what exactly annotations done too fast look like, more carefully defined criteria would clear out results. in addition, since the model fittings for subsets with more even distributions concerning the matrices showed better results, a more balanced corpus could yield more informative clmms, but there is no reason to believe that it would improve agreement. lastly, given the negatively skewed distribution for the label counts, the assumptions made about the veridicality of negation need to be revised. regarding the ambiguity of the relation between the different premises and their hypothesis, the two factors mentioned in the manual analysis (world knowledge and modal verbs) and the highly frequent divided agreement patterns shown in figure 3 , suggest that there is not always one clear label for a pair. this supports the existence of inherent disagreement between annotators, although more data is needed to confirm it. furthermore, the fact that these annotations are done from a pragmatic perspective and that mood alternation is a pragmatic phenomenon, also increases the uncertainty, and therefore ambiguity, of the annotations. concerning the fourth possible cause of disagreement, item difficulty is here a certain cause of disagreement. firstly, in the manual analysis, it was demonstrated that there are different factors to be considered when given a factuality judgment, mainly world knowledge. secondly, previous work has shown nli annotations to be difficult @xcite . as to the last factor for disagreement, subjectivity, it also influences the results presented here. although, to my knowledge, there is no previous work that supports this as cause for disagreement in nli annotations, the analysis of example 3 shows that world knowledge influences speakers' judgments, consequently making annotations dependent upon annotators' knowledge and point of view. now that it has been explained what caused disagreement in these studies, the question is if an inter-annotator agreement score, let it be ac 2 or any other, can reflect the nature and quality of the annotations gathered. the simple answer is no, at least not entirely. as stated in gwet (2014), inter-annotator agreement scores reflect how much the annotations change when small adjustments in the annotators like replacing a number of them are made, that is, it is a measure of data reproducibility based on the individual annotators. however given that it has been proven that these annotations are highly dependent on the speaker, measuring the reproducibility of the data based on such small variations is misguided and different evaluation scores are needed. ## conclusions based on the results presented here we can conclude that the specificity of the subject, defined in terms of the identifiability of the referent in the discourse universe and manipulated by having individual and collective nouns does not have a significant effect on the factuality of the embedded predicate in complex sentences, or in other words, individual vs. collective subjects are non-veridical contexts concerning the embedded predicate. as to the effect of mood alternation due to the negation of the matrix verb, that is, when due to negative adverb no (not) modifying the main verb of a complex sentence the subjunctive mood is induced in the embedded predicate but the indicative is also accepted; there is overall a significant difference on the factuality of the embedded predicate between having or not the negative adverb, but not between having the embedded verb in the subjunctive or in the indicative mood. however, the results from fitting different models to specific subsets of the corpus suggest that there is a small significant difference between the indicative and the subjunctive categories in specific cases. the analysis presented here has focused more on what the data looks like and it has scrapped the surface of why it looks like that. therefore, an important line of future work is a thorough analysis of the annotations in terms of what causes the results found. a second line of work is a different statistical analysis. the methods chosen here assume that there is one single underlying truth for each premise-hypothesis pair, but as the analysis of the disagreement patterns has shown, there is a non-neglectable number of pairs for which this is not the case. consequently, methods that expect disagreement, like the entropy distribution seen in @xcite , might be insightful. the third and last line of work proposed is the inclusion of out-of-sentence context in the corpus, especially since it was recommended in @xcite . the question about its inclusion was already raised while designing the corpus, but it was disregarded due to its cumbersome implementation and the increased difficulty in the analysis. but given the results obtained and the influence of context in mood alternation @xcite , adding context to the pairs could yield more informative results.