Coherence of Argumentative Dialogue Snippets: A New Method for Large Scale Evaluation with an Application to Inference Anchoring Theory (2025)

Abstract: This paper introduces a novel method for testing the components of theories of (dialogue) coherence through utterance substitution. The method is described and then applied to Inference Anchoring Theory (IAT) in a large scale experimental study with 933 dialogue snippets and 87 annotators. IAT has been used for substantial corpus annotation and practical applications. To address the aim of finding out if and to what extent two aspects of IAT – illocutionary acts and propositional relations – contribute to dialogue coherence, we designed an experiment for systematically comparing the coherence ratings for several variants of short debate snippets. The comparison is between original human-human debate snippets, snippets generated with an IAT-compliant algorithm and snippets produced with ablated versions of the algorithm. This allows us to systematically compare snippets that have identical underlying structures as well as IAT-deficient structures with each other. We found that propositional relations do impact on dialogue coherence (at a statistically highly significant level) whereas we found no such effect for illocutionary act expression. This result suggests that fine-grained inferential relations impact on dialogue coherence, complementing the higher-level coherence structures of, for instance, Rhetorical Structure Theory.

## introduction the proper modeling of argumentation in dialogue is a long-standing challenge, raising questions about how individual and collective reasoning and argumentation are connected @xcite . in particular, a significant question is how coherence relations in debate are connected to the propositional relations of logical reasoning, that is conflict/oppose, and inference/support. an important proposal clarifying this relation is inference anchoring theory (iat) @xcite @xcite . this theory aims to account for the coherence of debates and offers the tools for argument corpus development @xcite , finetuning llms @xcite , and shedding light on, for example, the role of questions in debates @xcite . figure 1 shows a snippet of dialogue on the welfare state from the moral maze corpus @xcite annotated with iat. on the right-hand side, we can see four locutions, labeled l1 to l4. as the example shows, a locution consists of a speaker designation ('neil' or 'clifford') and an utterance. each locution is anchored to a proposition (shown in the four left-most blue boxes) via an illocutionary connection (ic) (in the middle yellow boxes). in this case, the two first locutions' illocutionary connection, i.e. ic, is 'asserting', the third is 'assertive questioning' and the last one is 'asserting'. propositions and ics represent the propositional content and illocutionary force, i.e. the speaker's communicative intention, from speech act theory @xcite . the locutions are linked by a transition box signalling a locution is a response to its predecessor. each transition between locutions is anchored, via an illocutionary connection for transition (icta), to a propositional relation. in our example, one transition is anchored via the 'arguing' illocutionary connection to the 'inference' propositional relation and two transitions are anchored via the 'disagreeing' illocutionary connection to the conflict propositional relation. as the example shows, 'inference' is used when one proposition provides a reason to accept the other proposition. in contrast, 'conflict' is used when one proposition provides a reason to not accept the other proposition. foot_0 the structure that iat assigns to naturallyoccurring dialogues is proposed in the tradition of theories that 'aim to account for dialogue coherence' @xcite , extending such works by using locutions as anchors for underlying propositional relations. the current paper proposes a novel way to test if and to what extent two aspects of iat -illocutionary acts and propositional relations -contribute to dialogue coherence. we have developed an algorithm, dialogue propositional content replacement (dpcr), that can generate dialogue snippet variants whose structures are entirely or partially iat-compliant. for this purpose, the algorithm replaces locutions in an iat structure with content from an argument map (capturing inference/support and conflict/oppose relations between propositions on a given topic), but in such a way that the iat-structure constraints (on propositional and illocutionary relations) are either satisfied, partially satisfied or not satisfied at all. for instance, replacing l1 and l2 in figure 1 with (l1) 'neil: i think that all humans should be vegan' and (l2) 'neil: in the sense that a world of veganism would be a more ethical word: its morals would bring benefits to human society', using claims from a kialo map on veganism, results in a new dialogue snippet that satisfies the same underlying propositional and illocutionary relations as the original dialogue. the full example, including (l3) and (l4) and further examples are provided in appendix g. also, the supplementary materials for this paper include all experimental materials ( https: //doi.org/10.21954/ou.rd.30161896 ). we obtain coherence ratings from human readers for these dpcr-generated dialogue snippets as well as for the original naturally occurring dialogue snippets and snippets generated with ablated versions of dpcr. this provides us with insight into: our work makes two contributions to the study of (dialogue) coherence. firstly, it introduces a novel method for empirically testing theories of coherence through a method of utterance substitution. secondly, by applying our method in a large-scale empirical investigation of iat, we show the viability of both the method, as well as the potential of certain aspects of iat for a possible model of dialogue coherence. the method and study are offered as exemplars for further work contributing to a long-term research programme into the factors underlying coherence in discourse. as such, the current work sits firmly within computational linguistics, as it uses an empirical study -enabled by computational means, i.e. an algorithm for generating dialogue snippet variants -to validate certain aspects of iat, a linguistic theory within the remit of discourse analysis and pragmatics. identifying such aspects could lead to a possible synthesis of existing theories, as described further in the related work section. in the next section, we describe related work. the remainder of the paper follows the standard american psychological association (apa) format for reporting experimental research. ## related work it is notoriously difficult to evaluate discourse analysis theories and annotation schemes, such as iat -however, we agree with the assessment of @xcite , who proposes that evaluation of a theory of coherence is 'considerably more com-pelling as empirical support' when done by means of an application of the theory for text generation that can then be assessed against judgements from 'actual readers'. @xcite argues against the evaluation of theory-based annotations directly and instead suggests that 'the theory must be assessed in relation to its predictions about independently observable phenomena in discourse', a point also made by @xcite . @xcite commends @xcite for their work on evaluating graph-based against tree-based theories by comparing the quality of summarised texts achieved by these different underlying representations. such empirical theory evaluation seems to have, however, had little uptake and, unfortunately, may even have less chance in future where most effort goes to experimentation with large language models. the method proposed in the current paper adds an important twist to knott and zaenen's recommendations, by going beyond between-theory comparison, enabling (via ablation) a better understanding of individual aspects of a theory of coherence that account for coherence judgements. head-to-head comparison of theories of coherence, such as rst @xcite , sdrt @xcite , or qud @xcite , may be challenging and even unnecessary. for instance, rst could be evaluated along the lines of our evaluation of iat, using as a substitution-based approach. to our knowledge, the only existing work in this space, using a substitution-based approach grounded in rst, was trialled in @xcite , but for monologue-to-dialogue generation rather than theory testing. a substitution-based experiment along the lines described in the current paper, but with rst, would not aim to determine whether rst is the better theory, but could be used to shed light on the elements of rst that matter. this in turn could lead to an integration of the key effective elements of rst and iat, a possibility considered by @xcite . in our view, rst and iat are complementary in that rst has a much wider range of relations that are needed to account for a much wider range of discourse genres beyond debate, whereas iat provides the means for dealing with fine-grained inferential relations found mostly, but not exclusively, in debate and that are beyond the scope of rst. showing that these fine-grained relations affect judgements of coherence does not falsify rst -rather we propose that this is evidence of a complementary level of structure that affects coherence. the current research only considers short, up to four-turn, dialogue snippets, given that longer snippets typically do not have a fully connected iat structure underpinning them. iat works well where relations between locutions can be mapped to underlying propositional relations. where these are absent, other factors will influence dialogue coherence that are better accounted for by theories such as rst. thus we return to our earlier point that to fully account for dialogue coherence, in the long run, a synthesis of theories based on a thorough understanding of what makes each effective, seems the best route to success. our experimental method and the results of its application to iat are intended to contribute to this long-term programme. dialogue snippet variants to address our aim of validating iat, the participants in the current study rated the coherence of dialogue snippets belonging to one of the following categories: ## results annotator reliability and coherence ratings in our experiment, the lower categories 1-4 are used much less than the higher categories 5-7. this makes our annotation unbalanced towards the lower categories. under such conditions, chancecorrected coefficients such krippendorff's α (krippendorff, 1980), fleiss's κ @xcite and cohen's κ @xcite are subject to the prevalence paradox @xcite and suboptimal. for this reason, we decided to report iaa to interpret the iaa values we used the landis and @xcite benchmark scale as revised and adjusted by gwet (2014a). 10 based on this analysis we got a level of agreement equal to or higher than fair for 83% of the batches. more precisely, 3% of the bathes reached a substantial level of agreement, 30% of the bathes reached a moderate level of agreement and 50% of the batches reached a fair level of agreement. finally, a slight level of agreement was reached for 17% of the batches. judging the coherence of a dialogue is not straightforward. many factors can impact dialogue coherence, and make a dialogue more or less coherent. this made the task of judging dialogue coherence a subjective one. accordingly, we consider the agreement reached in our study a satisfactory level of agreement. hypotheses ## discussion inference anchoring theory (iat) is a widely used theory that presents an appealing perspective on how dialogue coherence is underwritten by an underlying structure involving illocutionary acts and logical relations of conflict and inference between propositional contents. the current research aimed to examine which aspects of iat might account for dialogue coherence. using our novel experimental method, we collected the, to our knowledge, first empirical evidence that some of the underlying structure iat assigns to debates accounts, at least partially, for the coherence of those debates. to address our overall aim of finding out if and to what extent two aspects of iat (illocutionary acts and propositional relations) contribute to dialogue coherence, we tested four hypotheses. three of our hypotheses -(h1), ( h2 ) and (h4)were confirmed: we saw that iat-generated snippets were at least as coherent as naturally-occurring dialogue snippets from the moral maze corpus (h1). we also saw that dialogue snippets whose underlying propositional relations were selected at random (which is the case for both dpcr -rel and dpcr -rel -templ ), rather than driven by iat, are judged to have lower coherence than dialogue snippets that conform with the propositional relations mandated by iat (h2 and h4). these results suggest that the propositional relations posited by iat and their anchoring in dialogue is a factor that influences dialogue coherence. the other factor we considered, illocutionary acts, was, however, not confirmed by our study. more specifically, (h3) regarding dpcr -templ was not confirmed. in other words, judgements of coherence were not affected negatively by whether the illocutionary force (question version assertion) of the dialogue acts was reflected in their surface verbalisation (specifically through interrogative form for questions). this result may have several reasons: (1) inadequacy of the handcrafted generation templates for illocutionary force, (2) the fact that even in the dpcr -templ condition for those utterances without a propositional relation, illocutionary force was always expressed (as we explain below it was impossible not to do so) and (3) the ability of readers to infer implicit force. this final point is an empirical fact. though dpcr -templ does not convert argument map claims into questions (i.e. interrogative sentences) where the illocutionary connection requires this, in dialogue, whether a locution with a declarative sentence type is intended as a question can usually be inferred from the communicative context @xcite . in our argumentative dialogue set-up, involving discussions between a panellists and witnesses about contentious topics, a natural interpretation of locutions consisting of a declarative sentence is as raising questions for discussion by the other party -this is also in line with the more general idea that assertions can can initiate issues, i.e. introduce questions for discussion, which then become part of the qud, i.e. questions under discussion @xcite . in the limitation section, we will return to points (1) and ( 2 ), providing examples that illustrate the concrete limitations of our approach in this respect. an at first sight surprising result was that the dpcr and dpcr -templ algorithm-generated snippets were judged as more coherent than the original dialogue snippets. a possible reason for this is that the claims that the algorithm takes from the kialo maps are generally well-written and self-contained. in contrast, some of the original spoken language locutions in the moral maze snippets are less selfcontained and context-dependent. in our experimental design we compensated for this by creating versions of the moral maze snippets with added context. this helped somewhat, with moral maze +context original dialogues rated slightly higher than moral maze original , but still not at the level of dpcr and dpcr -templ . in any event, our main comparison was between the (ablated) versions of dpcr, allowing us to determine whether propositional relations and/or illuctorionary act (expression) play a role in dialogue coherence ratings. the fact that dcpr dialogues received coherence ratings comparable to those of the original dialogues reassures us that comparing dcpr dialogue ratings with ablated version ratings is warranted. another interesting finding is that participants mostly use the upper end of the rating scale (4-7) for coherence ratings. this may be explained by the general tendency of interpreters to expend effort to make sense of utterances even if, on the face of it, they do not make sense. an insight that has been widely discussed following on from the introduction of the notion of implicatures (grice, 1975) (i.e. additional inferences that addressees make to explain away any apparent lack of cooperativeness of speaker contributions). this does, however, not undercut the idea central to the current project, i.e. that there are degrees of coherence. apart from validating the role of iat propositional relations in dialogue coherence, the current work contributes to the computational study of argumentative dialogue by offering the dpcr algorithm (full and ablated versions) and its implementation for use by the research community as well as the corpus of 933 generated and naturaloccurring dialogue snippets together with their coherence ratings, with each snippet rated by 10 annotators out of a group of 87 annotators. this dataset (see https://doi.org/10.21954/ou.rd. 30161896 ) may also prove valuable for assessing to what extent llms as judges agree with human dialogue coherence judgements in our dataset, especially since the dataset was collected just before the release of chatgpt (and thus not subject to the current risk of raters on crowdsourcing platforms such as mturk making use of llms). ## limitations contra our hypothesis (h3), the dpcr -templ algorithm-generated snippets were rated as highly in terms of coherence as dpcr-generated snippets. we had expected that by switching off the template generation, the coherence would detioriate. the idea behind the template generation was to convert propositions (stated as assertions) into the correct dialogue acts, i.e. the act type observed in the original naturally-occuring snippet from which the generated snipped was derived via dpcr. a qualitative analysis of the dialogue snippets generated with dpcr -templ and dpcr revealed coherence score differences where the snippets begin with a speaker that uttered a questioning followed by an asserting. this suggests that there is scope to improve the relevant sentence templates for this situation. for example: assertive questioning: do you believe that the uk should remain in the eu if a hard brexit is the only alternative option? followed by: asserting: in other words i think that by remaining in the eu, the uk would be able to operate broadly as before but with clear caveats regarding some issues that concern its citizens. in cases like this, a dialogue generated with the dpcr -templ algorithm can result in dialogue that is perceived as more coherent. for example: the uk should remain in the eu if a hard brexit is the only alternative option. as illustrated above, the dpcr -templ algorithm does not make use of the illocutionary connections (ics) to rephrase the argument map claims that are used. it will not convert the argument map claim to the sentence type associated with the ic and instead always use the declarative sentence from the argument map verbatim. however, for those locutions where there is no argument map claim involved, such as various forms of challenging, dpcr -templ does use the relevant canned text: for example a pure challenging ic is realised as one of the following 'why is that?' or 'why?'. similarly, there is canned text for assertive and rhetorical challenging. this means that dpcr -templ will not just yield a sequence of assertions: for all the aforementioned ics involving challenging, variety is introduced through the canned text associated with these ics. all in all this means that dpcr -templ is at least partially iat-compliant after all. as part of our qualitative analysis we also observed that the dialogue generated with dpcr does look more like natural dialogue, than the ones generated with the dpcr -templ algorithm. for an example of the contrast between the two types of dialogue that are generated, see appendix g starting on page 19: tables 18 and 19 . note the difference between the dpcr dialogue, which involves for example questions and hedges and dpcr -templ dialogue, which is a simple sequence of assertions. it may be that naturalness needs to be considered separately from coherence, which was the focus of this paper. whereas we found evidence for the relation between propositional relation choice and coherence, this relation does not seem to be as strong or existent between dialogue act type choice and coherence. further research is needed to establish whether the latter relation is more closely associated with dialogue naturalness. in our related work section we discuss our decision to limit the scope of the current work to short dialogue snippets rather than full-length dialogues. in our view, widening the scope to full-length dialogues requires a synthesis of theories -such as iat with rst, sdrt or qud, or elements of theories going further back presented in the work of e.g. @xcite or @xcite -based on a thorough understanding of what makes each effective. here, we would like to emphasize that our experimental method and the results of its application to iat are intended as a contribution to such a long-term research programme. we consider the current study large scale, with almost 1000 dialogue snippets judged by 87 annotators. of course, size is relative and compared to datasets and annotations undertaken by commercially-driven labs (e.g. to train llms), our dataset is comparatively small. and yet, for theory-driven empirical work, this study is of a significant size. it is also worth noting that with this paper we are making all data ( https://doi.org/ 10.21954/ou.rd.30161896 ) and code ( https:// doi.org/10.21954/ou.rd.30161926 ) available.