Actor Identification in Discourse: A Challenge for LLM s? (2024)

Abstract: The identification of political actors who put forward claims in public debate is a crucial step in the construction of discourse networks, which are helpful to analyze societal debates. Actor identification is, however, rather challenging: Often, the locally mentioned speaker of a claim is only a pronoun (“He proposed that [claim]”), so recovering the canonical actor name requires discourse understanding. We compare a traditional pipeline of dedicated NLP components (similar to those applied to the related task of coreference) with a LLM, which appears a good match for this generation task. Evaluating on a corpus of German actors in newspaper reports, we find surprisingly that the LLM performs worse. Further analysis reveals that the LLM is very good at identifying the right reference, but struggles to generate the correct canonical form. This points to an underlying issue in LLMs with controlling generated output. Indeed, a hybrid model combining the LLM with a classifier to normalize its output substantially outperforms both initial models.

## introduction political decision-making in democracies is generally preceded by political debates taking place in parliamentary forums (committees, plenary debates) or different public spheres (e.g., newspapers, television, social media). one way in which political scientists have analyzed such processes is to adopt the framework of political claims analysis @xcite , identifying the claims (i.e., calls for or against specific courses of action) and actors involved in a given debate. actors, claims, and the relations between them can then be represented as bipartite discourse networks @xcite , such as shown in figure 1 . such networks permit researchers to investigate debates on a fine-grained level, identifying, e.g., discourse coalitions, decision makers, or argumentative clusters. while early work on discourse networks was based on manual analysis, widespread use of discourse networks requires quick, ideally automatic, methods to construct them from text. this calls for nlp methods to (1) detect instances of claims, assign them to their categories (c i in figure 1 ), and (2) identify actors for these claims in terms of some canonical representation (a i ), cf. @xcite . at least for newswire, there are several nlp models for claim detection and categorization @xcite . in contrast, there is little work on actor identification. arguably, this is because claims are easier to handle: both detection and categorization are sentencelevel classification tasks which can be modeled based on predominantly sentence-internal features. in contrast, actor identification calls for a substantial amount of discourse understanding: models must locally identify an actor for the claim, but since these are often just a pronoun or a definite description (cf. this paper asks whether this situation has improved with the emergence of prompt-based llms @xcite and their promise for text-totext generation, which appears to be a good match for the actor identification task. we contrast an llm-based architecture with a traditionally trained local mention of actor canonical version 1 president joe biden pleaded with republicans . . . joe biden 2 biden signaled a willingness to make significant changes . . . joe biden 3 "we can't let putin win", he said. joe biden 4 however, senate republicans later on wednesday blocked . . . senate republicans 5 a u.s. official said washington had less than $1b . . . u.s. official ## results and analysis main results. we form two non-mutually exclusive hypotheses for this performance gap: either that the traditional model, through its supervised training, came to be more competent at predicting the correct political actor, or, through virtue of its inductive biases, it came to better and predicting the exact canonical name. we examine these hypotheses by evaluating the model with the other two settings. we also carry out a qualitative analysis of errors made by the llm-based model (see one simple factor that would lead an essentially correct llm to be inexact is formatting errors in its output -either mismatched spacing, punctuation, or capitalization, or natural language responses that could not be correctly post-processed. such effects should show up as a difference between the 'exact match' and the 'up to formatting' setting. however, the numbers (43.06 f 1 vs. 43.98 f 1 ) show that these types of error account for less than one percentage point. our qualitative error analysis ( the reliance of our exact evaluation metric on gold-standard canonical forms provides another opportunity for a largely correct model to show low performance due to an inability to pick the exact canonical form required. this factor should come to the fore when we compare exact match results to the 'up-to-canonicalization' setting. indeed, for this setting, both models show a substantial increase in performance -which implies that canonicalization represents a large part of the difficulty for this task. interestingly, the llm shows a much larger improvement, ultimately outperforming the traditional pipeline by about 2.5 points f 1 . our qualitative error analysis in ## conclusion in this work, we investigate alternative approaches to tackling the discourse-level actor identification task, comparing llm prompting with a conventional nlp pipeline. we find that our llm better recognize the appropriate actor entities compared to the traditional pipeline, but has a harder time controlling the exact output. this problem cannot be solved easily with tuning, as the failure of our fewshot setup shows, which is also in line with recent studies on the controllability of llm output @xcite . our solution is a hybrid model which integrates the llm-generated output as a cue in the pipeline approach, resulting in a clear improvement over the individual models. the current study is limited in several respects: it only considers one llm, one corpus, and one evaluation. in the future, we also plan to carry out an extrinsic evaluation of our actor identifier on generating full discourse networks. in terms of future directions, we believe that actor identification is a task which could plausibly profit from retrievalaugmented generation (rag) proposed by @xcite which would give the llm access to information beyond the current discourse.