Detection and attribution of quotes in F innish news media: BERT vs. rule-based approach (2023)

Abstract: We approach the problem of recognition and attribution of quotes in Finnish news media. Solving this task would create possibilities for large-scale analysis of media wrt. the presence and styles of presentation of different voices and opinions. We describe the annotation of a corpus of media texts, numbering around 1500 articles, with quote attribution and coreference information. Further, we compare two methods for automatic quote recognition: a rule-based one operating on dependency trees and a machine learning one built on top of the BERT language model. We conclude that BERT provides more promising results even with little training data, achieving 95% F-score on direct quote recognition and 84% for indirect quotes. Finally, we discuss open problems and further associated tasks, especially the necessity of resolving speaker mentions to entity references.

## introduction the recognition of quotes and reported speech is an important step towards the computational analysis of news media articles. it allows us to measure, on a large scale, who is given voice and how much, how opposing or competing views are presented alongside each other, as well as how the language of the quoted sources differs from the language of the journalistic reporting. in case of the finnish news media, such analyses have recently been attempted by @xcite . on the other hand, suomen @xcite have studied politicians' visibility in the media based on the mentions of their names. in the present paper, we focus on the technical task of recognizing direct and indirect quotes in the finnish news media texts. the task can be illustrated with the following example: sipilän mukaan lakiehdotuksia ollaan tuomassa eduskuntaan helmikuussa. according to sipilä, bill proposals will be brought to the parliament in february. such relations consists of three elements: the cue 'mukaan' ('according to') indicates an indirect quote, in which the source (juha sipilä, the finnish prime minister @xcite @xcite @xcite says the text referred to as proposition, or quotation span. 1 a complete approach for quote detection and attribution would solve the following tasks: ## related work to our knowledge, the most similar work to ours has been done by silvia pareti and colleagues @xcite @xcite , who annotated a corpus of attribution relations for english and experimented with machine learning models for recognizing such relations. for the latter they applied classification algorithms -crf, k-nn, logistic regression -working on data enriched with linguistic features, which was state-of-the art in nlp at the time. however, @xcite have criticized the choice of crfs for quote detection because of the markov assumption they make. more recently, @xcite presented a neural lstm-based model for recognizing quotations, but without attribution. @xcite compare different embedding-based models (including bert) on the task of recognizing types of speech, which includes direct and indirect quotes. as to nordic languages, a rule-based approach for norwegian has been presented by @xcite . it utilizes a dependency parser and a list of speech verbs. from among other languages, @xcite used a machine learning method on portuguese news corpora, while @xcite used a rule-based approach for multiple european languages. @xcite present a method for quote attribution. they thus start with quotation spans already recognized and perform two tasks: 1) attributing a quote to a speaker mention in the text, 2) linking the speaker mentions into entities. they use a rule-based strategy on top of tools performing dependency parsing and coreference resolution. they have also released a corpus of quote attributions consisting of three novels in english. although not dealing exactly with quote detection, @xcite provide a prominent example of computational analysis of political discourse using modern nlp methods. they use various neural models (including bert) to detect claims and attribute them to actors, with the goal of modeling the discourse as a network of relations between actors and claims. automatic quote detection could be a useful element of such a larger system as well. ## dataset and annotation the annotation process consisted of two parallel tasks: marking quotations and linking together chains of co-referencing expressions denoting people, institutions and other human-like actors present in the documents. both annotation tasks were conducted using the webanno platform (eckart de @xcite , by which each annotator was assigned their documents and by which the annotation itself was done. the annotation guidelines were written beforehand and further developed after a test run. the quotation detection annotation consisted of 1) marking the span in the text containing the content of the quote, 2) marking the speech act verb (if present), 3) marking the source of the quotation (if present), and 4) noting whether the quote was direct or indirect. the task was relatively straightforward, as all annotators were students with at least a minor degree in linguistics. the project employed 10 annotators. four of them were recruited in an earlier phase and annotated a test data set of 40 articles. after the test run, the guidelines were improved based on both inter-annotator agreement scores and feedback from the annotators, in accordance with the standard linguistic annotation methodology @xcite . the inter-annotator agreement scores (fleiss' κ) were between 0.77-0.8, which we deemed sufficient to consider the annotations consistent. the workload was balanced so that the 6 other annotators who were recruited at the later stage annotated more articles to compensate for the test run. the annotators worked independently on the webanno platform. the articles were sampled from a database containing the metadata for the online media sources and the sampled lists of articles were then scraped using a web crawler @xcite and automatically pre-processed to conll format containing lemmatization, part-of-speech and dependency taggings using turku neural parser @xcite . we used four sources for the articles: yle (the finnish national broadcasting company), helsingin sanomat (the most popular daily newspaper), iltalehti (an evening tabloid) and stt (the finnish news agency), covering different kinds of media texts wrt. length and style. the total number of articles annotated was 1500. except for the common part mentioned above, the remaining 1460 articles were assigned to one annotator each at the second stage. ## evaluation for the evaluation experiments we use a roughly 80-20 split of the data by taking the data provided by 2 annotators as evaluation set and the remaining 8 annotators as training set. the dataset sizes are summarized in quote detection. the results of quote span detection without taking into account the directindirect distinction are shown in both settings show a clear advantage of the bert model. in case of direct quotes, the rules for recognizing them are quite rigid. furthermore, they can suffer from paragraph segmentation errors and misplaced or incidental quotation marks (e.g. 'scare quotes'). this explains the lower recall of the rule-based method. indirect quotes have proven more challenging to the rule-based method as well. this can be to a variety of reasons: missing speech act verbs, incorrectly identifying quote spans based on syntactic criteria (also affected by parser, tagger and sentence segmentation errors), or uncommon structures not covered by the rules. moreover, rule 3 ('according to') has a tendency to produce false positives, e.g. something being described 'according to the plan'. in general, the bert model has shown to be more flexible wrt. the often unpredictable nature of text data, and does not suffer from the error propagation through the nlp pipeline. attribution. the evaluation of attribution is problematic because of the fact that our dataset was not annotated with the bert model in mind. thus, we present it as our best attempt given the current possibilities, but recognize the need for further work in this regard. the annotated data assigns each quote to a single token representing the mention of the quote's source in the text. if the source is represented by a longer phrase, the syntactic head (wrt. dependency parsing) of this phrase should be selected according to the annotation guidelines. on the other hand, mentions of quote sources are typically entities annotated as parts of coreference chains, and thus the entire span is marked for the purpose of coreference annotation. thus, by combining the quote and coreference annotations, we are able to obtain a span-to-span attribution relation for most cases. the exception are cases in which the quoted entity is mentioned only once in the article, and thus not annotated as a coreference chain. although the bert model outputs sources as entity spans, the rule-based model points to a single token -the syntactic head, similarly to the gold standard annotation. in order to make the results comparable, we reduced the output of the bert model to the first token of the span, and then evaluated a source annotation as correct if it either points to exactly the same token as the gold standard, or if it points to a token within the same coreference span. thus, the model's ability to correctly identify the entire span is currently not evaluated, as it is not implemented in the rule-based method. the results indicate a small advantage of the rule-based model. in both cases, the main source of errors are the unrecognized annotations, rather than the incorrect ones. for the rule-based model this is typically due to quotes not being recognized at all (see low recall in although we regard the work presented in the previous sections as a complete solution to a welldelimited problem, we see some potential for both incremental improvements, as well as work on further related tasks, that will be addressed in the future. entity annotation and detection. while designing our annotation project, we did not anticipate that a machine learning quote detection model will need to also detect entities that the quotes can be attributed to. we intended the coreference annotation to be used only in the further step (entity resolution). in result, entities that are mentioned only once were not annotated. the corpus could be improved by ensuring that at least tokens assigned as source to a quote are also annotated as an entity. this is expected to improve the bert model's performance on entity detection, and thus quote attribution. ## conclusion we have presented two methods for recognition of quotes in finnish news media, along with an annotated corpus for training and evaluation. to our knowledge, our solution is the first one proposed for finnish. we hope that the progress achieved on this task will facilitate more detailed large-scale quantitative analysis of voices in the finnish news media.