Who said what? Speaker Identification from Anonymous Minutes of Meetings (2023)

Abstract: We study the performance of machine learning techniques to the problem of identifying speakers at meetings from anonymous minutes issued afterwards. The data comes from board meetings of Sveriges Riksbank (Sweden’s Central Bank). The data is split in two ways, one where each reported contribution to the discussion is treated as a data point, and another where all contributions from a single speaker have been aggregated. Using interpretable models we find that lexical features and topic models generated from speeches held by the board members outside of board meetings are good predictors of speaker identity. Combining topic models with other features gives prediction accuracies close to 80% on aggregated data, though there is still a sizeable gap in performance compared to a not easily interpreted BERT-based transformer model that we offer as a benchmark.

## introduction attributing a text or a part thereof to an agent is a well-established sub-field of computational linguistics. apart from the traditional task of author attribution, it has also been applied in social media studies, to the identification of speakers in fiction dialogues, and for detection of plagiarism. in this work, we study a new but related problem: identifying speakers at meetings from anonymous minutes issued afterwards. the data at hand are minutes, in swedish, from the monetary policy meetings of the riksbank's executive board. the main monetary policy objective is to keep inflation low and stable, close to the target of 2 percent. the key issue at the meetings is to decide on the policy rate, and, since the global financial crisis in 2007-2009, on purchases of financial assets. minutes from meetings like these are not only common for central banks but also, for instance, corporates, c.f. @xcite . until june 2007 the minutes of the swedish riksbank's monetary policy meetings gave an anonymised account of the deliberations. since then, however, the identity of a board member is revealed in the minutes so that it is possible to know which member expressed which opinion during the meeting. this change towards increased transparency is of great interest to researchers on economic policy-making and there is a growing literature in this area @xcite . it could potentially affect board members' incentives and behaviour in different ways, not least because the minutes are published only around two weeks after a meeting. following the theoretical literature increased transparency can have different effects. it can make agents prepare more thoroughly -a disciplinary effect @xcite . it can also make agents behave differently due to career concerns, either by making them less inclined to oppose to the majority view -a herding, or conformism, mechanism -or by making them instead want to distinguish themselves more from others -an anti-herding or exaggeration mechanism. it may also make agents more committed to stick to a specific opinion once they have expressed it and less willing to change their mind, even if circumstances change @xcite . ## related work we have not been able to find studies that perform speaker attribution under equal circumstances. a study on cabinet meetings @xcite had the goal of annotating all sentences of cabinet protocols with its speaker. they used a rulebased approach exploiting properties of german morphology. speaker attribution of sentences has also been studied on dialogues in literature, where again the task is to annotate sentences or utterances with speaker information, where this is not explicit. an example is @xcite who applied supervised machine learning to the task. we do the same in this study but the genre is different and our data points are usually much longer than a single sentence. still, the task has similarities with closed-class author attribution. a taxonomy of six feature categories has been proposed for this task by stamatatos (2009): character, lexical, syntactic, structural, semantic, and application-specific. the first two types have the advantage that they can be computed with very little analysis of the text; they include frequency counts of function words, punctuation marks, and short ngrams. syntactic features can refer to part-of-speech tags or ngrams of these. structural features include word length and sentence length as well as layout features. features requiring detailed analysis of texts, such as full syntactic parsing and topic modelling have also been used. @xcite used dependency parsing as well as morphological and syntactic features, while @xcite employed topic modelling as a basis for feature selection. @xcite showed how variants of topic modelling can be used to predict authorship and @xcite used topic modelling to analyse which features are effective under different conditions, showing content-based features to be more effective when the diversity of topics in the document set is more varied. given a set of selected features that can be used for profiling documents as well as authors, a method is needed to decide among the authors for a given document. well-known methods based on a selection of frequent words are chi-square distance @xcite ), burrows's delta @xcite , and kullback-leibler distance @xcite . all of these compute a distance metric where the author model with the smallest distance to the document model is proposed as the most likely author. among machine learning methods k-nearest neighbours and support vector machines have been tried, often with good results. neural methods have also been applied, sometimes with mixed results. the best overall system at the pan-2015 author identification task was a character-level rnn language model @xcite , while the neural systems at the cross-domain author identification task at pan-2018 did not compete well @xcite . most systems at that event used svms while the best system was an ensemble system, combining features of three kinds with logistic regression. more recently, there have been a few examples of author attribution in which the transformer architecture @xcite , which does automatic feature extraction, has been utilised. for example, @xcite introduced bertaa, a fine-tuned bert language model for authorship classification. in experiments, the pre-trained model was fine-tuned on three different datasets in the domains of emails, blogs and movie reviews, respectively. state-of-the-art performance was obtained on all three datasets either with plain bertaa or with bertaa with additional features. ## data and preprocessing the data collected at the riksbank have two main sections: minutes from monetary policy meetings and public speeches given by executive board members. the minutes are from two periods: one batch starting in february 2000 and ending in may 2007, and another beginning in june 2007 and ending in april 2018. minutes from the earlier period are truly anonymous, while the minutes from the later period have been anonymised for the purposes of this study. an overview of the data can be found in the speeches have been collected during a somewhat longer period, from 1997 forward. the speeches mostly address the current economic situation and are addressed to a variety of audiences such as banks, regional authorities, chambers of commerce, and parliamentarians. both minutes and speeches were originally in either doc-or pdf-format. texts were extracted from the pdf-files using the apache tika parser foot_0 accessed via a python port foot_1 . from the minutes we then used regular expressions to remove data that was not text such as multiple empty lines, page headers, pagination and table cell data. the outline of the minutes has changed over the years but is typically divided into four numbered sections. some minutes have less than four sections and a few of them have more. each section is supplied with a heading that starts with an initial ' §'-sign. the contributions are found in a separate section with a heading such as penningpolitisk diskussion, 'discussion on monetary policies' or just diskussion. this section is the one from which we extract contributions for the experiments. a contribution from a board member in the minutes is as a rule introduced with the member's title, e.g., förste vice riksbankchef, 'first deputy governor' and full name. all text following this introductory phrase and lasting until a new introduction of the same type is encountered has been allocated to a single contribution. a member may speak at a meeting on several occasions and so we have also collected these together as aggregated contributions. the total number of individual contributions is 900, and the aggregated contributions amount to 385. in most meetings six members including the governor are present. there are a few meetings with fewer members present. it does not happen that a member does not contribute to the discussion at all. some members have been present at the majority of meetings, others at only a few e.g., because their period as director ended. the minutes and the speeches have all been parsed by the sparv parser @xcite . the information obtained from sparv includes lemmas, part-of-speech tags and word senses, which we have used in subsequent processing. the speeches, all in edited written form, are known to be given by certain members. all text of a speech, with the exception of some metadata information supplied in the header, has been kept. the main processing of the speeches is word based (frequency counts, topic modelling) and for this reason, we did not clean them to the same extent as the minutes. for fine-tuning the pre-trained bert model, we used the raw texts from the minutes as data (the speeches were not used in this setting), masking titles, names and gendered pronouns. the masking was done assuming such information could steer the model towards certain predictions, trivialising the task and hampering generalisation to the truly anonymous setting where this information is absent. for both the individual and the aggregated data, the length of the contributions varies significantly. as seen in ## features used in the experiments we have framed our problem as a closed set classification task and applied a number of different methods. burrows' delta uses lexical features, which are detailed below, in section 5.1, and the bert model uses its own feature selection. however, for the svm and mlp models, we have investigated various properties with the potential to differentiate between members. for each of the properties, one or more features were defined. the focus is on properties and features that relate to content and application. in the rest of this section we motivate the choice of features. ## experiments this section elaborates on the details of the different systems and their performance. all results are shown in ## results the results for classification accuracy of all tested systems are presented in as expected, the aggregated contributions score higher than the individual contributions. classifying aggregated (and longer) contributions are naturally a less complex problem compared to individual contributions due to the reduced number of classifications to be performed per meeting. it should however be noted that the tested systems, especially the system based on bert, are able to handle this change of scope in an acceptable manner considering the increased task complexity. furthermore, the results indicate that the svm and mlp classification methods performed significantly better than the random baseline and that the differences between these methods were relatively small. when including all the features listed in the fine-tuned bert model obtained 94.81% accuracy for all folds combined on the aggregated data, and an accuracy of 83.78% on the individual data. since the data used for the bert model had to be split into smaller chunks to fit the input limit of bert, we tried the same chunking approach for all the non-bert systems. since some of the aggregated contributions were fairly long (see ## conclusions and discussion in this work, the main purpose has been to investigate a set of interpretable features for identifying speakers from minutes. with the aid of feature selection algorithms, we are able to pin down the most important features, while also excluding some of the less relevant, and simultaneously improve the classification performance. topic models generated from speeches given by directors of the board of the riksbank turned out to be a good predictor of what they say in board meetings. combined with other features such as wordiness, speaking order and sentiment analysis we could reach an accuracy close to 80% in predicting which director said what. not surprisingly the fine-tuned bert model has the best performance in predicting which board member made a certain contribution. this is in line with the performance of similar models in other attribution tasks (cf. @xcite ) and points to transformer-based models being good feature extractors. while we have only investigated one corpus of minutes, the methods we've tried have a wider application; similar types of meetings and minutes are common in financial and other public institutions where transparency and accountability is an issue. the success of the bert-model suggests that members are consistent in their argumentation across meetings. an interesting aspect is the fact that the minutes of the meetings are not written by the members, which should make this task harder than standard author attribution. given this, we find that the bert model provides a strong benchmark for de-anonymisation of minutes. the bert-model, unlike the features used for the other models, is not easily interpretable. yet, as new techniques for interpreting models such as bert are emerging, we would like to investigate what the bert-model actually considers when making predictions. for example, whether it looks at stylistic features in how the minute taker writes about a particular member or features more related to the content and topics of the contributions. the properties, coupled with analysis of the overall differences of the minutes under the two conditions, are likely to be helpful in future research on de-anonymising the minutes from the earlier period. although members are not referred to by name there is a similar structure to the minutes and the discussions so that contributions can be identified. while performance may be lower for all models when applied to minutes for the earlier period, the data obtained from the non-anonymous minutes could then be used for training. for example, we know from confusion matrices which speaker models are often confused. there are features we have not yet investigated such as members' style of argumentation, or rhetorical structure, which potentially could be helpful. as we now also have identified the topics discussed during the meetings, we can analyse members' attitudes, i.e. sentiments, towards each topic. this can also be included in our model. an analysis of the parse trees of contributions could also yield features at a more fine-grained level than topics, such as individual members' hobby-horses.