Policy-focused Stance Detection in Parliamentary Debate Speeches (2022)

Abstract: Legislative debate transcripts provide citizens with information about the activities of their elected representatives, but are difficult for people to process. We propose the novel task of policy-focused stance detection, in which both the policy proposals under debate and the position of the speakers towards those proposals are identified. We adapt a previously existing dataset to include manual annotations of policy preferences, an established schema from political science. We evaluate a range of approaches to the automatic classification of policy preferences and speech sentiment polarity, including transformer-based text representations and a multi-task learning paradigm. We find that it is possible to identify the policies under discussion using features derived from the speeches, and that incorporating motion-dependent debate modelling, previously used to classify speech sentiment, also improves performance in the classification of policy preferences. We analyse the output of the best performing system, finding that discriminating features for the task are highly domain-specific, and that speeches that address policy preferences proposed by members of the same party can be among the most difficult to predict.

## introduction transcripts of legislative debates provide access to information concerning the policies that are publicly supported or opposed by politicians. they are of interest to political scientists, the media, the politicians themselves, and citizens who wish to monitor the activities of their representatives. however, such documents are complex and difficult for people to process. transcripts of debates in the united kingdom (uk) parliament are so hard for ordinary people to make sense of that parliamentary monitoring website www.theyworkforyou.com publishes manually annotated versions of the transcripts. these include crowd-sourced explanations of the debated proposals, as well as policy-focused aggregations of the voting records of parliamentarians. the large quantity and esoteric nature of the data in the parliamentary record (known as hansard) motivates the need for automatic analysis of its contents. previous work in the domain of legislative debate transcripts has focused on either (a) sentiment polarity classification @xcite @xcite , or (b) policy identification @xcite in isolation. as far as we are aware, these two tasks have not previously been combined in this do-main, despite the fact that: (1) the information yielded is complementary, and perhaps even necessary, for practical use (i.e., without analysis of debated policies, the target of sentiment in the speeches is unknown); and (2) these two tasks rely on features derived from shared information, which could assist with the learning of parameters for both tasks in a multi-task learning setting. borrowing the concept of policy preferences from political science, we compare approaches to automatically determining the policy preference that is under discussion in each debate, and whether each speaker supports or opposes it. our contributions building on the work of @xcite ; abercrombie and batista-navarro (2020), we combine policy preference identification and speech-level sentiment polarity analysis to formulate the task of policy-focused speech stance detection for the domain of legislative debate speeches, in which the position of each speaker in a debate is identified in relation to the proposal under discussion. unlike prior work, we thus obtain interpretable analysis of the positions taken by mps with respect to the policies presented in parliamentary debates. to this end, we add a set of manually annotated policy preference labels to a large existing english language corpus of uk parliamentary debates, creating the first dataset to be labelled with both topics (policy preferences) and positions (sentiment) in this domain. we make the enhanced corpus available to the research community. we use this dataset for the evaluation of approaches to the classification of policy-focused speaker stance. we test classification systems comprising combinations of single-and multi-task learning paradigms, different debate structure models, and varying approaches to text representation and machine learning methods. our results represent initial benchmarks for this task. ## related work sentiment classification is one of the the most active areas of research in natural language processing. within the domain of legislative debates, examples include classification of speeches from the us congress @xcite @xcite , and the uk parliament (abercrombie and batista-navarro, 2018b, 2020; @xcite @xcite . in these works-and in common with ours-speaker sentiment is assumed to be analogous to vote outcome. however, in the task undertaken in these previous works, the nature of the targets-the bills or motions under debateis not identified. the related task of stance detection-in which the target of sentiment is (pre-)determined-has been applied to such domains as social media (e.g. @xcite @xcite , online debate forums (e.g. @xcite @xcite , and news articles @xcite . for a recent survey, see @xcite . in most of this work the target is pre-chosen by the user or the system. in the political domain, this has been framed as agreement detection in which two pieces of text are compared @xcite , or classification of support or attack towards pre-defined policies @xcite . while vamvas and sennrich (2020) carry out stance detection on the positions expressed by swiss politicians, they do not perform automatic identification of the policies discussed, only conducting binary in favour/against classi-fication in a similar vein to the sentiment/position classification work discussed above. more similarly to this work, bar-haim et al. (2017) used a supervised approach to identify both the stances of extracts from wikipedia articles and the targets of those stances from a closed list of 'controversial topics'. however, this labelling scheme does not cover the policy positions proposed in parliament. a common framework for stance detection is the sdqc (support-deny-query-comment) annotation scheme of @xcite . while potentially suitable for our data (support and deny are equivalent to our support and oppose labels), application of this framework would require manual annoation of each instance in the dataset with the more fine-grained labels. instead, we follow the majority of work on legislative debates (e.g. @xcite @xcite in taking advantage of pre-existing vote-derived binary labels at the speech level, and thus only requiring the addition of policy preference labels for each debate. in most of the reviewed work, stance targets are explicitly selected by the authors of the task (e.g. donald trump @xcite , richard nixon and john f. kennedy @xcite , or atheism (mohammad et al., 2016)). unlike these, we frame target selection as a multiclass topic classification problem, making use of an existing schema validated by political scientists. document classification is an active area of research for tasks such as identification of news and wikipedia categories @xcite . for classification of hoc debates, @xcite used 'policy' labels crowdsourced by the parliamentary monitoring website https://www . publicwhip.org.uk/ but found this framework limited as it could not be easily scaled up from the small existing labelled dataset. @xcite created a manually annotated dataset of policy preferences in debate motions, and achieved promising results in classifying debate motions according to the marpor coding scheme. however, this corpus is unsuitable for our purposes as: (1) it does not include speeches made in response to the motions; and (2) the motions in this dataset are all substantive-that is, they 'express an opinion about something' @xcite , and tend to be of a highly partisan nature, leading to debates in which the stance of mps can be trivially predicted from their party affiliations. for this study, we seek a mixture of motion types, more represenative of the hansard record as a whole. additionally, while they classified debate motions with policy preference labels using textual features derived from the motions themselves, many of the motions in hansard-and in the corpus used in this study-contain little in the way of informative textual content (figure 1a is a typical example). rather than the motions, we therefore rely on features derived from the response speeches, which we use as input for the classification of both motions and speeches. multi-task learning approaches have been taken to many tasks, including part-of-speech tagging, chunking, and named entity recognition @xcite . while such approaches have been applied to sentiment classification of customer reviews (yu and jiang, 2016), we are not aware of any uses of multi-task learning in the legislative debate domain. the most common approach to multi-task learning-which we compare with the single task paradigm-is that of hard parameter sharing, first proposed by @xcite . ## data parlvote (abercrombie and batista-navarro, 2020) is a large corpus (34,010 examples) of hoc debate speeches made between from 1997 and 2019. each example speech consists of the concatenated utterances of an individual speaker in a given debate, and is presented with the debate motion to which it responds, as well as the vote of the speaker (in support or opposition to the motion), and metadata associated with the debate and the speakers. we adapted this corpus to include an additional, manually annotated policy preference label for each example. as capitalization can be informative in this domain (for example in the terms of address 'friend', 'lady', 'gentleman'), we did not lowercase the text. ## method we investigate approaches to determining, for each example in the dataset, (a) the policy preference expressed in the debate motion, and (b) the sentiment (position) expressed in the speech towards that motion: support (positive) or oppose (negative). we compare the performance of systems comprised of combinations of the following: shared input layer multiclass hidden layer binary hidden layer policy preference output speaker sentiment output speeches shared hidden layer multi-task input layer a input layer b hidden layer a hidden layer b multiclass hidden layer binary hidden layer policy preference output speaker sentiment output single task policy preference: support/oppose speeches speaker sentiment output policy preference: support/oppose batista-navarro, 2018a)). aside from not lowercasing the text, we used the default settings from scikit-learn to tokenize and extract ti-idf features from the texts. foot_0 we use google's bert tokenizer, foot_1 and pad the texts to the maximum input of 512 tokens, then fine-tune the top 3 layers of the bert model. the (fine-tuned) final layer of bert embeddings is then used as input to one of the following neural classifiers. we used a randomly sampled 80/10/10 split of the data. the experiments can be reproduced using our python notebook, which we make available with all code and data at https://tinyurl.com/y62jrkyt . ## results we evaluated the systems described above against the majority class for each task. slight differences in these baseline scores in the motion-dependent and independent settings arise from variations in the class distributions in the test sets in these settings. due to the class imbalances in the dataset, we report the macroweighted f1 score as the evaluation metric. ## discussion and conclusion policy-focused stance detection of parliamentary speeches is a challenging task, which we have framed as combined binary and multiclass classification. for this, we enhanced an existing dataset with an additional set of policy preference labels. while inter-annotator agreement on policy preference labels is modest, it is similar to that reported in previous work on both parliamentary debates and election manifestos. to address the issue of low annotator agreement, and the fact that classifiers frequently misclassify speeches across policy domains, future work could take a perspectivist approach to annotator disagreement @xcite , and consider reframing the task as a multiclass and multilabel problem, in which more than one policy preference code may be valid per speech. notwithstanding this issue, and despite the large number of classes in the policy prediction task, and the fact that the input features we used were based only on the content of speeches (not the motions or titles, as in previous work @xcite )), we have been able to obtain reasonable results, comfortably beating the majority class baselines. modelling of the structure of parliamentary debates in the form of motion-dependent classification was seen to improve performance on speech sentiment classification in prior work. in this study, we found that it is not only consistently superior for speech sentiment classification, but also improves the identification of policy preferences, the topics under discussion. we have shown that the differences between supportive and opposing speeches do not derive from generally sentiment bearing words, but from the relationships between the speaker, the mp who proposes the motion in question, and the party affiliations of both actors. the application of multi-task learning did not, in most configurations, improve system performances. however, we used a fairly simple framework in which just one of the network's hidden layers was shared with one further hidden layer per classification task. there is therefore plenty of scope for further experimentation with more complex architectures for this approach. in these experiments, fine-tuning on bert embeddings led to considerably worse performances. considering the widespread successes of this approach, this also warrants further investigation. with recent work suggesting that, for real-world tasks and datasets, pretraining the embeddings on in-domain data may be necessary @xcite , a more domain-specific approach may be desirable. while other work on sentiment and stance detection in the domain of parliamentary debates has effectively overlooked the targets of those opinions, we have combined approaches to sentiment and topic detection to formulate a task with potential for real-world application. although there remains much room for improvement in classification performance, we have shown that the task of policy-focused speech stance detection can be feasibly automated, even with simple features and neural architectures. although we have focussed our annotation effort and analysis on debates from the uk parliament, the proposed approach is generalisable to other legislatures, or indeed any political debates that feature proposed motions and supporting and opposing documents. in future work, we will focus on refining the annotation scheme in order to obtain greater labelling consistency and improved classification performance, as well as adapting the methods for the legislative debate domain. yu, jianfei and jing jiang. 2016. learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 236-246, austin, texas. association for computational linguistics. zhang, xiang, junbo zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in c. cortes, n. d. lawrence, d. d. lee, m. sugiyama, and r. garnett, editors, advances in neural information processing systems 28, pages 649-657. curran associates, inc. zubiaga, arkaitz, maria liakata, rob procter, geraldine wong sak hoi, and peter tolmie. 2016. analysing how people orient to and spread rumours in social media by looking at conversational threads. plos one, 11(3):1-29.