Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts? (2024)

Abstract: Authorship verification is the task of determining if two distinct writing samples share the same author and is typically concerned with the attribution of written text. In this paper, we explore the attribution of transcribed speech, which poses novel challenges. The main challenge is that many stylistic features, such as punctuation and capitalization, are not informative in this setting. On the other hand, transcribed speech exhibits other patterns, such as filler words and backchannels (e.g., um, uh-huh), which may be characteristic of different speakers. We propose a new benchmark for speaker attribution focused on human-transcribed conversational speech transcripts. To limit spurious associations of speakers with topic, we employ both conversation prompts and speakers participating in the same conversation to construct verification trials of varying difficulties. We establish the state of the art on this new benchmark by comparing a suite of neural and non-neural baselines, finding that although written text attribution models achieve surprisingly good performance in certain settings, they perform markedly worse as conversational topic is increasingly controlled. We present analyses of the impact of transcription style on performance as well as the ability of fine-tuning on speech transcripts to improve performance.1

## introduction identifying individuals based on their language use can be attempted using speech (speaker recognition) or writing (authorship attribution). traditionally, speech data have been analyzed using phonetic features, such as pitch and articulation rate, or embeddings related to these features, such as x-vectors @xcite , with broader linguistic information about the content of what is said playing little role @xcite . textual data, by contrast, have been analyzed historically via lexical, syntactic, semantic, and stylistic features @xcite and more recently via textual embeddings from neural networks @xcite . there are a number of motivations for exploring the use of textual authorship attribution methods on transcribed speech. in challenging acoustic settings, such as with degraded audio or a disguised voice, speech content and syntax may be the only reliable signal available. in other cases, the original audio may no longer be available, leaving only a textual version of the speech. this is common with transcripts of interviews, court proceedings, and in commercial settings where text is preferred for archiving. developing reliable identification of a speaker based on transcripts of their speech may also expose potential blind spots in current speaker anonymization approaches, which alter the speech signal but leave the speech content intact @xcite . given ever-increasing spoken media (such as podcasts) and social communication as well as the growing popularity of automatic transcription systems (e.g., descript for podcasters, otter for meetings), there will be a need for understanding this domain and being able to accurately identify speakers, especially in forensic settings, in the future. furthermore, understanding the generality of text attribution methods for transferring to not only new domains, but also new modalities (textual representations of auditory input) may indicate that these methods use deeper linguistic features rather than mainly relying on surface features, such as punctuation and capitalization. despite the potential utility of distinguishing speakers via transcribed speech, there are a number of challenges. one that applies to both transcript-based and text-based attribution is the need for substantial writing samples to characterize style, which, in the speech domain, means a large number of utterances. also, since transcribed speech is both a different domain and a different modality, foot_0 text-based attribution models have more of a gulf to bridge than in a standard cross-domain task. for example, while written text contains features such as punctuation, spelling, and use of emoticons that can vary systematically across individuals, transcribed speech may lack these particular stylistic cues. instead, speech contains other potentially identifying features, such as filler words (e.g., um), backchannels (e.g., uh-huh), and other discourse markers (e.g., well, i mean) @xcite , and, more rarely, indications of pause length or intonation. finally, working with transcribed speech, much like working with translated language, introduces noise into the system via the transcription or translation process. the present work contributes a benchmark for text-based authorship attribution models on human-transcribed conversational speech transcripts. using the task of verification-i.e., determining if two transcripts have the same speaker or different speakers-we perform a systematic comparison of existing methods and provide a detailed analysis of the results. this work focuses on the following research questions. first, despite the difference in modality, do textual authorship models transfer to speech transcripts? once we have established this benchmark, we probe other aspects of the task to help determine the limits of authorship models applied to transcripts. since speech is transcribed based on a transcription style, what is the impact of transcription style on attribution performance? as alluded to above, many state-of-the-art authorship models focus on features such as capitalization that can be erased or standardized with transcription. additionally, many models rely on topic as a clue for attribution, so to what extent does controlling for topic make the task harder? next, does fine-tuning on speech transcripts significantly improve performance, and then, does further pre-training on speech transcripts improve it more? finally, since in many forensic settings there are often limited data, how many utterances are required to achieve a given level of verification performance? by addressing these questions we provide a proof of concept for the viability of applying authorship models to speech transcripts. ## related work early work in speaker identification for text used special-purpose models for novels that match utterances to the character who 'said' them @xcite . the first work we are aware of to perform attribution on speech transcripts is a study that looks at word frequency to distinguish 250 speakers of dutch @xcite . in contrast, we consider a larger set of speakers and compare a range of different methods, both neural and non-neural. a contemporaneous related task is the pan 2023 competition, which looked at cross-discourse type authorship verification between essays, emails, interviews, and speech transcripts @xcite . they perform a weaker version of topic control, though, replacing named entities with generic tags. recently, tripto et al. ( 2023 ) compare a number of statistical and neural authorship models on human spoken texts and large language models prompted to emulate spoken texts, finding that even simple n-gram-based authorship models can perform well on speech transcripts. however, we present contradictory results in this work, finding that most text-based authorship models have almost no predictive power once topic is controlled. we conduct further experiments that tease apart which factors influence performance, such as the transcription style and the number of utterances in the transcript. the effect of conversational topic (and text genre) on text-based authorship attribution performance has been studied particularly in (forensic) stylometry to address cases of a domain mismatch between the texts of unknown authorship that are under investigation (test data) and the available comparison texts of known authorship (training data) @xcite . for instance, an anonymous social media post might have to be compared to the news articles and blog posts of potential authors. these stylometric studies often focus on small amounts of data and/or few candidate authors, such as manually elicited data @xcite or select literary authors @xcite . authorship studies on larger amounts of data across many authors use either no topic control or approximate topic control through domain labels, such as categories of amazon product reviews @xcite and subreddits of reddit @xcite . although it is unclear how representative domain labels are of various topics, studies that implement some version of topic control generally find that performance decreases as topic control increases @xcite . ## a speaker attribution benchmark to compare performance across the range of intended conditions, we focus on one data set that fit all our requirements, namely, large enough in number of speakers, number of utterances per speaker, and number of conversations to allow fine-tuning. we also focus on gold standard transcriptions since our aim is to establish an initial benchmark of performance under ideal conditions that can be used as a reference point for future experiments on noisier data. in fact, experimenting on the noisier output of automatic transcribers is out-of-scope for this paper but is an important next step in future work. to be able to adjust the difficulty level of the verification task to obtain a range of performance for the models, it was crucial that speakers discuss fixed topics, which would allow matching transcripts not only based on speaker, but also on the content of what is discussed. finally, a conversational datas et aligns with many likely use cases of speaker attribution, especially in the forensic setting. we chose the fisher english training speech transcripts corpus @xcite , a collection of human-transcribed phone calls, due to its accessibility, size, manual error correction, conversational topic assignments, and gender balance among speakers. the corpus consists of 11,699 transcribed phone calls totalling 1,960 hours. participants on calls generally did not know each other and had an assigned discussion topic that was randomly selected from a list. in general, participants stayed on topic throughout the duration of the call @xcite ), which we confirmed through a manual check of a random sample of the transcripts. the calls lasted 10 minutes and speakers often participated in multiple calls. the fisher corpus contains two transcript 'encodings', or annotation styles. one was manually transcribed using wordwave quick transcription with error corrections and post-processing by bbn technologies. this 'bbn' encoding includes prescriptive punctuation and capitalization according to an existing wordwave style guide @xcite . the linguistic data consortium (ldc) provided the second encoding, with automatic segmentation of the audio data and manual transcription of the words, including a basic spell check @xcite . this transcription did not include punctuation (other than apostrophes and hyphens), put text in all lowercase, and often grouped together text by the same speaker despite interjected backchannels. comparing performance on each encoding can help elucidate the extent to which the models capture 'deeper' authorship features rather than surface-level low-hanging fruit. figure 1 presents an example of each. both encodings include non-speech sounds, such as laughing and undistinguished noise, in square brackets. the ldc encoding employs double parentheses for unclear productions that were guessed by the transcriber. we select the same transcripts from both encodings to test the impact of annotation on a model's attribution performance, according to the following procedure. first, we use a 50/25/25 training/validation/test split with no overlap in speakers between training and evaluation splits, making the task more challenging. for the speakers in each set, verification trials are formed by matching two transcripts from either the same speaker ('positive') or different speakers ('negative'). transcripts have ∼1400 tokens across an average of 100 utterances. in section 4.6 we vary the number of utterances used from each transcript; the results revealed that the appearance of names during introductions at the beginning of the call significantly helped model performance; as a result, we remove the first five utterances of each transcript for all experiments. 3 we create three different datasets according to level of difficulty by controlling (or not) for topic to the extent possible. the 'base' level does not have any restrictions on topic: positive trials have one speaker on different calls while in negative trials, the speaker and call are different. the 'hard' level introduces some topic control: positive trials consist of two transcripts from the same speaker in different calls in which the assigned discussion topic is different, and negative trials contain two transcripts from different speakers in calls in which the assigned topic is the same. the 'harder' level contains the same positive trials as the 'hard' level, but the negative trials are further restricted by only pairing speakers on the same call, so not only is the assigned topic the same, but the content within that topic also matches. in other words, the 'hard' level is a rough measure of topic given that a number of subtopics can be discussed in ten-minute conversations, whereas the 'harder' level is a more reliable measure of topic given that two people in the same call will cover the same range of subtopics. as a rough computational estimate of topic, running the authorship models on these datasets thus tests their ability to look beyond simplistic proxies for content and to utilize more structural features used by the speaker. for each difficulty level, the training set has ∼3300 verification trials, the validation set ∼1700, and the test set ∼1800 on average. the number of positive and negative verification trials per difficulty level is roughly balanced, cf. the final model we consider is the recurrent neural network model adhominem foot_4 @xcite , which performed well on speech transcripts in recent work @xcite . this model uses a hierarchical architecture to aggregate character, word, and sentence features from each document. for adhominem, we converted the transcript trials into the appropriate format and then saved a model checkpoint, whose weights were restored for extracting features from the utterances in each trial. for reference, we also include two baselines: tf-idf, a weighted measure of word overlap, and pan's authorship verification baseline of tf-idf-weighted character 4-grams (pangrams) foot_5 (stamatatos et al., 2023). 9 in line with the research questions in section 1, we set up the experiments as follows. section 4.1 creates a performance benchmark of the aforementioned models evaluated out-of-the-box on speech transcripts. section 4.2 tests these out-of-the-box models on both the bbn and ldc encodings to determine the effect of transcription style on performance. this includes an additional comparison between the default luar pre-trained on reddit and an instantiation of luar pre-trained on reddit that has been normalized to look like the ldc data. section 4.3 varies the difficulty level of the task ('base', 'hard', 'harder') by controlling for the topic discussed across the verification trials. section 4.4 adds a step of fine-tuning on each training set difficulty level and then evaluates the models on each difficulty level of the test set. as a first look into the impact of pre-training domain on performance, section 4.5 tests the best performing model, luar, pre-trained on the speech transcripts themselves. finally, section 4.6 varies the number of utterances used in each transcript from 25 to the full transcript.