Modeling Language Usage and Listener Engagement in Podcasts (2021)

Abstract: While there is an abundance of advice to podcast creators on how to speak in ways that engage their listeners, there has been little data-driven analysis of podcasts that relates linguistic style with engagement. In this paper, we investigate how various factors – vocabulary diversity, distinctiveness, emotion, and syntax, among others – correlate with engagement, based on analysis of the creators’ written descriptions and transcripts of the audio. We build models with different textual representations, and show that the identified features are highly predictive of engagement. Our analysis tests popular wisdom about stylistic elements in high-engagement podcasts, corroborating some pieces of advice and adding new perspectives on others.

## introduction what makes a particular podcast broadly engaging? as a media form, podcasting is new enough that such questions are only beginning to be understood @xcite . websites exist with advice on podcast production, including language-related tips such as reducing filler words and disfluencies, or incorporating emotion, but there has been little quantitative research into how aspects of language usage contribute to overall listener engagement. this paper investigates the linguistic factors that correlate with engagement, leveraging the written descriptions of the parent show and episode as well as the transcript of the audio. our metric of engagement is stream rate, which we define as the proportion of first-time listeners -of those who have begun streaming the episode -who listen for at least five minutes. notably, stream rate is different from the metric of popularity as given by the raw number of streams; the latter is inevitably influenced by factors unrelated to the content, such as the host or publisher reputation, publicity, expo-sure in recommendations and search engines, and time of publication, whereas a listener's decision to continue listening for as long as five minutes is likely to be influenced by the content. we perform a series of descriptive tests to examine differences in language usage between high and low engagement podcasts, and build predictive models. our tests show that while much of the conventional wisdom on engaging podcasting style (such as to use positive language) bears out in the data, other assumptions (such as to speak slowly) are contradicted and deserve a closer look. we find that stylistic features tend to be more correlated with engagement for podcasts with low absolute numbers of streams than for the most popular podcasts, suggesting that listeners may be less sensitive to style in podcasts made by well-known creators. we also identify those linguistic factors that correlate with our engagement metric across the popularity spectrum, and those that are limited to podcasts within a certain popularity range. our predictive models prove that stylistic factors alone play a significant role in determining if a podcast has high or low engagement, achieving an accuracy of 72% in distinguishing between very high engagement (top 25% of podcasts by stream rate in the corpus) and very low engagement (bottom 25%) examples. we also show that the overall textual information in podcasts is highly predictive of engagement in this experiment, with an accuracy as high as 81%. to understand how style in podcasts compares to other spoken media, we apply our analysis to a corpus of ted talks. finally, we manually examine the highest engagement podcasts in our dataset to characterize their content. predicting performance from language previous research in natural language processing has explored the connections between textual features and audience engagement in books (ganjigunte @xcite , youtube @xcite , news @xcite , ted talks @xcite , and tweets @xcite . other works have modeled the relationship between text and various performance metrics such as movie quote memorability @xcite , forecasting ability @xcite , congressional bill survival @xcite , success of job interviews @xcite , and impact of academic papers @xcite , in addition to the entire field of sentiment and opinion mining of data such as user reviews @xcite . ## dataset construction the spotify podcast dataset @xcite is a recently released corpus of over 100, 000 podcast episodes, mostly in english, that are transcribed with google's speech to text commercial speech recognition, reported in the paper to have an 18% word error on podcasts. a podcast, also known as a 'show' in the dataset, is a collection of episodes. in addition to the speech transcripts, the textual information associated with each podcast episode includes the title and description of the episode and the parent show ( ## linguistic features we define a set of explainable linguistic features that are hypothesized to affect engagement. these features have been drawn from different podcasting advice blogs, alongside some of our own intuitions. length descriptions are known to be important for listeners on their first encounter with the pod-cast. we also measure audio duration, since surveys show it is a consideration @xcite . proportion of ads and show notes descriptions of well-known podcasts tend to contain advertisements of other podcasts made by the same network, links to the hosts' or guests' social media presence and websites, or show notes and transcripts, and podcast creators are often advised to include such information @xcite , and surveys have shown that the majority of podcast listeners do not mind sponsor ads in the content @xcite . we measure the the proportion of text detected on episode descriptions by the extraneous content classifier described in §3.1. the proportion of ads in transcripts is given by a manually identified lda topic that corresponds to words indicative of ads. faithfulness of episode descriptions to transcripts length is a weak signal of informativeness. do listeners seem to prefer descriptions that accurately convey the topics and synopsis of the episode? we measure faithfulness of the episode description to the first ten minutes of the transcript as the cosine similarity between the tf-idf bag of words representation of both texts. while we do not have ground-truth labels to evaluate this definition of faithfulness, we assessed it to be a good heuristic by anecdotally reviewing some examples. 4 distinctiveness podcast creators are often encouraged to develop a distinctive style @xcite . we define distinctiveness as the perplexity of the given text under a unigram language model trained over all the episodes in the dataset. to control for length, we follow the protocol in @xcite of randomly sampling a constant number of words from each text and taking the mean cross entropy over a few samples. 5 reading grade level similarly to zong et al. (2020), we make two measurements: the flesch-kincaid grade level @xcite that measures the number of syllables per word and the number of words per sentence, and the dale-chall grade level @xcite which measures word vocabulary diversity we examine whether podcast creators of high engagement podcasts use more diverse vocabularies, quantified by the entropy of the unigram words in the text, motivated by advice to avoid word repetition @xcite . sentiment and emotion popular advice often encourages podcast creators to be upbeat and positive @xcite . the nrc emotion lexicon (mohammad and turney, 2013) contains positive and negative sentiment assignments, as well as emotions such as anger, trust, and fear, for 14182 words. 6 we measure the proportion of words associated to each of the emotions and sentiments. since a lexicon lookup for sentiment is naturally limited in that it does not account for compositionality and cannot model words and variants that are missing in the lexicon, we also apply a fullsentence classifier, the sentiment model from the google natural language api foot_6 . the output of the classifier is a score between +1 and -1 for each sentence. we define positive and negative polarities for each text as as the proportion of sentences in the text with highly positive (over +0.5) or highly negative (under -0.5) scores. syntax syntactic features are measured by the relative frequencies of each part-of-speech tag. while previous work of this nature finds strong effects of syntactic patterns from parses (ganjigunte @xcite , we find that the noisy speech transcripts result in particularly noisy parses from off-the-shelf parsers. swearing and fillers we conjecture that podcasts with swearing and adult language may not have broad appeal. public speaking recommendations in podcasting guides @xcite emphasize the reduction of filler words like 'yeah' or 'okay', and the use of professional speech. we attempted to manually define lexicons of these types of categories, but found that it is challenging and prone to human biases, especially given the novel domain and automatic transcripts. instead, we take advantage of the observation that some of the topics inferred by the lda model correspond to swear words and filler terms, and measure the proportions of these topics. speech rate and non-speech time podcast creators are often encouraged to speak slowly, since novice speakers tend to rush their delivery @xcite . since the transcripts in the dataset contain time alignments of each word, we measure the duration of speech segments in the audio, giving us the speech rate in terms of words per minute. we also measure the amount of time spent on non-speech. ## conclusion this paper presents the first quantitative analysis of how linguistic style and textual attributes in podcasts relate to listener engagement using automatically computed features. we test several hypotheses, and identify factors that validate popular advice on podcast creation, as well as those with unexpected correlations. our predictive models perform well at distinguishing high and low engagement podcasts using only textual information. our comparison with a similar task on ted data shows similarities and differences between podcasts and public lectures vis a vis engagement. opportunities for future research include the investigation of other podcast creation advice based on paralinguistic features from the podcast audio (such as pitch and intonation), speaker identities and shifts within a conversation, trajectories of linguistic features over the course of the episode, and models using manual transcripts.