Language that Captivates the Audience: Predicting Affective Ratings of TED Talks in a Multi-Label Classification Task (2021)

Abstract: The aim of the paper is twofold: (1) to automatically predict the ratings assigned by viewers to 14 categories available for TED talks in a multi-label classification task and (2) to determine what types of features drive classification accuracy for each of the categories. The focus is on features of language usage from five groups pertaining to syntactic complexity, lexical richness, register-based n-gram measures, information-theoretic measures and LIWC-style measures. We show that a Recurrent Neural Network classifier trained exclusively on within-text distributions of such features can reach relatively high levels of overall accuracy (69%) across the 14 categories. We find that features from two groups are strong predictors of the affective ratings across all categories and that there are distinct patterns of language usage for each rating category.

## introduction the ability to communicate competently and effectively is key to personal contentment, academic achievement and professional career success. the ability to communicate competently is even found to enhance social relationships @xcite . in educational and vocational contexts, competent speakers experience more success in sharing their knowledge, ideas and views @xcite . one of the essential communication skills is that of giving an informative and impactful public speech. the development and mastery of public speaking is recognized as a core communicative competence @xcite and has been incorporated into the educational curric-ula and standards for both first and second/foreign language @xcite . across various assessment grids and evaluation forms, a speech is considered competent and effective when the communicative intention is fulfilled -such as that of informing, persuading or entertaining an audience (and most speeches aim at achieving one or more of these) is achieved -when it is appropriate to the specific communicative context and when it matches the expectations of the audience. given its central role, it is hardly surprising that research on public speaking has given rise to a vast field, scattered across disciplines using a range of methodological approaches @xcite . this research has been directed towards understanding the role of both verbal and non-verbal components in the assessment of the multidimensional construct of public speaking competence @xcite . in more recent years, there has been a growing interest in automatic assessment of speaking skills. most studies in this area have primarily focused on the role of auditory and acoustic measures of prosody (such as loudness, voice quality and pitch) and non-verbal cues (such as the use of gestures, eye-contact and posture) in predicting human ratings (see section 2 on related work for more details). the present paper contributes to and expands this emerging line of research by modeling public speaking skills in a multi-label classification task on the basis of five groups of features of language usage. more specifically, the aim of the paper is twofold: (1) to automatically predict the affective ratings of public speeches assigned by online viewers across fourteen rating categories and (2) to determine what types of features drive classification accuracy in predicting each of the categories. in pursuit of these aims, we use a large open repository of public speaking, ted talks 1 . ted (technology, entertainment and design) talks are designed to not exceed the length of 18 minutes and provide succinct and enlightening insights on various topics or ideas that are "worth spreading." topics presented in these talks range from global warming to what keeps us happy and healthy as we go through life. most popular ted talks, such as brené brown's "the power of vulnerability" has garnered almost 48.000.000 million views. ted presenters are often selected not only on the basis of their expertise on a given topic but also for their ability to effectively and succinctly communicate. the dataset used in the paper included 2392 public speaking videos aligned with 5.89 million human ratings. each ted speech is rated in terms of 14 categories: (1) beautiful, (2) confusing, (3) courageous, (4) fascinating, (5) funny, (6) informative, (7) ingenious, (8) inspiring, (9) jawdropping, (10) long-winded, (11) obnoxious, (12) ok, (13) persuasive and ( 14 ) unconvincing. the speech transcripts were automatically analyzed using cocogen, a computational tool that implements a sliding-window technique to compute a series of measurements for a given language feature (contours) that captures the distribution of that feature within a text. a recurrent neural network (rnn) classifier -that exploits the sequential information in those contours -was trained on these distributions to predict 14 rating categories investigated in the paper. the remainder of the paper is organized as follows: section 2 briefly presents a concise review of related work. section 3 describes the ted talk corpus we used alongside with viewer ratings. sections 4 presents the tool used for automated text analysis and the five groups of features used in the paper. section 5 gives a description of the classification model architecture. section 6 presents the main results and discusses them. finally, section 7 summarizes the main findings reported in the paper and proposes future research directions. ## data we analysed the tedtalk data gathered on the ted.com website foot_0 . we crawled the site and obtained every ted talk transcript and its metadata from 2006 through 2017, which yielded a total of 2668 talks. viewers on the internet can vote for three impression-related labels out of the 14 types of labels: beautiful, confusing, courageous, fascinating, funny, informative, ingenious, inspiring, jaw-dropping, longwinded, obnoxious, ok, persuasive, and unconvincing. the labels are not mutually exclusive and users can select up to three labels for each talk. if only a single label is chosen, it is counted three times. all talks that featured more than one speaker as well as talks that centered around music performances were removed. this resulted in a dataset of 2392 ted talks with a total number of views of 4139 million and a total number of 5.89 million ratings (see ## automated text analysis the speech transcripts were automatically analyzed using cocogen, a computational tool that implements a sliding window technique to calculate within-text distributions of scores for a given language feature (for current applications of the tool in the context of text classification, see @xcite @xcite ). here, in this paper, we employ a total of 119 features derived from multi-disciplinary integrated approaches to language @xcite that fall into five categories: (1) measures of syntactic complexity, (2) measures of lexical richness, (3) register-based n-gram frequency measures, (4) information-theoretic measures, and (5) liwc-style (linguistic inquiry and word count) measures. the first four categories of features are derived from the literature on language development showing that, in the course of their lifespan, humans learn to produce and understand complex syntactic structures, more sophisticated and diverse vocabulary and informationally denser language @xcite @xcite @xcite . the inclusion of features in the fourth category is motivated by recent research on language adaptation @xcite and research that looks at language from the perspective of complex adaptive systems @xcite indicating that, based on accumulated language knowledge emerging from lifelong exposure to various types of language inputs, humans learn to adapt their language to meet the functional requirements of different communicative contexts. the fifth group of features is based on insights from many years of research conducted by pennebaker and colleagues @xcite , showing that the words people use in their everyday life provide important psychological cues to their thought processes, emotional states, intentions, and motivations. in contrast to the standard approach implemented in other software for automated text analysis that relies on aggregate scores representing the average value of a feature in a text, the slidingwindow approach employed in cocogen tracks the distribution of the feature scores within a text. a sliding window can be conceived of as a window of size ws, which is defined by the number of sentences it contains. the window is moved across a text sentence-by-sentence, computing one value per window for a given indicator. the series of measurements generated by coco-gen captures the progression of language performance within a text for a given indicator and is referred here to as a 'complexity contour'. in general, for a text comprising n sentences, there are @xmath0. 3 cocogen uses the stanford corenlp suite @xcite for performing tokenization, sentence splitting, part-of-speech tagging, lemmatization and syntactic parsing (probabilistic context free grammar parser @xcite ). a total of 25 measures results from the combination of (a) a 'reference list' containing the top 100,000 most frequent n-grams and their frequencies from one of five register subcomponents of the coca corpus and (b) the size of the n-gram (n 2 [1, 5]). the fourth group includes three information-theoretic measures that are based on kolmogorov complexity. these measures use the deflate algorithm @xcite to compress a text and obtain complexity scores by relating the size of the compressed file to the size of the original file (see @xcite for the operationalization and implementation of these measures). ## classification models we used a recurrent neural network (rnn) classifier, specifically a bidirectional dynamic rnn model with long short-term memory (lstm) cells. a dynamic rnn was chosen as it can handle sequences of variable length foot_3 . as shown in figure 1 , the input of the contour-based model is a sequence @xmath1, x 2 , . . . , x l , x l+1 , . . . , x n ), where x i , the output of cocogen for the ith window of a document, is a 119 dimensional vector, l is the length of the sequence, n 2 z is a number, which is greater or equal to the length of the longest sequence in the dataset and x l+1 , • • • , x n are padded 0-vectors. the input of the contourbased model is fed into a rnn which consists of 5 bidirectional lstm layers with 400 hidden units in each cell. to predict the class of a sequence, we concatenate the hidden variable of the last lstm cell in layer 5 ! h 5l , i.e. the hidden variable of 5th rnn layer right after the feeding of x l , with hidden variable of the last lstm cell in the backward direction ! h 51 . the result vector of concatenation [ in which @xmath2, y 2 , . . . , y n ), @xmath3, ŷ2 , . . . , ŷn ) is the output vector of the sigmoid layer rounded to closest integer. for optimization, we used adamax with a learning rate ⌘ = 0.0001. the dropout rate of the rnn layers and the dropout layer is set to 0.5. the minibatch size is 32, which was shown as a reasonable value for modern gpus @xcite . all models were implemented using pytorch @xcite . our baseline model has the same network archi-tecture as the model described above, but instead of being trained on complexity contours, it was trained on sentence embeddings extracted from the universal sentence encoder (use) @xcite . use takes a sentence as input and a 512dimensional sentence representation as its output. the pretrained use model we used was obtained from the tensorflow (tf) hub website and, according to tf hub, the model was trained with a deep averaging network (dan) encoder @xcite . ## results and discussion we report the results of multi-label classification with five-fold cross validation. the performance metrics of the rnn classification model (global accuracy, precision, recall and macro f1 scores) are presented in the result that liwc-style features emerged as strong predictors is not surprising since previous research employing these measures has provided valuable insight into psychological processes and behavioral outcomes @xcite . a closer examination of how individual features within each feature-group distinguished between higher-rated and lower-rated ted talks in a given rating category revealed some interesting patterns. for reasons of space, we focus on the results for the seven most frequently selected categories (persuasive, courageous, fascinating, beautiful, inspiring, informative and funny) for which classification accuracy was consistently greater than 70%. for each feature, we determined the differences between the group means for each indicator of language use (m higher rated -m lower rated ) for the persuasive category. for the persuasive rating category, the top liwc-style features of highly rated talks concern words associated with words related to core drives and needs (power) and personal concerns (risk, money and work), while words related with perceptual processes (see, hear, feel) are associated with talks with lower ratings on the persuasive dimension. highly persuasive talks are further characterized by a stronger reliance on higher frequency n-grams from more formal language registers: the top-five n-gram features associated with highly persuasive talks concern frequent 3-, 4-and 5-grams from the academic register and 3-and 4-grams from the news register. at the same time, frequent n-grams from the fiction register are indicative of lower persuasive talks. regarding lexical richness, we found that highly-rated talks in the persuasive rating category exhibit high lexical diversity (cttr, rttr, ndw) in combination with high word prevalence and low lexical sophistication (nawl, ngsl, anc, bnc), indicating that persuasive talks use words that are widely known rather than those that are advanced and infrequent. at the syntactic level, persuasive talks show tendencies towards higher degrees of subordination (e.g. dependent clauses per t-unit) and phrasal complexity (complex nominals per t-unit) and lower degrees of coordination (coordinate phrases per clause). figure 2 in the appendix shows the frequency with which features of particular types appeared in the top-five most associated (left) or top-five most dissociated features (right). 6 the figure discloses distinct patterns of language usage for particular rating categories: ted talks with high ratings in the categories beautiful, inspiring, courageous and funny are characterised by a strong reliance on frequent n-grams from the fiction register, words from the liwc types pronoun and social, and relatively higher scores on indicators of lexical sophistication. at the same time, these talks exhibit relatively low proportions of frequent academic n-grams, low lexical diversity and shorter length of production units. highly rated talks on the fascinating and ingenious dimensions group together and exhibit higher proportions of n-grams from the spoken register, higher proportions of words from the liwc function-type -notably the personal pronoun i and are characterized by higher lexical sophistication. finally, persuasive, ingenious, fascinating and inspiring talks display higher word prevalence score, which estimate the proportion of the population that knows a given word. this result indicates that the inclusion of this newly introduced crowd-sourced language metric @xcite is a valuable addition to the existing features employed in research on automated assessment of spontaneous speech and calls for the development of additional crowdsourced language features. ## conclusion the ability to communicate competently and efficiently yields innumerable benefits across a range of social areas, including the enjoyment of congenial personal relationships, educational success, career advancement and, more generally, successful participation in the complex communicative environments of the 21st century. public speaking is the epitome of spoken communication and, at the same time, the most feared form of communication. the paper contributes to the growing body of research that relies on automatic assessment of speaking skills and machine learning to better understand what makes a public speech effective. we performed a multi-label classification task to automatically predict human affective ratings associated with 14 categories on a large dataset of ted talk speech transcripts. we demonstrated that a recurrent neural network classifier trained exclusively on within-text distributions of 119 language features can reach relatively high levels of accuracy (> 70%) on eight out of fourteen rating categories. we further showed that all rating categories are best predicted by (1) liwc-style features, which counts words in psychologically meaningful categories, and (2) n-gram frequency measures, which reflect the use of register/genrespecific multiword sequences. closer analysis of the distributions of particular language features disclosed distinct patterns of language usage for particular rating categories. more generally, the present paper responds to recent calls in the international scientific community of machine learning to use not only black box models but also explainable (white-box) models, since in any given application domain there is a need for both accurate but also understandable models @xcite . in this paper, we have demonstrated in the domain of public speaking that models trained on human interpretable features in combination with deep learning classifiers can compete with black box models based on word embeddings.