Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users (2022)

Abstract: Deaf and hard of hearing individuals regularly rely on captioning while watching live TV. Live TV captioning is evaluated by regulatory agencies using various caption evaluation metrics. However, caption evaluation metrics are often not informed by preferences of DHH users or how meaningful the captions are. There is a need to construct caption evaluation metrics that take the relative importance of words in transcript into account. We conducted correlation analysis between two types of word embeddings and human-annotated labelled word-importance scores in existing corpus. We found that normalized contextualized word embeddings generated using BERT correlated better with manually annotated importance scores than word2vec-based word embeddings. We make available a pairing of word embeddings and their human-annotated importance scores. We also provide proof-of-concept utility by training word importance models, achieving an F1-score of 0.57 in the 6-class word importance classification task.

## introduction over 360 million people worldwide are deaf or hard of hearing (dhh) @xcite . in the u.s. alone, over 15% people are dhh, and regularly rely on captioning while watching videos to perceive salient auditory information @xcite . to provide quality captioning services to this group, it is essential to monitor the quality of captioning regularly. regulators, e.g., the federal communication commission (fcc) in the u.s. @xcite are entrusted with regularly checking the quality of caption transcription generated by different broadcasters. however, given the abundant production of captioned live tv broadcasts, caption evaluation is a tedious and costly task. dhh viewers are often dissatisfied with the quality of captioning provided in live contexts, which provide less time for caption production than prerecorded contexts @xcite . if regulatory organizations that measure the quality of captions used quality metrics that better reflect the dhh users' preferences, dhh viewers' experience may improve. existing metrics used in transcription or captioning include word error rate (wer) @xcite or number of error in recognition (ner) @xcite . as noted by @xcite , a major shortcoming of these metrics is that they do not consider the importance of individual words when measuring the accuracy of captioned transcripts (comparing to the reference transcript) and most metrics assign equal weights to each word. dhh viewers rely more heavily on important keywords while skimming through caption text @xcite . motivated by these shortcomings, prior work had proposed metrics which assign differential importance weights to individual words in captioned text when calculating an evaluation score @xcite . specifically, this prior work leveraged word2vec-based word embeddings to generate and propagate features to another layer of the network @xcite . we build on this prior work and propose an updated approach. the feature space we are using contains both contextual and semantic information of the captioned text, which is crucial in conversational setting, often common in tv, and may better capture long-distance semantic and syntactic relationships. thus, in this work, we contribute more current strategies for calculating importance of words in transcript text, toward a metric that takes word-importance into account when evaluating captions. our contributions in this paper include: ## predicting word importance to demonstrate how to use our dataset to predict the importance of each word, we have begun to investigate several supervised learning methods. the independent variable is the processed 768 × 1 bert-embedding vector of each word, and the output variable is the human-labeled importance score, discretized into six classes, for each word in the dataset. this classification experiment partitioned the corpus into 80% training, 10% development, ## limitations and future work there are several limitations of this ongoing research that we intend to address in future work. ## conclusion the analysis presented above has revealed that bert contextualized word-embedding can better represent the importance of words compared to word2vec embeddings, which had been used in prior work on word-importance prediction @xcite . research indicates that dhh viewers often follow key terms while skimming through captions, and researchers have proposed approaches to guide dhh readers to quickly identify keywords in caption text through visual highlighting @xcite . our findings may allow broadcasters to use embeddings to determine the important words within a sentence and to highlight those words in captions, to support dhh viewers' ability to read (amin et al., 2021a) the captions effectively. in this study, a traditional logistic regression algorithm performed better at predicting importance classes. we are also broadly investigating how to accurately measure the quality of caption transcriptions that are broadcast during live tv programs from the perspective of dhh viewers. we plan to incorporate predictive models into new word-importance weighted metrics, to better capture the usability of live captioning from dhh users' perspective.