Exploring Abstractive Text Summarisation for Podcasts: A Comparative Study of BART and T5 Models (2023)

Abstract: Podcasts have become increasingly popular in recent years, resulting in a massive amount of audio content being produced every day. Efficient summarisation of podcast episodes can enable better content management and discovery for users. In this paper, we explore the use of abstractive text summarisation methods to generate high-quality summaries of podcast episodes. We use pre-trained models, BART and T5, to fine-tune on a dataset of Spotify’s 100K podcast. We evaluate our models using automated metrics and human evaluation, and find that the BART model fine-tuned on the podcast dataset achieved a higher ROUGE-1 and ROUGE-L score compared to other models, while the T5 model performed better in terms of semantic meaning. The human evaluation indicates that both models produced high-quality summaries that were well received by participants. Our study demonstrates the effectiveness of abstractive summarisation methods for podcast episodes and offers insights for improving the summarisation of audio content.

## introduction podcasts are a rapidly growing and popular medium for consuming knowledge and entertainment through spoken audio files that can be streamed or downloaded. with the podcast industry reaching new heights, there is a need for innovative and computationally effective methods for processing, analysing, and summarising podcast content. this need is further highlighted by the fact that spotify 1 acquired anchor and gimlet media, two leading podcast companies, for @xmath0500 million in this industry, demonstrating the growth and importance of podcasting @xcite . the abundance of data generated in various forms worldwide necessitates the need for methods to compress and understand this data, enabling 1 www.spotify.com the discovery of content available globally. one of these methods is automatic text summarisation, a technique used to shorten lengthy input texts to small coherent texts that convey the original meaning, reducing reading time and facilitating faster processing of research information. the field of natural language processing (nlp) has witnessed significant progress in the development and research of text summarisation, resulting in an active research area in both information retrieval (ir) and nlp. the successful integration of transformer architecture and attention mechanisms has also contributed significantly to the advancement of text summarisation techniques. the podcast domain presents unique challenges in text summarisation due to its conversational nature and informal language use, which could cause difficulties in extracting salient information. furthermore, podcast episodes can be lengthy, making it a daunting task for listeners to find and identify those of interest. summarising podcast transcripts into short, readable text can alleviate these issues, allowing podcast listeners to make more informed decisions on which episodes to listen to and enabling easier retrieval of information @xcite . the two fundamental approaches in text summarisation are extractive and abstractive summarisation. extractive summarisation involves selecting the most relevant sentences from the input text to form a summary, while abstractive summarisation involves generating new sentences to capture the content of the input. although both approaches can be used to summarise podcast transcripts, abstractive text summarisation is more challenging and requires the use of sophisticated techniques @xcite . this paper focuses on generating automated summaries of podcast episodes using abstractive text summarisation to provide users with a concise summary of the podcast's content. the paper aims to explore how state-of-the-art nlp (sota) models can generate summaries that convey the essence of the podcast and investigate the effectiveness of these summaries for podcast listeners. to achieve these objectives, the paper presents a systematic review of the background research on summarisation, the podcast domain, and related work in this field. it also analyses a dataset of podcast transcripts to develop a conceptual framework for the methods and techniques applied in this study. finally, the paper examines the results and findings of the study, including the state-of-the-art evaluation metrics adopted for this study, and presents a conclusion that discusses the study's limitations, potential future work, and reflection on the study. ## background recent years have seen significant advancements in the field of automatic text summarisation, with the development of new techniques and models for generating summaries. two main approaches are commonly used in text summarisation: extractive and abstractive summarisation. extractive summarisation involves selecting the most relevant sentences from the source text, while abstractive summarisation generates a summary by rephrasing the text into new sentences. while extractive summarisation can be simpler and more efficient, abstractive summarisation has the potential to produce more informative and coherent summaries @xcite . in recent years, the use of deep learning models, such as transformers, has led to significant improvements in the quality of abstractive text summarisation. models such as bart, t5, and gpt-3 have achieved state-of-the-art performance on summarisation tasks, demonstrating the potential of these models for generating high-quality summaries. these models are pre-trained on large amounts of text data and can be fine-tuned on taskspecific datasets to generate summaries that capture the essential content of the source text @xcite @xcite the task of summarising podcasts has gained attention in recent years, with several studies proposing approaches for automated podcast summarisation. @xcite proposed an interactive summarisation approach for news podcasts, which allows users to engage with the summarisation process by providing feedback. this approach uses extractive summarisation to select important sen-tences from the podcast transcript and then generates a summary from these sentences. the system then allows users to rate the summary and provide feedback, which is used to refine the summary for future users. @xcite proposed an extractive summarisation approach for podcast episodes, where the summary is generated by selecting the most important sentences from the podcast transcript. they used a graph-based ranking algorithm to score each sentence based on its importance, and then selected the top sentences to form the summary. the system was evaluated on a dataset of 100 podcast episodes and achieved a rouge-2 score of 0.26. @xcite introduced both extractive and abstractive summarisation approaches for news and podcast data using transfer learning. they fine-tuned bert and gpt-2 models on a dataset of news articles and podcast transcripts to perform extractive and abstractive summarisation, respectively. the authors reported that the abstractive model outperformed the extractive model in terms of rouge scores on both news and podcast data. vartakavi and garg (2020) presented an extractive summarisation approach for podcast episodes that uses sentence embeddings to score the importance of each sentence. the system selects the top sentences based on their scores to generate the summary. the authors evaluated their system on a dataset of 400 podcast episodes and achieved a rouge-2 score of 0.30. @xcite proposed an abstractive summarisation approach for podcast episodes, which uses a transformer-based model to generate summaries. the system was trained on a dataset of podcast transcripts and evaluated on a separate test set. the author reported that the system was able to generate coherent and informative summaries of the podcast episodes. in summary, podcast summarisation has gained attention in recent years, with both extractive and abstractive approaches proposed for the task. extractive approaches are simpler to implement but often produce less informative summaries, while abstractive approaches require more complex models but can produce more informative summaries. ## dataset the podcast domain is notably distinct from other domains such as news in terms of its style and struc-ture. to conduct this study, a dataset was obtained from spotify, which has created the most extensive corpus of transcribed data and audio files in this emerging domain. the corpus is comprised of over 100,000 podcast episodes, amounting to almost 60,000 hours of speech. the transcriptions were generated using google cloud platform's speech-to-text api (gcp-api), revealing a unique and noisy set of data that has yet to be fully explored in the field of nlp. additional information about the dataset can be found in @xcite . podcasts are structured in different ways such as scripted and unscripted monologues, interviews, conversations, debate, and includes clips of non-speech audio material. this dataset includes a diverse range of topics, subject matter, speaking styles, and formats, comprising both audio files and transcripts of podcast episodes in portuguese and english. however, the study solely focuses on summarizing english-language podcasts. future work could incorporate audio files and transcripts in portuguese. additionally, the dataset features metadata, such as descriptions provided by the creators, which can serve as labeled data or reference summaries for summarization.taset covers a wide range of topics, speaking styles, and formats, and includes both audio files and transcripts of podcast episodes in portuguese, although this study focuses solely on summarising english-language podcast episodes. furthermore, the dataset provides metadata, including descriptions provided by creators, which are used as labelled data for the summarisation task. ## design and methodology the primary objective of this study is to develop an abstractive summarisation system that generates a concise and informative summary of podcast episodes, enabling users to make an informed decision about which podcast to listen to. the ideal summary should accurately convey the essence and most important attributes of the episode, including topical content, participants, and genre, and it should be easily readable on a smartphone with less than 200 words @xcite . to achieve this, we aim to fine-tune state-of-theart transformer-based models (e.g. t5 and bart) on podcast data. to achieve this we use podcast transcripts rather than audio files to fine-tune the models. audio data has not been selected for this project due to several reasons. one of the main considerations is the significant variability in audio quality across different podcast episodes. the audio content ranges from professionally produced podcasts with high-quality audio to amateur podcasts that exhibit a wide variety of audio quality. additionally, the dataset includes episodes self-published through a phone application, further introducing variations in the quality and equipment used by the creators. given these factors, opting for transcript data ensures a more consistent and standardised input for the summarisation task. our research pipeline follows a sequential approach that involves several processes to ensure effective summarisation. the process includes text pre-processing, model fine-tuning, and summary generation. this study evaluates the quality of the generated summaries using state-of-the-art evaluation metrics and investigates user attitudes towards the produced summaries. the ultimate goal is to improve the accessibility of podcast content by providing a concise summary that saves users' time and effort in selecting podcasts to listen to. ## evaluation in this section, we present the results of our evaluation, which was conducted on the test set consisting of 1,027 podcast episodes. in addition to the automated metrics, we conducted a human evaluation to gain a better understanding of how people interpret the generated summaries, especially since the creator-provided descriptions were of poor quality. ## conclusion in conclusion, this paper presents a study on the summarisation of podcast episodes using abstractive methods. we explored the use of the bart and t5 models, fine-tuned on a dataset of podcast episode descriptions, and evaluated their performance using automated metrics and human judgement. our results showed that the fine-tuned models outperformed their pre-trained counterparts and achieved high scores in both rouge and bertscore metrics. moreover, the human evaluation indicated that the generated summaries were of high quality and well-received by participants. overall, our findings demonstrate the potential of using abstractive summarisation for podcasts, providing listeners with a quick and accurate summary of episodes.with the help of abstractive text summarisation, podcast creators can implement this technology to automatically generate descriptions for their episodes, which was a manual process in the podcasting industry, helping them save time and allowing the users to read high-quality descriptions for their favourite podcasts. in light of these findings, it is clear that the podcast domain could greatly benefit from the use of nlp technology in generating accurate and concise summaries of audio content. this could help users better manage and discover relevant content, while also making podcast episodes more accessible to individuals with hearing impairments or language barriers in the future. future work could explore the use of additional features or fine-tuning methods to further improve the performance of the summarisation models on podcast data. another aspect that can be explored is finding methods to tackle disfluencies in spoken text.overall, this project has provided valuable insights into the application of nlp in the podcast domain and the potential for improving the accessibility and usability of podcast content.