E du Q uick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media (2023)

Abstract: This study explores the capacity of large language models (LLMs) to efficiently generate summaries of informal educational content tailored for platforms like TikTok. It also investigates how both humans and LLMs assess the quality of these summaries, based on a series of experiments, exploring the potential replacement of human evaluation with LLMs. Furthermore, the study delves into how experienced content creators perceive the utility of automatic summaries for TikTok videos. We employ strategic prompt selection techniques to guide LLMs in producing engaging summaries based on the characteristics of viral TikTok content, including hashtags, captivating hooks, storytelling, and user engagement. The study leverages OpenAI’s GPT-4 model to generate TikTok content summaries, aiming to align them with the essential features identified. By employing this model and incorporating human evaluation and expert assessment, this research endeavors to shed light on the intricate dynamics of modern content creation, where AI and human ingenuity converge. Ultimately, it seeks to enhance strategies for disseminating and evaluating educational information effectively in the realm of social media.

## introduction and motivation the omnipresence of social media in recent years is well-known @xcite . the short video platform tiktok is a notable example due to its advanced content recommendation algorithm that is related to the users' flow experience @xcite . the algorithm uses multiple features, such as user interaction or watch time, to tailor the presented content to each individual. the average usage of the tiktok app is roughly 45-60 minutes per * these authors contributed equally to this work. day, mostly prevalent among school students @xcite . therefore, motivating students to allocate some of their tiktok screen time to educational content can substantially boost the amount of time they dedicate to educational activities. it seems natural then to leverage the same mechanisms that lead to high usage time in a positive way, e.g. to increase engagement with and consumption of educational videos @xcite . however, educators are already heavily burdened in their day-to-day work, as shown by the high reported burnout rates @xcite , and should not have the additional task of summarizing classroom content to fit a short video format. therefore, we see a need for effective summarization as a strategy to distil intricate information and to adapt educational content to social media short videos, especially tiktok. we focus on text summarization, which serves to condense comprehensive information into concise yet coherent forms while preserving fundamental essence and enabling effective knowledge transmission in various domains, ranging from news articles to research papers @xcite @xcite . particularly within the educational context, content summarization emerges as a potential solution to bridge the disparity between information overflow and the necessity for accessible and comprehensible insights. in social media-driven day-today life, the educational landscape has not wholly adapted to this. content summarization would thus align seamlessly with the evolving demands of pedagogical practices on the one hand, and creating engaging content on the other hand. the experiments in this paper led to eduquick, a textual dataset for educational content summarization for tiktok video content creation. eduquick is a multidomain textual dataset containing 500 items, topics, source text, summarized articles, and metadata regarding the source text. we sourced the educational material from howstuffworks and constructed a template with features we identified for creating successful tiktok content (section 4). we effectively instructed the gpt-4 model (ope-nai, 2023) to generate educational content adhering to the defined template. next, for 150 of these summaries, we assessed the generated summaries through both human evaluation and an instructionbased evaluation using gpt-4 model (section 5). additionally, we elaborate on our efforts to gauge the quality and suitability of the educational content summaries for tiktok by seeking the insights of experienced tiktok content creators. to sum up, we see a lot of unused potential in the use of short videos on social media to drive education, but are also aware that the production of these videos is too time-consuming to be adopted universally. this is further aggravated because the short format requires precise planning of the content and its deliverance, which in turn also makes them an effective tool for learning @xcite . hence, we propose to leverage the summarization capabilities of modern llms to automatically create scripts for short educational videos suitable for social media platforms from input documents of educational content. this would shorten the video production process and make it a more feasible approach to modern, blended learning. in order to make progress on these summaries and their special requirements to be successful at tiktok, an automatic evaluation procedure is necessary. we will investigate if llms can fill this gap. our endeavors culminated in the creation of a novel summarization dataset, along with the formulation of a comprehensive set of experimental designs. these designs were meticulously crafted to assess the proficiency of the gpt-4 model in both summarization and evaluation tasks. furthermore, our work has generated a series of insights highlighting the existing deficiencies within the summarization domain and offering valuable guidance on its future trajectory. with this study we hope to inspire more efforts in this research direction and offer an approach to the summarization of data tailored for social media. the following sections address the research questions listed below. ## data collection and preprocessing to create a dataset containing educational yet entertaining content for tiktok videos, the data was extracted using web scraping techniques from "how-stuffworks" foot_0 @xcite , a website known for its diverse educational content on subjects including science, history, animals, entertainment, culture, technology, and lifestyle. this choice was based on the website's abundance of interesting and informative material, aligning perfectly with our aim to produce engaging and educational tiktok content. we have extracted 100 articles per topic, resulting in a dataset comprising 500 articles across 5 diverse topics (health, entertainment, animals, science and auto). throughout the collection process, we applied minimal preprocessing, ensuring that the entirety of each article's content was retained to maintain its integrity and authenticity. in addition to the articles' content, we collected valuable metadata, including citation information, such as article links, authors' names, publication dates, and extracting dates. the metadata offers crucial contextual information and simplifies the process of citing the tiktok content accurately. ## enhancing tiktok content creation through strategic prompt selection prompt design plays a pivotal role in guiding llm models to create engaging tiktok content summaries based on the collected articles (as described in section 3). we decided to adopt a template prompt that incorporates essential features identified for viral educational tiktok video content, as described in section 4.1. these features were curated based on insights from a qualitative analysis of renowned educational tiktok content creators (i.e. @veritasium, @renegadescienceteacher, @distilledscience, @chemteacherphil) and the research cited above. the selected prompt approaches were chosen for their ability to enhance relevance, captivate viewers' attention, and ensure an appealing learning experience. ## the dataset the presented dataset is a curated collection of model-generated text for educational tiktok content, abbreviated as eduquick. this dataset is the result of evaluating and selecting high-quality content generated by the gpt-4 model following an empirical study. it aims to provide engaging and informative summaries suitable for tiktok's educational audience. while our evaluation of gpt-4's assessment capability, as discussed in the preceding section, did not meet our expectations, it is worth noting that llms have already demonstrated their capacity to generate high-quality summaries @xcite . ## conclusion and future work in this work we focused on generating engaging educational content for tiktok. we extracted materials from howstuffworks and created a template based on successful tiktok features. using gpt-4, we instructed the model to generate educational content based on the template we crafted. we evaluated the generated content through human assessment and gpt4 evaluation, resulting in two sets of evaluation scores, which we term silver foot_3 standard dataset. the released dataset and evaluation scores offer valuable resources for future research and development in natural language generation for tiktok's educational content creation. in future work, exploring advanced techniques for fine-tuning llm models specifically for tiktok content generation could lead to higher-quality and more engaging educational content. we argue that the automatic evaluation of this specific content is still a challenging task since gpt-4 was not able to fill this gap. while human evaluation through crowdsourcing is possible, we argue that due to its high cost, it is impracticable for the development cycle of summarization systems. we therefore call on the scientific community to devise an automatic evaluation procedure, that will in turn facilitate research into the automatic summarization for educational short videos. moreover, integrating summaries with aigenerated talking-head videos and audio presents an intriguing niche for enhancing the educational impact and viewer engagement of the generated tiktok content as well as providing a complete automatic pipeline for social media video genera-tion. finally, conducting user studies and collecting feedback directly from tiktok users can provide valuable insights into their preferences and interests in educational content, guiding the refinement of the content generation process and creating tiktok videos that resonate more effectively with the platform's diverse audience.