Small But Funny: A Feedback-Driven Approach to Humor Distillation (2024)

Abstract: The emergence of Large Language Models (LLMs) has brought to light promising language generation capabilities, particularly in performing tasks like complex reasoning and creative writing. Consequently, distillation through imitation of teacher responses has emerged as a popular technique to transfer knowledge from LLMs to more accessible, Small Language Models (SLMs). While this works well for simpler tasks, there is a substantial performance gap on tasks requiring intricate language comprehension and creativity, such as humor generation. We hypothesize that this gap may stem from the fact that creative tasks might be hard to learn by imitation alone and explore whether an approach, involving supplementary guidance from the teacher, could yield higher performance. To address this, we study the effect of assigning a dual role to the LLM - as a “teacher” generating data, as well as a “critic” evaluating the student’s performance. Our experiments on humor generation reveal that the incorporation of feedback significantly narrows the performance gap between SLMs and their larger counterparts compared to merely relying on imitation. As a result, our research highlights the potential of using feedback as an additional dimension to data when transferring complex language abilities via distillation.

## introduction nlp is on a trajectory towards creating increasingly large models @xcite . llms achieve high performance across many tasks in both zero and few-shot settings. however, there are growing concerns about the computational efficiency and environmental sustainability of such approaches @xcite . knowledge distillation @xcite has thus gained a renewed interest, where the term * work done during internship at meta ai. has evolved to denote the process of distilling the responses of llms to slms @xcite . recent work has explored the distillation of commonsense knowledge @xcite , chain-of-thought reasoning @xcite , and summarization abilities @xcite to smaller language models. however, the exploration of distilling creative abilities such as humor into slms remains an open and challenging area of research. in this work, we explore the application of knowledge distillation in the context of the creative style-transfer task of conditional humor generation. given a literal text, the goal is to generate a humorous meaning-preserving paraphrase, as shown in figure 1 . while llms do not match the subtleties of humor in human-written text @xcite , they surpass their smaller counterparts @xcite @xcite . we argue that such creative tasks are challenging for slms to learn. first, due to the inherent constraints of slms, such as reduced model capacity, they are limited in their ability to explore diverse solution spaces and generate innovative outputs. second, although imitating a teacher model is a good starting point, it may result in superficial overfitting to the teacher's style rather than learning the task itself @xcite . fig. 1 demonstrates that even after fine-tuning on the humorous responses from the llm, the slm outputs fall flat. existing techniques that address this gap attempt to improve task understanding through distilling chains of thoughts @xcite . this approach is less applicable for creative generation tasks, which can't be scripted. for instance, it is possible to solve a math reasoning problem systematically, but it is difficult to come up with a recipe for a joke @xcite . to bridge this gap, and improve upon mimicry of the teacher, we propose a novel distillation framework involving both imitation and feedback. following the typical imitation stage in which the slm learns from the llm's outputs, we use the llm as a critic to provide feedback on the student's outputs, facilitating iterative improvement. evaluation of our student models on the proposed task based on empatheticdialogues @xcite and samsum @xcite datasets confirms the advantage of learning from feedback; our student, based on the small bart model, performs on par with llms that are orders of magnitude larger, such as llama2-70b upto 65% of the time, and significantly outperforms supervised fine-tuning by a margin of 18-20% . we assess the strengths and limitations of the critic in evaluating the slm by comparing it against human judgments. we found that our critics can match human judgments with up to 76% accuracy, but can also suffer from biases due to length, position, or other biases. we explore the effect of data size, frequency of critic intervention, and the effect of potential evaluation biases on narrowing the gap between the slm and llm. our work on distilling humor is a step towards more natural and engaging conversations @xcite , making slms more appealing for downstream applications where latency and computational efficiency need to be prioritized. ## method our proposed knowledge distillation framework is depicted in fig. 2 . assuming that the teacher outperforms the student in the humor generation task, our objective is to bring the student's performance closer to that of the teacher. we attempt to achieve this in two phases. first, in the imitation phase (sec 3.1), the student undergoes finetuning using a set of humorous outputs @xmath0, o 2 , ..., o n generated by the teacher for a given input i. in the critique phase (sec 3.2), the finetuned student generates candidate output pairs @xmath1, p 2 , which are then evaluated by a critic model. the scores obtained from the critic model are employed to train the student through a feedback-incorporation method. ## experimental setup dataset. similar to flute @xcite , we pick literal input sentences from the empatheticdialogues dataset @xcite . each conversation is obtained by pairing a speaker and a listener, where the speaker talks emotionally about personal matters, and the listener infers the underlying emotion and responds empathetically. we sample 12,000 sentences from the training set and generate @xmath2,000 literal-humorous text pairs. we sampled 1,000 sentences from the validation and test sets for evaluating the student and 100 samples from the test set for human evaluation. for outof-distribution (ood) evaluation, we sample 500 literal inputs from samsum @xcite . teacher model. we use the 70b chat version of llama 2 @xcite as the teacher model. we generate responses from the teacher using a temperature of 0.8 using nucleus sampling with @xmath3. student model. for student model, we focus on bart-large @xcite . all the bart student responses are obtained using beam search with the number of beams set to 5 during generation. ## conclusions we present a novel framework for knowledge distillation in the context of humor generation, with the teacher llm providing references and feedback on a smaller student model's performance. our approach involves leveraging a critic to guide the student model toward generating more humorous outputs. the effectiveness of our method is demonstrated through evaluations conducted by both the llm and humans. additionally, we analyze the effect of various design choices on the performance, such as the frequency of feedback and training set size. our analysis sheds light on the limitations of llm-based critics, and serves as motivation for future research in mitigation of biases in llm-based evaluation and ai feedback.