GOVERN : Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation (2024)

Abstract: Pre-trained language models have become an integral component of question-answering systems, achieving remarkable performance. However, for practical deployment, it is crucial to perform knowledge distillation to maintain high performance while operating under computational constraints. In this paper, we address a key question: given the importance of unsupervised distillation for student model performance, how can knowledge from multiple teacher models be effectively ensemble during this stage without the guidance of labels? We propose a novel algorithm, GOVERN, to tackle this issue. GOVERN has demonstrated significant improvements in both offline and online experiments, enabling the student model to achieve results comparable to that of teacher ensembles. Our experiments show that GOVERN remarkably requires a mere 1% of the ensemble method’s inference budget to achieve 99.5% of performance. The proposed algorithm has been successfully deployed in a real-world commercial question-answering system, demonstrating its real-world applicability.

## introduction traditional search engine aims at deliver relevant web pages to satisfy users' question, while sometimes the single paragraph that answer the question might buried deep in a web page, it asks for a web-based open domain question answering (openqa) system to find that needle-in-a-haystack info (e.g. @xcite . bert-liked pre-trained language models have achieved state-of-the-art performance in openqa (e.g. @xcite . however, due to computational costs, the direct application of these models in real-time search engines like google is currently unfeasible. for instance, the top-performing models on the natural question dataset, r2-d2 @xcite and unitedqa @xcite come with 1.29b and 2.09b model parameters. further complicating matters is the fact that ensemble methods, which can enhance performance, entail even greater computational overheads. the distillation of knowledge from multiple teachers has emerged as a powerful technique for improving the performance and generalization of dnn while reducing the computational cost. this two-stage training paradigm, which training large model with limited labeled data as teacher and then using it to generate soft label on large amount unlabeled data for the purpose of student training, was first proposed by @xcite . since the knowledge from single teacher may be biased and inaccurate, ensemble distillation from multiple teachers was considered by previous works to achieve more promising performance (e.g. @xcite . several dynamic distillation methods were proposed to solve the problem that different teacher is good at different sample and low-quality teachers may mislead the student. e.g. @xcite proposed a novel rl-based approach to dynamically assigns weights among teachers, @xcite ensembles multi-teacher logits supervised by human-annotated labels in an iterative way. but these dynamic teacher selection methods need supervision signal as guidance, that means they can not apply to unsupervised distillation which is the most important stage in distillation @xcite . in this paper, we propose gradient orientation vote ensemble reinforced distillation (govern) to do sample-wise dynamic teacher selection without the need of label guidance. our main contributions are summarized as follows: to the best of our knowledge, govern is the first method which can find sample-wise high-quality teachers without label guidance. ## answer selection task in a web-based open domain question answering (openqa) system, the primary objective is to select the relevant paragraphs a @xmath0, where p q is a collection of paragraphs obtained in web pages retrieved by search engine. a classic framework of this system is made up of two-stage modules including retriever and ranker, where both modules can be distilled down to a task of classifying the relevance between a question and an answer. our work focus on improving the performance of classification model with the limit of model size. the classification model assesses the relevance of a paragraph, denoted as p, to a specific question, denoted as q, by calculating the relevance score, f (q, p; θ). this scoring function, f , which is parameterized by θ, symbolizes the degree of relevance between the question q and the paragraph p. in practical application, a score threshold is established for the purpose of classification. during training, the classification model is optimized by minimizing the loss over training data: ## methodology we use multiple teachers ensemble distillation as the method to improving the performance of online model with the constriction of computational cost. within a frequently employed knowledge distillation (kd) framework, a large teacher model, denoted as t, is meticulously pretrained or finetuned well ahead of time. the knowledge contained within the teacher model is subsequently transferred to a smaller student model, denoted as s, by minimizing the disparity between the two. this process can be mathematically formulated: specifically, we first utilize unsupervised distillation on a vast amount of task-specific, unlabeled data, followed by supervised distillation on the labeled data. the procedures of the distillation can be viewed in figure 1 . ## related work following the seminal work of @xcite , several studies have sought to develop advanced ensemble algorithms for distillation. we categorize these works into two groups based on their dependency on ground-truth labels. ## conclusion in this paper, we present a novel algorithm, gov-ern, which dynamically selects teachers based on their gradient descent orientation. it does not require ground-truth labels, making it suitable for unsupervised distillation stages. additionally, it can be integrated with existing supervised ensemble methods. the effectiveness of our method is affirmed through extensive experimentation.