Co-Teaching Student-Model through Submission Results of Shared Task (2021)

Abstract: Shared tasks have a long history and have become the mainstream of NLP research. Most of the shared tasks require participants to submit only system outputs and descriptions. It is uncommon for the shared task to request submission of the system itself because of the license issues and implementation differences. Therefore, many systems are abandoned without being used in real applications or contributing to better systems. In this research, we propose a scheme to utilize all those systems which participated in the shared tasks. We use all participated system outputs as task teachers in this scheme and develop a new model as a student aiming to learn the characteristics of each system. We call this scheme “Co-Teaching.” This scheme creates a unified system that performs better than the task’s single best system. It only requires the system outputs, and slightly extra effort is needed for the participants and organizers. We apply this scheme to the “SHINRA2019-JP” shared task, which has nine participants with various output accuracies, confirming that the unified system outperforms the best system. Moreover, the code used in our experiments has been released.

## introduction shared tasks have a long history and have become the highlight of nlp research @xcite @xcite . these tasks have contributed to natural language processing technology development by attracting researchers interested in being the best task player. the systems are evaluated using the output submitted to the task, and they usually have no obligation to submit the system. it limits the participant's contribution once the task is over because the system is a future asset. we believe all participating systems have values as a resource, even if they are not the best. it may be desirable to share it for the sake of innovation in the field as a whole. however, sharing the system is challenging because of the license issue and the running environment. although sharing the system is ideal, we believe sharing system outputs are much easier, and only slight additional effort is needed for the task participants and organizers. we propose a scheme to utilize all system outputs in the shared task to build a unified system that is better than the best single system. more specifically, we construct a system that reproduces the participating systems embedded in the submission results by treating the system submission results as training data (teacher) and building a new model (student). here, those submissions include evaluation data and large unlabeled data submissions. this is an adaptation of the teacher-student architecture of model compression methods such as knowledge distillation @xcite . we call this scheme "co-teaching," borrowing from realworld educational terminology, because the group of participants in the shared task act as teachers and teach a common student. this scheme can be applied to most shared tasks, as it requires submitting the evaluation data and the unlabeled data results. its benefits include building a better system for the task, as well as salvaging the effort of the participants who did not produce the top results. the scheme can provide the best-performing system without violating the participant's license, and it is also possible to design a shared task so that it aims to build a single system from the beginning. in order to prove the effectiveness of the proposed scheme, we conducted an experiment on the shinra2019-jp task. the task is to extract values corresponding to predefined attributes from wikipedia articles to structure the japanese wikipedia. shinra2019-jp follows the con-cept of "resource by collaborative contribution (rbcc)" @xcite , in which a resource (e.g., a knowledge base) is built within the framework of a shared task, and the submission results are made publicly available as a resource. the evaluation data is not announced for this task, and the participants are required to submit results for all wikipedia articles. there are many system predictions for the unlabeled portion of the data due to this evaluation setting. we use ensemble learning to create better results for this task since the outputs are all public. the details of the shinra2019-jp task are described in sec. 4.1. the contributions of this paper can be summarized below: ## proposal scheme we propose a "co-teaching" scheme to address the system submission, which is problematic due to license protection and operating environment issues in the shared task, even when the system developed by the task participant is valuable. in the proposed scheme, the systems submitted to the shared task are considered teachers, and a new student model is trained through their outputs. we expect that this scheme allows us to build a system that comprehensively and integrally reproduces the teacher characteristics. the training data distributed to participants in the shared task can also be used for student learning. this scheme can be performed even after completing the shared task, as long as the training data distributed in the task and the submission results are available. however, it is better to have more information about the teachers available for training the students. it is desirable to be able to use the prediction probabilities assigned to the teacher outputs, as well as training and unlabeled data predictions. therefore, cooperation from shared tasks is essential for this scheme to work effectively. an overview of the teacher-student architecture used in the scheme is presented in fig. 1 . let us assume that we have the outputs {y s } @xmath0, where s is the number of participation systems. each data point can be described as {x i , y 1 i , ..., y s i } @xmath1, where n is the number of data instances, x i is i-th instance, and y s i is the output of s-th teacher for i-th instance. some instances also have {y g i } @xmath2,@xmath3. in this scheme, our goal is to learn the student model {θ sh , θ 1 , ..., θ s , θ out }, where θ sh is the shared parameter to reproduce the features common among the teachers, θ s is the private parameter to reproduce each teacher output, and θ out is the parameter to output the overall prediction results. here, we simultaneously reproduce each teacher model f (x; θ sh , θ s ) : @xmath4; θ sh , θ out ) : @xmath5. the loss function used for learning is written as ## discussion in order to apply the co-teaching scheme to a shared task, it is required to disclose the participants' results. the task organizer needs to obtain permission to publish the results in advance, which may become an obstacle for the participants. how2000 3000 4000 5000 6000 7000 8000 9000 10000 number of instances used for student learning (including development data) 71.0 71.5 72.0 72.5 73.0 micro-average f1 final output : p mean private layers : p mean output layer : p out best system figure 5: changes in student model scores on data for analysis in city category when the number of articles used to train the student model is increased. suppose a co-teaching scheme is to be incorporated into a shared task. in that case, it must be possible to design the student model technically, and it is also desirable that a large amount of unsupervised data is easily obtainable. we discuss the shared tasks that have been implemented in the past from those perspectives. we believe that the conditions are satisfied for the classification tasks such as the sentiment analysis and relation classification tasks in semeval @xcite , dialect classification task in nadi @xcite , and word classification task in conll @xcite . also, the conditions are satisfied for translation tasks such as those in wmt @xcite and wat @xcite as well as generation tasks such as those in sdp @xcite and fns @xcite . the similarity between these tasks is that the formats of the training data and the task submissions are essentially the same. that is, a student model can be designed with few modifications to the model designed for the training data (e.g., by adding an output layer or decoder for each participating system). however, there are cases where the format of the training data and the task submission are different. for example, the training data in the iwslt speech translation task (ansari et al., 2020) consists of the end-to-end speech translation or the transcription dataset and the bilingual corpus, but only the final target-language text is submitted. in this case, if the participant uses the latter non-end-to-end dataset, the dataset used and the task submission format are different. however, the task organizers can solve this formatting problem by requesting the participants to submit the transcribed text. in the above tasks, except for the relational classification task requiring entity pair information, the unlabeled data, such as plain text or speech, is easily obtainable. the prediction results of the systems developed by task participants for those unlabeled data help make the co-teaching scheme work effectively, as shown in the experiments in this paper. as mentioned above, many shared tasks are suitable for the application of the proposed scheme. we have discussed the design of the student model and the data required in the co-teaching scheme, but whether the student model can successfully reproduce the submission system needs to be shown in many future experiments. for these experiments, we need the submission results of many shared tasks. in addition, although it was not available in this experiment, if the output probability of the system is available, further improvement can be expected using the kl-divergence and other methods. we hope that these data become more open in the future. we have given approximate computational costs at the end of sec. 4.2. as shown, the effort and cost involved in implementing this scheme are not trivial because we need to build a student model with sufficient capacities to apply the co-teaching scheme, such as bert. however, we believe that these costs and efforts are much smaller than the total costs and efforts spent by the shared task participants. the teacher-student architecture that we used in this study is simple. nevertheless, we were able to demonstrate the usefulness of the co-teaching scheme. however, there is room for improvement in the architecture, e.g., preventing score degradation in cases where only a single system is superior. in the future, we would like to compare knowledge distillation methods and develop a more suitable architecture for the co-teaching scheme. also, we aim to conduct detailed validation of our proposed scheme by ablation tests with the removal of each system and stress tests with the addition of noise systems. finally, we would like to introduce research with similar motivations. @xcite developed an architecture for shared tasks called tira, consisting of virtual environments for system development and evaluation modules. on tira, each participant of the shared task develops a system on the given virtual environment and receives an evaluation by submitting the system to the evaluation module. in this process, tira stores the systems that have been submitted. but, third parties cannot access such systems directly. instead, tira provides api, and they can receive the system results for any input via api. thus, the systems are public virtually, but licensing issues are minimized as long as the whole system or part of it does not leak out of tira. although this research has a different focus, integrating the co-teaching scheme into the tira architecture would allow seamless student model learning and efficient leveraging of the participants' efforts. ## conclusion in this paper, we proposed a new scheme for shared tasks called "co-teaching." it is a scheme to build a single system from the participants' outputs under the teacher-student architecture. we conducted an experiment based on the shinra2019-jp shared task to demonstrate the effectiveness of our scheme. as a result, we were able to construct a system that was 2.38 points higher in f1-value than the best participating system. we hope that this scheme will be applied to many shared tasks to utilize the participant's efforts effectively. furthermore, we believe the shared tasks can be a more useful scheme if it is not only the place for the optimization of the given task but the outcome is designed so that some resource is obtained, such as a knowledge base (rbcc) or a superior system can be created from the participant's system collection (co-teaching).