C on G en: Unsupervised Control and Generalization Distillation For Sentence Representation (2022)

Abstract: Sentence representations are essential in many NLP tasks operating at the sentence level.Recently, research attention has shifted towards learning how to represent sentences without any annotations, i.e., unsupervised representation learning. Despite the benefit of training without supervised data, there is still a performance penalty compared to supervised methods.Furthermore, the supervised-unsupervised performance gap widens as we reduce the model size. In this paper, we propose an unsupervised sentence representation method to reduce the supervised-unsupervised performance gap, especially for smaller models. Utilizing the concept for knowledge distillation, we derive a distillation framework comprising two training objectives, control and generalize, called ConGen. Experiments on semantic textual similarity (STS), text classification (transfer), and natural language inference (NLI) tasks show that ConGen is on par with supervised training even on smaller models.Furthermore, our method consistently outperformed competitors on multilingual STS.The code and models are available at https://github.com/KornWtp/ConGen.

## introduction in recent years, sentence representation has played a crucial role in various nlp tasks operating at the sentence level @xcite @xcite . many researchers use a transformer language model (lm) @xcite , as a backbone of sentence representation by finetuning lm on natural language inference (nli) and semantic textual similarity (sts) labeled data, which yields promising results @xcite . however, these techniques require labeled data during the finetuning process, which can be a limiting factor in lowresource settings. in order to incorporate unlabeled data into the training process, unsupervised learning paradigms have gained popularity. the contrastive learning paradigm has recently led to significant advancement in unsupervised learning. the main idea of contrastive learning in sentence representation is learning a meaningful representation by maximizing the similarity between differently augmented views @xcite @xcite @xcite @xcite . for example, @xcite proposed a state-ofthe-art contrastive framework called simcse, the learning framework that benefits from the dropout normalization to produce differently augmented views and works well with unsupervised and supervised learning. in particular, simcse is sota on the sts benchmark, and the performance gap between unsupervised and supervised settings in simcse on large networks (e.g., bert-base) is only five points when evaluated on the sts benchmark. however, the performance of simcse rapidly degrades as we decrease the model size, which is undesirable when the computational resource is limited, e.g., edge computing @xcite . for instance, when we use the minilm-l3 (#parameters: 17m) instead of the bert-base model (#parameters: 109m), the spearman rank correlation of simcse-unsupervised drops from 76.25 to 55.10 (averaged across 7 sts corpora). the gap between compressed and base lms is 21.15, as shown in figure 1 . in addition, the gap between unsupervised and supervised learning in minilm-l3 is 21.56, while the gap of larger models like bert-base is only 5.32 points. maintaining a high performance for supervised and unsupervised learning is challenging for smaller lms. in this paper, we aim to retain the advantage of unsupervised learning while mitigating the performance penalty from model compression at the same time. we propose an unsupervised control and generalization distillation, congen, a distillation framework that transfers knowledge from a large model to any model regardless of its architecture and size. not only does congen outperform state-of-the-art unsupervised sentence representation, its performance is also similar to supervised learning (figure 1 ). the crux of congen lies in the distillation mechanism, which handles two different data augmentation views. in particular, we employ inputs derived from two data augmentation operations, which we refer to as control and generalization. the student observes both control and generalization inputs, whereas the teacher observes only control input, which we refer to as a reference input. we derive a similarity distribution between the student inputs (control and generalization) and the instance queue @xcite , and we also do the same to the teacher input (reference). to compare similarity distributions from the teacher to student models, we minimize the discrepancy between the student and teacher distributions in the two following control & generalization learning objectives. first, we control the similarity distribution of the control distribution and the reference distribution to be the same. second, to increase the model's generalizability, we enforce the similarity distribution of the generalization distribution to be the same as the reference distribution. to demonstrate our method's effectiveness, we compare it to other distillation methods in three tasks: semantic textual similarity (sts), text classification (transfer), and natural language inference (nli). the experimental results from sts demonstrate that our method significantly improves the performance of compression models and consis-tently outperforms competitors. in addition, when the model parameters are less than 33 million, con-gen outperforms or matches the supervised baseline (figure 1 ). moreover, in transfer and nli, congen outperforms unsupervised learning, i.e., simcse, and other distillation methods in 11 of 12 cases. additionally, we extend our method to multilingual sentence representation; experimental results from multilingual sts demonstrate that congen outperforms competitors in all cases. the contributions of our work are as follows: • we extend the relational learning approach often used in computer vision to sentence representation learning by designing a novel training process for textual learning. ## proposed method in this section, we describe our control and generalization distillation (congen) method. con-gen is a knowledge distillation technique comprising two objectives: (i) transferring the knowledge from large to small models and (ii) improving the model's generalizability. as illustrated in figure 2 , we describe our framework's training process, including how we organize the inputs and outputs, compare the outputs, and train the model.