Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models (2023)

Abstract: Recently, using large pre-trained Transformer models for transfer learning tasks has evolved to the point where they have become one of the flagship trends in the Natural Language Processing (NLP) community, giving rise to various outlooks such as prompt-based, adapters, or combinations with unsupervised approaches, among many others. In this work, we propose a 3-Phase technique to adjust a base model for a classification task. First, we adapt the model’s signal to the data distribution by performing further training with a Denoising Autoencoder (DAE). Second, we adjust the representation space of the output to the corresponding classes by clustering through a Contrastive Learning (CL) method. In addition, we introduce a new data augmentation approach for Supervised Contrastive Learning to correct the unbalanced datasets. Third, we apply fine-tuning to delimit the predefined categories. These different phases provide relevant and complementary knowledge to the model to learn the final task. We supply extensive experimental results on several datasets to demonstrate these claims. Moreover, we include an ablation study and compare the proposed method against other ways of combining these techniques.

## introduction the never-ending starvation of pre-trained transformer models has led to fine-tuning these models becoming the most common way to solve target tasks. a standard methodology uses a pre-trained model with self-supervised learning on large data as a base model. then, replace/increase the deep layers to learn the task in a supervised way by leveraging knowledge from the initial base model. even though specialised transformers, such as pegasus @xcite for summarisation, have appeared in recent years, the complexity and resources needed to train transformer models of these scales make fine-tuning methods the most reasonable choice. this paradigm has raised interest in improving these techniques, either from the point of the architecture @xcite , by altering the definition of the fine-tuning task @xcite or the input itself @xcite , but also by revisiting and proposing different selfsupervised training practices @xcite . we decided to explore the latter and offer a novel approach that could be utilised broadly for nlp classification tasks. we combine some self-supervised methods with fine-tuning to prepare the base model for the data and the task. thus, we will have a model adjusted to the data even before we start fine-tuning without needing to train a model from scratch, thus producing better results, as shown in the experiments. a typical way to adapt a neural network to the input distribution is based on autoencoders. these systems introduce a bottleneck for the input data distribution through a reduced layer, a sparse layer activation, or a restrictive loss, forcing the model to reduce a sample's representation at the narrowing. a more robust variant of this architecture is dae, which corrupts the input data to prevent it from learning the identity function. the first phase of the proposed method is a dae (fig. 1a ), replacing the final layer of the encoder with one more adapted to the data distribution. contrastive learning has attracted the attention of the nlp community in recent years. this family of approaches is based on comparing an anchor sample to negative and positive samples. there are several losses in cl, like the triplet loss @xcite , the contrastive loss @xcite , or the cosine similarity loss. the second phase proposed in this work consists of a contrastive learning using the cosine similarity (as shown in fig. 1b ). since the data is labelled, we use a supervised approach similar to the one presented in @xcite but through siamese neural networks. in contrast, we consider contrastive 1 2021 learning and fine-tuning the classifier (ft) as two distinct stages. we also add a new imbalance correction during data augmentation that avoids overfitting. this cl stage has a clustering impact since the vector representation belonging to the same class will tend to get closer during the training. we chose some benchmark datasets in classification tasks to support our claims. for the hierarchical ones, we can adjust labels based on the number of similar levels among samples. finally, once we have adapted the model to the data distribution in the first phase and clustered the representations in the second, we apply fine-tuning at the very last. among the different variants from fine-tuning, we use the traditional one for natural language understanding (nlu), i.e., we add a small feedforward neural network (fnn) as the classifier on top of the encoder with two layers. we use only the target task data without any auxiliary dataset, making our outlook self-contained. the source code is publicly available at github foot_0 . to summarise, our contribution is fourfold: ## related work one of the first implementations was presented in @xcite , an application of siamese neural networks using bert @xcite to learn the similarity between sentences. autoencoders were introduced in (kramer, 1991) and have been a standard for self-supervised learning in neural networks since then. however, new modifications were created with the explosion of deep learning architectures, such as dae @xcite and masked autoencoders @xcite . the variational autoencoder (vae) has been applied for nlp in @xcite or @xcite with rnn networks or a dae with transformers in @xcite . transformers-based sequential denoising auto-encoder @xcite ) is an unsupervised method for encoding the sentence into a vector representation with limited or no labelled data, creating noise through an mlm task. the authors of that work evaluated their approach on the following three tasks: information retrieval, re-ranking, and paraphrase identification, showing an increase of up to 6.4 points compared with previous state-of-the-art approaches. in @xcite , the authors employed a new transformer architecture called step-unrolled denoising autoencoders. in the present work, we will apply a dae approach to some transformer models and extend its application to sentence-based and more general classification tasks. the first work published on contrastive learning was @xcite . after that, several versions have been created, like the triplet net in facenet @xcite for computer vision. the triplet-loss compares a given sample and randomly selected negative and positive samples making the distances larger and shorter, respectively. one alternative approach for creating positive pairs is slightly altering the original sample. this method was followed by improved losses such as n -pairloss @xcite and the noise contrastive estimation (n ce) @xcite , extending the family of cl techniques. in recent years, further research has been done on applying these losses, e.g. by supervised methods such as @xcite , which is the one that most closely resembles one in our second phase. sentence-bert @xcite employs a siamese neural network using bert with a pooling layer to encode two sentences into a sentence embedding and measure their similarity score. sentence-bert was evaluated on semantic textual similarity (st s) tasks and the senteval toolkit @xcite , outperforming other embedding strategies in most tasks. in 2 2022 our particular case, we also use siamese networks within the cl options. similar to this approach, the simple contrastive sentence embedding @xcite is used to produce better embedding representations. this unlabelled data outlook uses two different representations from the same sample, simply adding the noise through the standard dropout. in addition, they tested it using entailment sentences as positive examples and contradiction sentences as negative examples and obtained better results than sbert in the st s tasks. whereas until a few years ago, models were trained for a target task, the emergence of pretrained transformers has changed the picture. most implementations apply transfer learning on a transformer model previously pre-trained on general nlu, on one or more languages, to the particular datasets and for a predefined task. this new paradigm has guided the search for the best practice in each of the trending areas such as prompting strategies @xcite , few shot learning @xcite , meta-learning @xcite , also some concrete tasks like intent detection @xcite , or noisy data @xcite . here, we present a taskagnostic approach, outperforming some competitive methods for their corresponding tasks. it should also be noted that our method benefits from using mostly unlabelled data despite some little labelled data. in many state-of-the-art procedures, like @xcite , other datasets than the target dataset are used during training. in our case, we use only the target dataset. ## model in this section, we describe the different proposed phases: a denoising autoencoder that makes the inputs robust against noise, a contrastive learning approach to identify the similarities between different samples in the same class and the dissimilarities with the other class examples together with a novel imbalance correction, and finally, the traditional fine-tuning of the classification model. in the last part of the section, we describe another approach combining the first two phases into one loss. ## experimental setup the datasets for the experiments, the two base models used and the metrics employed are detailed below. ## results we assess the performance of the three methods against the several prominent publicly available datasets. thus, here we evaluate the 3-p hase procedure compared to joint and ft approaches. we report the performance of these approaches in terms of accuracy in we did not have a bigger model for those datasets with long input, so we tried to compare against approaches with similar base models. we truncated the output for the datasets with the longest inputs, which may reflect smaller values in our case. since the advent of @xcite , several current techniques are based on combining different datasets in the first phase through multitask learning and then fine-tuning each task in detail. apart from @xcite , this is the case for the prompt-base procedure from ef l @xcite as well. our method focuses on obtaining the best results for a single task and dataset. several datasets to pre-train the model could be used as a phase before all the others. however, we 7 2027 doubt the possible advantages of this as the first and second phases would negatively affect this learning, and those techniques focused on training the classifier with several models would negatively affect the first two phases. to complete the picture, cae (phuong et al., 2022) is an architecture method, which is also independent of the pre-training practice. a selfexplaining framework is proposed in @xcite as an architecture method that could be implemented on top of our approach. ## conclusion the work presented here shows the advantages of fine-tuning a model in different phases with an imbalance correction, where each stage considers certain aspects, either as an adaptation to the text characteristics, the class differentiation, the imbalance, or the classification itself. we have shown that the proposed method can be equally or more effective than other methods explicitly created for a particular task, even if we do not use auxiliary datasets. moreover, in all cases, it outperforms classical fine-tuning, thus proving that classical fine-tuning only partially exploits the potential of the datasets. squeezing out all the juice from the data requires adapting to the data distribution and grouping the vector representations according to the task before the fine-tuning, which in our case is targeted towards classification.