C ontra CLM : Contrastive Learning For Causal Language Model (2023)

Abstract: Despite exciting progress in causal language models, the expressiveness of their representations is largely limited due to poor discrimination ability. To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level. We assess CONTRACLM on a variety of downstream tasks. We show that CONTRACLM enhances the discrimination of representations and bridges the gap with encoder-only models, which makes causal language models better suited for tasks beyond language generation. Specifically, we attain 44% relative improvement on the Semantic Textual Similarity tasks and 34% on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of representations, CONTRACLM also boosts the source code generation capability with 9% relative improvement on execution accuracy on the HumanEval benchmark.

## introduction causal language models (clm) have seen remarkable success in language generation, both in natural language @xcite @xcite and programming language @xcite . however, one limitation at their core is the poor discrimination ability of the representations, which often causes a large performance gap with encoder-only or encoderdecoder models on discriminative tasks (see appendix d.1), and hence limits the wide usage of clm beyond language generation. prior studies posit that the anisotropy issue, i.e., representations being squeezed into a tiny cone in the vector space @xcite , can be the main cause of the poor discrimination ability of language models across different architectures and objectives. many efforts have focused on resolving the anisotropy issue on encoder-only or encoderdecoder models, either through post-processing @xcite or integrating different regularization terms into the training objective @xcite . a recent work @xcite shows that the decoder-only clm does not suffer from the anisotropic problem as long as the model is beyond a certain size. however, we find that such conclusions can vary across domains. as shown in figure 1a , clms pretrained on text, i.e., @xcite , do yield representations with good isotropy and discrimination as long as the model is not smaller than 774m parameters (gpt2-large), whilst @xcite , pretrained on programming language data, consistently suffers from anisotropy and poor discrimination across different model sizes. therefore, an effective training strategy is still essential for clms to improve representation quality with better isotropy and discrimination (figure 1b ). we conjecture that this is essential not only for models suffering from inferior representations, e.g., codegen, and gpt2 (124m) but also for those with a good starting point (suffer less e.g., gpt2-large (774m)). we argue that an ideal clm should yield isotropic representations to better leverage the representation space, as well as discriminative representations such that tokens or sequences from the same context are mapped to comparatively closer locations in the vector space compared to those from randomly sampled contexts. to this end, we developed contraclm, a novel contrastive learning framework at both the token-level and sequence-level. contraclm is able to promote more uniformly distributed and hence isotropic representations by separating the instances at different semantic levels, e.g., tokens or sequences, apart from each figure 1: evaluating the representation quality with respect to both isotropy := 1 -intra-similarity and discrimination := 1-intra-similarity / inter-similarity, where intra-similarity refers to the average cosine similarity between tokens from the same sequence and inter-similarity is defined with respect to tokens from two randomly sampled sequences. we use wit @xcite and code search dataset @xcite to evaluate gpt2 and codegen models, respectively. a natural question arises as to how would the improved representations affect the generation ability of clms. towards addressing this, we assess contraclm on language generation tasks in different domains, where we achieve better mauve @xcite ) on text generation and 9% relative improvement on pass@1 accuracy on hu-maneval @xcite . the improvement in code completion is indeed significant as it reflects that more model-generated programs pass a suite of test cases. on the discriminative tasks, contraclm attains 44% relative improvement on semantic textual similarity tasks and 34% on code-to-code search tasks, which largely bridges the gap with the encoder-only or encoder-decoder models (see section 4.4 and appendix d.1). such improvements allow us to boost the performance of decoder-only models on a wide range of discriminative tasks where encoder-only models are currently the workhorse. ## related work anisotropic representation of language models despite the remarkable success achieved by language models @xcite @xcite @xcite , they suffer from the anisotropy issue where the representations are distributed into a tiny cone in the vector space @xcite @xcite . in particular, @xcite shows that the degeneration is severer on clm, where the average cosine similarity between two words sampled from randomly selected sequences is almost at one when evaluating the outputs from the last hidden layer of @xcite . however, @xcite show that clms @xcite are indeed coherent as long as the model is larger than a certain size. we find such conclusions can vary across domains, e.g., when pretraining on code, codegen @xcite consistently suffers from the anisotropy issue over a wide range of model sizes. on the bright side, figure 1b shows that contraclm can effectively improve the representation quality when we continue to train the existing clms with our proposed objectives, regardless of whether the clms suffer from inferior representations initially. contrastive learning contrastive learning @xcite has seen remarkable successes in natural language processing (nlp). a large amount of research has focused on sentence representation learning for encoder-only models, with the main differences lying in how the augmentations are generated @xcite @xcite @xcite @xcite . recently there is an emerging interest in developing effective contrastive learning approaches for text generation models. however, most existing work mainly focuses on the encoder-decoder structure @xcite @xcite ) by contrasting suboptimal model generations obtained via diverse sampling @xcite or adding perturbations on the embedding space @xcite , against the ground truth. on the other hand, it is not intuitive to develop an effective contrastive learning strategy for decoder-only models. a recent work @xcite proposes simctg, a token-level contrastive learning approach that aims to separate each token apart from others within the same sequence by a predefined distance. as shown in section 4, our temperature-based token-level contrastive learning approach, contraclm-tok, consistently outperforms simctg across different tasks. we conjecture that the fixed margin-based objective allows less flexibility for the token-level representation separation, especially considering how the semantic relevance among tokens can vary across contexts (sequences). code generation and beyond language modeling for source code is a fast growing area of research. various model architectures have been explored recently, including encoder-only @xcite , encoder-decoder @xcite @xcite , and decoder-only models @xcite @xcite . among them, the decoder-only models have been found to be effective for code generation. however, as shown in section 4.3.2 and appendix d.1, they suffer from unsatisfactory performance on various discriminative tasks @xcite @xcite . this motivates us to improve the decoder-only models on the discriminative tasks to extend their main usage beyond language generation. furthermore, code is fundamentally different from natural language in that it is more structured, which helps validate the generalization of our approach beyond plain text. ## experiments to demonstrate the effectiveness of our proposed framework in different application domains, we evaluate our models and baselines on natural language and programming language tasks. we design our experiments to address -(1) does contrastive learning improve the discrimination ability of representations? (2) do the representations learned by contrastive learning lead to better performance on language generation tasks? (3) is the joint contrastive learning at both token-and sequence-level necessary, and how do they bene-fit from each other? (4) how does the impact of contrastive learning vary across language domains? ## conclusion in this paper, we present contraclm, an effective contrastive learning framework to resolve the representation degeneration issue of clms trained with the autoregression objective. we assess the effectiveness of contraclm on various downstream tasks in both the natural language and code domains, where we attain significant improvements on both discrimination and generation tasks. while we explored only the decoder-only clms, our proposed contrastive learning framework can serve as a drop-in term for encoder-decoder, encoder-only, or prefixlm models also. we leave these explorations as future work.