CLOWER : A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022)

Abstract: Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interactions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effective PLM CLOWER, which adopts the Contrastive Learning Over Word and charactER representations. In particular, CLOWER implicitly encodes the coarse-grained information (i.e., words) into the fine-grained representations (i.e., characters) through contrastive learning on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine-grained based PLMs without modifying the production pipelines. Extensive experiments conducted on a range of downstream tasks demonstrate the superior performance of CLOWER over several state-of-the-art baselines.

## introduction pre-trained language models (plms) have gained tremendous success in the field of natural language processing recently. as a major milestone of plms, bert @xcite and its variants @xcite @xcite have demonstrated outstanding performance on a wide variety of natural language understanding (nlu) tasks, such as sentiment analysis and machine reading comprehension tasks. the architecture of transformer @xcite is typically the foundation for these models, which models the semantic and syntactic relationships between the tokens of the entire input text and learns the contextual representations for each token. early chinese plms @xcite often take the sequences of chinese characters as the input. these models require relatively small vocabulary and learn the representations of each character from the corpus, which avoids the out-of-vocabulary problem @xcite . however, the meanings of a chinese word can be totally different from the meanings of each chinese character in the word. for example, the meaning of "小 心" (careful) can not be derived from summing the meaning of "小" (small) and "心" (heart). in general, the phenomenon of semantic gaps between coarse-grained language units and fine-grained language units (e.g., words & characters, phrases & words) exists not only in chinese but also in many other languages. to alleviate the gap, prior studies improve the pre-trained models in two directions. one direction is to enrich the masking strategies in the masked language model (mlm) objective to mask coarsegrained units, such as the whole word masking (wwm) @xcite and phrase masking @xcite . these methods encourage the pretrained model to recover the coarse-grained masks with fine-grained tokens. however, the relation between the coarse-grained and fine-grained representations is modeled in an implicit manner, leading to less effective representations. the other direction is to leverage the multi-grained tokenizations as input. ambert @xcite encodes both the fine-grained and coarse-grained token sequences and performs the masked language modeling tasks correspondingly, while lichee @xcite merges the multi-grained token embeddings explicitly to integrate the information. lattice-bert @xcite adopts the lattice graph to construct the multi-grained input. nevertheless, these mod-els require additional computational costs (e.g., tokenization, graph construction, multi-grained encoding) and the multi-grained information is only integrated in the embedding layer other than the full encoder, leading to limited usability with low effectiveness. to fully leverage the semantic information of multi-granularity and preserve the flexibility of single-grained models in the fine-tuning stage, we propose a novel plm named clower to efficiently model the multi-grained semantic information in pre-training to improve the representation capability. clower adopts the contrastive learning framework to carry out the semantic interaction between multi-grained representations. specifically, in the pre-training stage, we perform both character and word level tokenizations separately for each input sequence and feed them into the encoder to obtain the contextual representations. then we conduct the contrastive learning over character and word representations on both token-level and sentence-level. in this way, the word-level semantic information is encoded into the character tokens by bringing their representations closer. different from ambert or lichee, in fine-tuning, clower requires no additional computation and can be directly used in any fine-grained plms. the merit makes clower production-friendly since it could be deployed easily without modifying the established production pipeline. we perform comprehensive experiments on different downstream nlu tasks. the experimental results show that clower achieves considerable improvements over several baselines. ablation studies demonstrate the effectiveness of contrastive learning in our pre-training framework. our contributions are summarized as follows: ## methodology in this section, we present clower, the pretrained language model based on contrastive learning over word and character representations. we first present the overall model architecture of clower, and then we introduce its details in the pre-training stage. finally, we discuss the strategy of fine-tuning the model efficiently using only the fine-grained input. ## experiments we conducted comprehensive experiments on various chinese nlu tasks to examine the effectiveness of clower. in this section, we first introduce the details of pre-training and fine-tuning, including the datasets and experimental settings. then, we present the overall results on different tasks and conduct an in-depth analysis. ablation studies are also conducted to evaluate the impact of multi-level contrastive learning in our model. ## conclusion to fully leverage the information of characters and words in chinese plms, we propose a novel plm clower based on contrastive learning over word and character representations jointly. through the token-level and sentence-level contrastive learning in the pre-training stage, the model encodes the coarse-grained semantic information into finegrained tokens. we can not only enhance the model with coarse-grained semantics but also enjoy the flexibility of fine-grained inputs/outputs. the flexibility promises that our model could be deployed conveniently in real scenarios, where certain plms like bert have been established. comprehensive experiments on a variety of downstream natural language understanding tasks demonstrate the competitive performance of clower. we also conduct a ablation study to evaluate the multi-grained contrastive learning mechanism in clower.