HMCL : Task-Optimal Text Representation Adaptation through Hierarchical Contrastive Learning (2025)

Abstract: As general large language models continue to advance, their real-world adaptation through effective fine-tuning remains a significant challenge. We introduce Hierarchical Multilevel Contrastive Learning (HMCL), a new contrastive learning framework that improves task-specific text representation for general models. HMCL integrates 3-level semantic differentiation (positive, weak-positive, and negative) and unifies contrastive learning, pair classification, and ranking objectives into a cohesive optimization strategy. HMCL demonstrates exceptional results across multi-domain and multilingual benchmarks, including text similarity, retrieval, reranking and Retrieval-Augmented Generation (RAG) tasks. It outperforms top unsupervised methods and supervised fine-tuning approaches while maintaining broad compatibility with architectures ranging from BERT to Qwen, 330M to 7B. In real-world merchant consultation scenarios, HMCL shows a 0.70-6.24 point improvement over original fine-tuning methods in large-scale base models. This establishes HMCL as a versatile solution that bridges the gap between general-purpose models and specialized industrial applications.

## introduction from the past to the present, text representation has played a fundamental role in natural language processing @xcite . especially in the era of large language models, as the most crucial part of the rag process, knowledge retrieval and reranking demand better text representations in specific fields @xcite , which determine the accuracy and completeness of the answers generated by large language models (llms). additionally, it is also central to unsupervised classification, evaluation, and clustering @xcite . in summary, the quality of text representation significantly affects the performance of all fields related to language understanding in artificial intelligence. nowadays, with the continuous growth of pretrained model parameters and the expansion of public datasets, more general models are emerging to address the challenges in text representation. in recent years, successful general base models in this field have become a focal point of research, achieving top scores in public evaluation benchmarks @xcite @xcite . however, even with the success of general models, real-world text representation applications often require modifying these models to better match the specific standards, special expressions, and styles of each distinct task. this is because the definitions of what answers a query, what is considered similar, partially acceptable, or unacceptable can vary across different tasks. to illustrate this, we provide an analysis of seven prominent datasets across semantic textual similarity (sts), retrieval, and reranking tasks in sts measures sentence similarity with 0-5 scores, but sts-b and sick-r have starkly different standards: sts-b rates positive and negative forms of the same sentence as dissimilar (1.6), while sick-r assigns a "very similar" score (4.2). their tolerance for subject specificity (e.g., @xmath0, @xmath1. in reranking, positive pairs involve deeper associations rather than surface-level semantic similarity. for example, in mindsmallreranking, positives are based on user preferences, making connections of language and knowledge vague, while in stackoverflowdupquestions, positives must strictly follow question-answer relationships. in retrieval, dataset differences extend beyond language style and knowledge domains. scifact requires only a detail in long passages to answer queries; fiqa2018 expects retrieved passages to serve as ## (relevant) hotpotqa what science fantasy young adult series, told in first person, has a set of companion books narrating the stories of enslaved worlds and alien species? the hork-bajir chronicles...the book is introduced by tobias, who flies to the valley of the free hork-bajir, where jara hamee tells him the story of how the yeerks enslaved the hork-bajir... similar phenomena are more widespread in realworld industrial applications. therefore, powerful general-purpose models still need to be adapted to specific tasks, learning the language style and labeling standards of those tasks, to ensure optimal performance in specific scenarios. we introduce hmcl, a novel framework for task-oriented fine-tuning of text embedding models. hmcl extends traditional positivenegative contrastive learning by incorporating multi-hierarchical comparisons, excelling in sentence similarity, reranking, retrieval and rag tasks. it performs effectively across various base models, including simcse, e5, gte, bge, and large language models like qwen-gte. hmcl consistently outperforms other contrastive learning methods, whether unsupervised or supervised, such as e5's distilled methods @xcite , gte's improved contrastive loss @xcite , or bge's ann-style sampling strategy @xcite . notably, hmcl achieves superior results with small-scale datasets and low computational costs, addressing the challenge of acquiring task-specific fine-tuning data while preserving the model's general capabilities. relevant materials are available in the project repository: https://github.com/antgroup/hmcl-merchantquestion. tors @xcite @xcite @xcite . however, long paragraphs, extensive word dictionaries, and complex contextual meanings in texts demanded more accurate and efficient ways to match relevant passages and sentences. with the advent of pre-trained language models, contrastive learning based on these models has emerged as the most effective method for acquiring semantic representations. this learning framework gained popularity due to simclr, which achieved remarkable results in image classification @xcite . by aligning similar pairs and distinguishing unrelated ones during training, this framework successfully generates distinct and meaningful vectors for texts. in the field of text representation, numerous works have proposed ways to enhance positive pairs (such as cert, consert) or negative pairs (such as sim-cse, esimcse) to improve the effectiveness of contrastive learning @xcite @xcite . with the development of comprehensive labeled datasets, many general models trained on very large-scale corpora have become key solutions for text representation. examples include sentence-t5, e5, instructor, gte, bge, and even using llms as text encoders @xcite @xcite @xcite . to some extent, the com-petition among these general models has shifted to a race for larger data and more extensive training resources. from the era of simcse to the current era of llms, the number of training pairs has increased from millions to billions, leading to a significant improvement in performance on public benchmarks such as senteval, mteb and cmteb @xcite @xcite . when general base models achieve excellent performance in text embedding benchmarks, taskspecific fine-tuning is still needed to help them achieve optimal performance in real-world industrial scenarios. although these general models are typically trained using a multi-stage training framework and incorporate improved methods for the second or third training stages, there is still limited research focused on developing effective methods to fully leverage the potential of all kinds of base model to learn the standards of a specific task and perform exceptionally well in that actual scenario. ## conclusion we propose hmcl, a novel fine-tuning framework that enhances task-specific text representations. by integrating 3-level semantic differentiation and unifying multiple learning objectives, hmcl achieves outstanding performance across diverse benchmarks and various model architectures. it demonstrates significant gains in realworld applications, such as a merchant consultation, bridging the gap between general-purpose models and specialized tasks.