Unlocking LLM s’ Self-Improvement Capacity with Autonomous Learning for Domain Adaptation (2025)

Abstract: Self-supervised pre-training and instruction fine-tuning demonstrate the potential of large language models (LLMs) for domain adaptation (DA). In pursuit of superhuman performance, LLMs have demonstrated significant potential in math and coding through self-improvement algorithms that rely on iterative training with self-generated data. This success stems from the clear reward signals in these environments, which provide a solid foundation for self-improvement. However, when it comes to general DA scenarios, two main challenges emerge: 1) ambiguous self-improvement reward signals and 2) lack of high-quality instruction fine-tuning datasets. This motivates this paper addresses how LLMs can adapt autonomously to new domains using only a large amount of unlabeled target corpora. Inspired by the human practice of self-reflection through open- and closed-book exercises to achieve domain generalization, we propose autonomous learning, which creates a self-improvement learning environment for DA. Here, the model generates questions from documents and conducts two explorations—one with the original document and one with a masked version. By comparing these explorations, the LLMs can independently identify and enhance its policy for reducing knowledge gaps. Experiments across various DA tasks demonstrate that autonomous learning enhances the DA performance of existing models, outperforming traditional fine-tuning and self-improvement methods. Our code is publicly available at https://github.com/FreedomIntelligence/AL.

## introduction due to the success of self-supervised and instruction tuning methods, large language models (llms) could learn from unsupervised corpora * benyou and anningzhe are the corresponding authors with email: wangbenyou@cuhk.edu.cn. the first two authors contributed to this work equally. @xcite @xcite , supervised human-annotated instruction data @xcite @xcite . recently, a series of self-improvement methods @xcite are proposed to enable llms to be trained based on its self-generated data, @xcite highlights the challenges of further aligning superhuman models, as their complex behaviors are difficult for humans to effectively supervise. since the quality of the chain of thought (cot) can be assessed by the correctness of the final answer @xcite , a series of self-training methods @xcite @xcite have been proposed to significantly improve llms' performance in math and code. however, when we try to deploy these approaches on general da scenarios, there are two main challenges that limit the advancement of this field. 1) ambiguous self-improvement reward signals: in general da problems, the signal used to compare the quality of two responses is implicit. 2) lack of high-quality instruction fine-tuning datasets: the requirement of previous methods for high-quality data further limits the potential of model self-improvement. it motivates us to study autonomous learning in a more practical da setting, where llms adapt to a new domain using only a large amount of target domain unlabeled corpora. in real-world scenarios, humans demonstrate the capacity for autonomous learning, such as self-education through reading books or independent research of scientific papers. most human learning processes are subjective and require minimal guidance, exhibiting strong autonomous characteristics. to mimic human learning, it reminds us to use autonomous learning, an ideal approach to human education. according to @xcite , it is not merely a teaching method; hence, it does not involve teachers dictating behaviors for students to replicate. in @xcite , the authors define autonomous learning as the capacity of learners to direct their own learning, implying their responsibility in shaping various aspects of the learning process. this includes critical thinking, planning, evaluating, and reflecting on learning, with learners actively monitoring the entire process @xcite . therefore, autonomous learners are reflective individuals who consciously strive to comprehend what, why, and how they learning @xcite . consequently, while autonomous learning is considered an ideal approach, modern llm training methods emphasize reliance on human-annotated data and predefined objectives when meet new downstream domain or knowledge, hindering learners' ability to monitor their learning process. this inspirs us to adopt autonomous learn-ing for llms. the core idea is to enable llms to learn autonomously, without human involvement. autonomous learning framework provide a selfimprovement environment for da, therefore, the only prerequisites are the llms itself and the learning resources, such as books or documents. the process mimics how a person learns from a book: reading to understand and closing the book to recall and identify areas that require further study to reinforce knowledge. this approach boasts several unique advantages: to assess the efficacy of this learning method, we have set up experiments with learning materials of varying scales, such as books (10k paragraphs), domain-specific documents (100k paragraphs), and wikipedia (1000k paragraphs), along with corresponding public quizzes to evaluate the learning outcomes. our experiments demonstrate that autonomous learning significantly outperforms pre-training and human-annotated sft methods, suggesting that a model that has diligently 'studied' could outperform one that has 'open-book' access but no review. we also introduce recent selfimprovement methods for comparison, and the experiments demonstrate the superiority of our al's "document in the self-improvement loop." our findings confirm that autonomous learning is a more effective learning method, and its independence from annotations and human involvement significantly reduces the complexity and effort involved in model training. the main contributions of this paper are listed as follows: ## related work in this section, we list some research directions related to this paper. it is important to emphasize that this paper focuses on how to leverage the powerful knowledge and instruction-following capabilities obtained through pre-training and sft for self-learning within the document to continuously enhance domain adaptability, rather than replacing these techniques. at the end of each part, we will discuss the limitations of each section in the context of further autonomous learning. ## preliminary we define a straightforward learning objective: given a corpus @xmath0, d 2 , . . . , d n } consisting of n documents, and a llm φ θ with parameters θ, the goal is to enable φ θ to effectively learn from this corpus. the effectiveness of this learning can be evaluated using benchmarks related to d. this process is akin to a person studying a textbook for a course and then being assessed through course exams to gauge their understanding. in our settings, al only utilizes the source-trained model and unlabeled target data to adapt to the target domain. ## methodology: autonomous learning in this section, we provide a detailed implementation of our proposed autonomous learning. the overview of our autonomous learning framework is shown in figure 1 . this process consists of two stages: stage 1. open-book learning (warmup): the model comprehends and absorbs the textual information. stage 2. closed-book learning train the model with unlabeled corpora web corpus books papers wikipedia open-book answer document (d): earth orbiting the sun causes seasons to change... unlabeled corpora web corpus books papers wikipedia question (q): what causes the change in seasons on earth? answer (a o ): the change in seasons on earth is caused by its orbit around the sun... ## experiments we evaluate our autonomous learning (al) framework across various domains, including commonsense reasoning and domain-specific qa. we compare al to traditional knowledge injection methods, assess its scalability with different dataset sizes, and its efficacy in specialized fields like medicine. we also analyze the impact of open-book and closed-book learning on performance, and evaluate al's ability of data-efficiency under different low-resource settings. ## conclusion in this paper, we explore the significant challenges associated with enabling llms to autonomously adapt to new domains by leveraging extensive unlabeled target corpora. we propose and validate autonomous learning, which innovatively introduces a self-improvement environment for da. by enabling llms to self-educate through direct interaction with diverse textual materials, this approach not only mimics human learning processes but also significantly enhances the capabilities of llms be-yond the constraints of traditional training methods reliant on human-annotated data. with the help of sufficient pre-training and sft, our results show that al outperforms all baselines without any additional human annotations.