Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training (2025)

Abstract: Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous agents typically rely on complex prompting or extensive fine-tuning, which often fails to introduce new capabilities while preserving strong generalizability. We introduce Hephaestus-Forge, the first large-scale pre-training corpus designed to enhance the fundamental capabilities of LLM agents in API function calling, intrinsic reasoning and planning, and adapting to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data encompassing 76,537 APIs, including both tool documentation to introduce knowledge of API functions and function calling trajectories to strengthen intrinsic reasoning. To explore effective training protocols, we investigate scaling laws to identify the optimal recipe in data mixing ratios. By continual pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale open-source LLMs and rivals commercial LLMs on three agent benchmarks, demonstrating the effectiveness of our pre-training corpus in enhancing fundamental agentic capabilities and generalization of LLMs to new tasks or environments.

## introduction large language models (llms) are rapidly evolving beyond traditional natural language processing tasks @xcite @xcite , demonstrating increasing intelligence and autonomy by exhibiting capabilities in perception, reasoning, planning, and action within complex real-world environments @xcite @xcite . through well-crafted prompting or extensive post-training, llm-based autonomous agents augmented with external tools (e.g., apis) have demonstrated exceptional instruction-following capabilities in a wide range of tasks @xcite @xcite . despite their remarkable task-specific performance, existing llm agents often face the following challenges: (1) overemphasis on instruction fine-tuning while ignoring the pre-training stage. llms typically undergo a two-stage training process: pre-training to learn general knowledge and instruction fine-tuning to align to specific tasks and user preferences. the superficial alignment hypothesis @xcite @xcite posits that llms acquire most of their knowledge during pre-training, which is more important than instruction fine-tuning in terms of obtaining generalizable fundamental capabilities. however, the majority of existing agent frameworks (figure 1 ) focus on instruction finetuning to align with specific patterns or formats, rather than fundamentally enhancing model knowledge or capabilities (e.g., api function calling). agent instructions and trajectories significantly differ from general instructions and responses @xcite . thus, function-calling knowledge is difficult to derive directly from web archives, the primary pre-training data source. this notable lack of agent-specific pre-training corpora constrains llms from effectively acquiring new agentic knowledge and capabilities ( to address these challenges, we introduce hephaestus-forge, a large-scale pre-training corpus specifically designed to enhance the fundamental capabilities of llm agents in api function calling, intrinsic reasoning and planning, and adaptation to environmental feedback. specifically, we focus on two primary objectives: (a) improving comprehension of individual function calls, and (b) strengthening intrinsic reasoning capabilities for solving problems requiring multiple function calls. to enhance (a) comprehension of api functions and alignment with their formats, we collect a large-scale dataset of tool documentation tailored for llm pre-training on api function calls. given the expanding range of tasks with growing complexity, we incorporate a vast number of function calling trajectories to improve (b) intrinsic reasoning abilities in sequencing api function calls. we then integrate this meticulously curated tool documentation and function-calling data with code (to bolster reasoning capabilities) and text data (to maintain robust text generation capabilities), creating a multi-source, large-scale, and high-quality training corpus, hephaestus-forge. building upon hephaestus-forge, we introduce a continual pre-trained open-source llm, hephaestus, an llm with strong agentic and autonomous capabilities across domains, bringing open-source models closer to the capabilities of commercial llms. our empirical evaluations demonstrate that hephaestus-8b outperforms open-source llms at small to medium scales (e.g., 9.6% over llama-3-8b and 17.6% over mixtral-8x22b) and performs comparably to apibased large commercial llms (e.g., 18.9% over claude-3-haiku and 4.1% over gpt-3.5-turbo) across three agent benchmarks. our large-scale ablation studies further demonstrate the effectiveness of retrieved agent data in scaling up and diversify-ing the coverage of scenarios in pre-training. our contributions can be summarized as follows: instruction finetuning-based llm agents. considering the extensive diversity of apis and the complexity of multi-tool instructions, tool learning inherently presents greater challenges than natural language tasks, such as text generation @xcite . post-training techniques focus more on instruction following and aligning output with specific formats @xcite @xcite , rather than fundamentally improving model knowledge or capabilities. moreover, heavy fine-tuning can hinder generalization or even degrade performance in nonagent use cases, potentially suppressing the original base model capabilities @xcite . ## hephaestus-forge to scale and diversify the pre-training corpus for llm agents, we introduce a three-stage construction process for hephaestus-forge (see figure 2 ): ## scaling laws for data composition when designing llms, the scaling law @xcite is an important predictive tool that can estimate the performance (e.g., benchmark loss) of a large-sized target model using a scaling curve fitted over much smaller models (referred to as sampling models). we develop scaling laws to determine the optimal data proportion among agent data, text data, and code data. with the total budget of the data volume fixed, our scaling law experiments show that the effect of agent data ratio x on the loss l of a pre-trained model follows power laws: following @xcite , we leverage the benchmark loss of nexus @xcite , api-bank @xcite , @xcite to monitor the agent capabilities, and mmlu @xcite to monitor the general capabilities of llms. optimal data mixing ratio. figure 3 illustrates that the optimal mixture of agent data within the entire pre-training corpus is approximately 36%, indicating that the proportion of agent data, text data, and code data should be roughly 1 : 1 : 1. this balanced distribution promotes both specialized agent capabilities and general language understanding, ensuring that the model remains versatile and robust across diverse tasks and domains. remark. the established scaling laws provide critical insights into the data composition for pretraining llm agents. by identifying the optimal ratio of agent data, we ensure that the model effectively balances specialized agentic capabilities with general language proficiency. fine-tuning to follow general instructions and downstream task requirements ( § 6.2). stage i infuses general agent knowledge, accompanied by commonsense knowledge and code snippets. we pre-train hephaestus on the entire hephaestus-forge, whose data distribution is carefully balanced between general corpus and agent-specific data, facilitating a smooth and gradual integration of agent knowledge. ## conclusion in summary, hephaestus-forge and hephaestus collectively advance open-source llm-based autonomous agents by addressing critical gaps in pretraining corpora. through exhaustive scaling law experiments, we identify an empirically optimal data mix ratio of approximately 1:1:1 for agent, code, and text data, maximizing the fundamental and generalization capabilities of llm agents. empirical evaluations underscore the efficacy and validity of hephaestus-forge in fostering enhanced fundamental agentic capabilities and superior generalization in llm-based autonomous agents.