SLIM : Subtrajectory-Level Elimination for More Effective Reasoning (2025)

Abstract: In recent months, substantial progress has been made in complex reasoning of Large Language Models (LLMs), particularly through the application of test-time scaling. Notable examples include, though are not limited to, OpenAI’s o1/o3/o4 series and DeepSeek-R1. When responding to a query, these models generate an extended reasoning trajectory, during which the model explores, reflects, backtracks, and self-verifies before arriving at a conclusion. However, fine-tuning models with such reasoning trajectories may not always be optimal. Our findings indicate that not all components within these reasoning trajectories contribute positively to the reasoning process; in fact, some components may affect the overall performance negatively. In this study, we divide a reasoning trajectory into individual subtrajectories and develop a “5+2” framework to: (1) systematically identify suboptimal subtrajectories within the reasoning trajectory based on five human-established criteria; (2) assess the independence of the suboptimal subtrajectories identified in (1) from the subsequent content, ensuring that their elimination does not compromise overall flow and coherence of the reasoning process. Additionally, a sampling algorithm, built upon the “5+2” framework, is employed to select data whose reasoning process is free from suboptimal subtrajectories to the highest degree. Experimental results demonstrate that our method can reduce the number of suboptimal subtrajectories by 25.9% during the inference. Furthermore, our method achieves an average accuracy of 58.92% on highly challenging AIME24, AIME25, AMC24 and MATH500 benchmarks with only two thirds of training data, surpassing the average accuracy of 58.06% achieved with the entire data, and outperforming open-source datasets, including s1K-1.1, Light-R1-SFT-stage-1, OpenR1-Math-94k, and OpenThoughts-114k, when fine-tuning Qwen2.5-Math-7B. Finally, we have validated the efficacy of our method under resource-constrained scenarios, where it exhibits performance improvements across different maximum inference token limits: 2k, 4k, 8k, and 16k tokens.

## introduction large language models (llms) have been rapidly evolving in their ability to tackle complex reasoning tasks. recently, in the domain of llms, reinforcement learning (rl) employing an outcomebased reward has attracted public attention, as it grants the model extensive freedom to explore, reflect, backtrack, and self-verify, a process known as test-time scaling @xcite . rl-ed llms, exemplified by @xcite , have demonstrated robust capabilities in handling complex reasoning tasks, and are consequently often used as teacher models in knowledge distillation, enhancing the reasoning capabilities in other models or cold-starting them with a test-time scaled output format through supervised fine-tuning (sft) @xcite @xcite . however, reponses generated by rled llms do not always guarantee the highest quality, as the unconstrained freedom during the rl training can introduce inefficiencies or counterproductive elements, such as prematurely abandoned steps or repetitive verifications, even within correct solutions, which will be illustrated in section 3.1. fine-tuning a model using such solutions would be suboptimal, as they could potentially decrease both the model's accuracy @xcite and thinking efficacy @xcite . naturally, the following question arises: given a set of qa pairs, where the answers are distilled from a rl-ed llm, how can we select the qa pairs that are free from these inefficiencies and counterproductive elements to fine-tune another model? to address this question, we first recall that answers from a rl-ed llm, such as deepseek-r1, typically take the form in appendix a. for a qa pair, we divide its thinking process into individual approaches, referred to as subtrajectories in later discussions. we propose five criteria: effort, effectiveness, coherence, preliminary conclusion, valid verification, to assess each subtrajectory, determining whether it contributes positively to problem-solving from a specific perspective. if a subtrajectory fails to meet a criterion, we will further assess its independence within the thinking process and determine whether it can be removed without impacting the understanding and coherence of subsequent reasoning process. after eliminating suboptimal and independent subtrajectories, we will assign a quality score to the qa pair: first, assign a score to each subtrajectory based on the five criteria; second, aggregate these scores with weights proportional to the number of tokens in each subtrajectory. in addition to data quality, our analysis reveals that the distribution of the number of subtrajectories within the dataset also influences the model's reasoning ability. accordingly, following the modification and computation of the quality scores, we develop a sampling algorithm of selecting qa pairs for supervised fine-tuning. this algorithm considers both the quality scores and the number of subtrajectories in the thinking process, achieving a balance through weights determined by the kullback-leibler (kl) divergence @xcite between the distribution of number of subtrajectories in the entire dataset and that in the sampled dataset. this approach enables the selection of efficient and productive qa pairs based on an in-depth assessment of their thinking processes, while preventing the algorithm from disproportionately favoring thinking process with fewer subtrajectories. comprehensive experimental results illustrate that our methods achieve an average accuracy of 58.92% on the highly challenging aime24, aime25, amc24, and math benchmarks, utilizing merely two-thirds of the curated training data. this performance surpasses the 58.06% accuracy obtained with the full dataset. meanwhile, the number of suboptimal subtrajectories decreases by 25.9% during the inference, which suggests a more profound and efficient reasoning paradigm. the concurrent enhancement in accuracy and thinking efficacy underscores that the responses generated from rl-ed llms indeed exhibit significant qual-ity issues, and our data quality pipeline, encompassing suboptimal subtrajectory elimination and sampling strategy, demonstrates a robust capability in mitigating these issues. in summary, our contributions are: (1) we propose a "5+2" framework to assess and modify the thinking processes generated from rl-ed llms at subtrajectory level. (2) we develop a sampling algorithm aimed at selecting efficient and productive qa pairs for supervised fine-tuning based on subtrajectory assessment. (3) we conduct comprehensive experiments and ablation studies to illustrate the effectiveness of '5+2" framework and samping strategy, which enhance both model accuracy and thinking efficacy. ## data curation in this section, we discuss the process of constructing opensourcer1-hard and deepmath-hard, the source datasets that we use for our subsequent studies. it should be noted that our hypotheses and methodologies in section 3 are both formulated and validated using opensourcer1-hard, and the deepmath-hard dataset is regarded as an out-ofdistribution test set. both datasets undergo the following filtering processes, including basic quality filtering and difficulty filtering. the construction and filtration processes of the two datasets are elaborately detailed in appendix b. we also decontaminate the collected dataset against the evaluation benchmarks mentioned in 4.1 using 15-grams. ## conclusion and future works in this paper, we conducted a comprehensive analysis of the quality of subtrajectories within the reasoning process of rl-llms. this analysis led to the identification of five critical quality issues that negatively impact both the accuracy and thinking efficacy of these models. to address these issues, we introduce a "5+2" framework to: