AL in F i K : Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation (2025)

Abstract: Large Language Models (LLMs) heavily rely on high-quality training data, making data valuation crucial for optimizing model performance, especially when working within a limited budget. In this work, we aim to offer a third-party data valuation approach that benefits both data providers and model developers. We introduce a linearized future influence kernel (LinFiK), which assesses the value of individual data samples in improving LLM performance during training. We further propose ALinFiK, a learning strategy to approximate LinFiK, enabling scalable data valuation. Our comprehensive evaluations demonstrate that this approach surpasses existing baselines in effectiveness and efficiency, demonstrating significant scalability advantages as LLM parameters increase.

## introduction large language models (llms) @xcite @xcite are transforming fields such as natural language processing @xcite @xcite and automated reasoning @xcite @xcite . their effectiveness is closely tied to the quality of the training data. in real-world scenarios, model owners and data providers often have distinct objectives. model owners aim to maximize model performance while minimizing costs, including data acquisition costs and training costs @xcite . conversely, data providers seek fair compensation for their contributions @xcite . third-party data valuation: our research presents a third-party data valuation algorithm that serves the needs of both model owners and data providers. by identifying high-value data subsets from the data provider and delivers them to model owners for training @xcite . this process helps reduce data acquisition and training costs for model owners while ensuring data providers receive compensation based on the value their data contributes. for model owners, the platform efficiently selects the most impactful data from large datasets, minimizing acquisition and computational costs. model owners can achieve improved performance with fewer resources by focusing on high-value data @xcite . for data providers, the platform offers a transparent and quantitative method for valuing data, allowing them to monetize their contributions based on the proven impact on model performance @xcite . this promotes a fairer data marketplace @xcite and encourages the creation of high-quality, relevant datasets. challenges in third-party data valuation: we identify three major challenges in third-party data valuation for llm training. (1) future influence estimation: because the goal is to perform pretraining data selection, our approach requires thirdparty platforms to estimate the potential impact of training data on a model's future performance before the training process begins, which presents significant challenges. (2) limitation in scalability: traditional data valuation methods, developed for smaller models and datasets, often rely on computationally intensive techniques like leave-oneout training or influence functions @xcite . these become prohibitively expensive or infeasible when applied to llms and their massive training datasets, due to non-linear scaling of computational requirements. (3) data contamination concerns: there's a scarcity of quantitative methods to evaluate how data valuation approaches help llms detect valuable, unseen information. current methods often focus on fine-tuning well-trained llms, raising concerns about data contamination from pre-training exposure. our proposal: learning to approximate linearized future influence kernel (linfik). in this paper, we propose a novel approach to perform data valuation for llms: the linearized future influence kernel (linfik). linfik assesses each data point's potential impact on the model's final performance using first-order approximations of training dynamics. to address the scalability issue, we introduce alinfik, a leaning-based strategy to approximate linfik scores with significantly reduced computational overhead. our approach not only provides a robust mechanism for third-party data valuation but also facilitates fair compensation for data providers based on the predicted value of their contributions. our contributions are threefold: through extensive experiments, we demonstrate that our method significantly outperforms existing data valuation approaches in terms of scalability, memory efficiency, and computational cost, particularly when applied to large llms. this work paves the way for more efficient model training and a more equitable data marketplace. ## related work the effectiveness of llms is heavily influenced by the quality and composition of the training data @xcite @xcite . recent research has increasingly focused on data quality and its impact on model performance @xcite @xcite . ## conclusion in this paper, we first propose linfik, and prove its capability of predicting the influence of data on the model in very early stage of llm training. we further introduce alinfik, an efficient approximation of linfik. using alinfik as the foundation, we design a scalable third-party data valuation system that meets the needs of both model owners and data providers. the experimental results not only prove the effectiveness of alinfik, but also demonstrate its great potential in scalability. alinfik's performance on non-english datasets and other more complex application scenarios remains to be tested. testing llm data selection on large amounts of data is very time-consuming and resource-intensive. these experiments involve validation at different checkpoints of the model and many rounds of llm re-trainings. besides, our current experiments are conducted on free public datasets. the specific strategy of using alin-fik scores to set prices for private data needs further research. calculating data influence on private datasets may also raise privacy concerns.