Quantifying Memorization in Continual Pre-training with J apanese General or Industry-Specific Corpora (2025)

Abstract: Despite the growing concern about memorization of training data using large language models (LLMs), there has been insufficient analysis under conditions using non-English or industry-specific corpora.This study focuses on continual pre-training, a common approach in building non-English LLMs, and quantifies memorization of training data.Specifically, we trained two models based on Llama 3 using Japanese Wikipedia (general) and Japanese financial news articles (industry-specific).Experiments showed a tendency for the amount of memorization to increase as training progressed, similar to the empirical findings for English.This trend was clear in the industry-specific corpus, suggesting potential risks when using valuable, non-general industry corpora.We also identified issues specific to Japanese, and emphasized the importance of analysis other than in English.

## introduction with the increasing practical use of llms, concerns about the memorization of training data have emerged @xcite . memorization refers to the phenomenon where models reproduce exact or highly similar sequences from training data. such memorization can lead to privacy and copyright infringements while reducing model generalization. regarding privacy, an early study by @xcite warned that personal information could be extracted from gpt-2 @xcite . in terms of copyright, @xcite analyzed gpt-2 and highlighted ethical concerns related to plagiarism. additionally, there is concern that llms memorizing benchmark datasets may undermine the validity of model evaluations @xcite . to address these concerns, previous research on memorization in llms has predominantly exam-* these authors contributed equally. open: training set is known. figure 1 : overview of this study. we quantify memorization of training data in llms continually pre-trained using japanese general and industry-specific corpora. in the open setting (upper), where training data is known, the training data is split into prompts and reference data, and the similarity between the generated continuation and the reference is measured. in the closed setting (lower), where it is unknown whether a given text is part of the training data, generation probabilities are used to estimate the likelihood of inclusion in the training data. with the expansion of non-english and industryspecific llms, there is an increasing need for research on memorization in such models. for the japanese, the llm-jp project @xcite was launched as a cross-organizational initiative to develop japanese llms. @xcite analyzed a gpt-2 model built within llmjp and found that empirical findings from englishlanguage studies also applied to general japanese corpora. in the industry sector, ishihara and takahashi (2024) pre-trained a gpt-2 model using articles of nikkei online edition foot_0 and confirmed that english findings on memorization were reproducible even in models trained on japanese industry-specific corpora. they also demonstrated that membership inference attacks achieved the area under the curve (auc) of approximately 0.60 in the closed setting. there is also a study that compares the performance of membership inference methods in japanese and english @xcite . however, the limited research on japanese memorization @xcite @xcite only covers models that are pre-trained from scratch ( to bridge this gap, we built japanese continual pre-trained models and analyzed the tendency to memorize training data. specifically, we fine-tuned llama 3 @xcite with @xcite using japanese wikipedia as a general non-english corpus and japanese financial news articles (nikkei online edition) as an industryspecific corpus ( §2). the experiments involve two tasks: generating a continuation of the training data (open) and detecting the training data (closed) using the two constructed models ( §3). furthermore, we discussed our findings and prospects by comparing the results of our experiment with previous research ( §4). the contribution of this paper is to quantify the memorization of the training data of llms for the first time with the setting of continual pretraining using non-english or industry-specific corpora. three main findings are as follows: ## background & problem settings this section outlines the memorization quantification framework employed in this study. we follow the systematic taxonomy proposed by @xcite and evaluate memorization under both open and closed settings as shown in figure 1 . procedure. in the open setting, where training data is explicitly available, we quantify memorization following prior studies @xcite @xcite . the procedure illustrated in figure 2 is as follows: the ideal approach would be to try multiple decoding strategies multiple times and perform statistical analysis. however, since llm inference requires significant computing resources and time, we decided to use a greedy method and generate only once in this study. ## experiments this section details our experimental setup, including datasets, continual pre-training methodology, and evaluation results. ## discussion this section discusses our findings with previous research and future research directions. ## conclusion this study is the first to systematically quantify training data memorization in continual pretraining settings using non-english and industryspecific corpora. our experiments with japanese wikipedia and nikkei online edition demonstrated that continual pre-training can significantly increase memorization, particularly when using industry-specific corpora. these findings highlight the heightened privacy and intellectual property risks associated with these corpora. in addition, we also highlighted the limitations of directly applying english-centered methods to other languages. our work underscores the necessity of language-and domain-aware memorization analysis for the safe and responsible development of llms.