On the Importance of Data Size in Probing Fine-tuned Models (2022)

Abstract: Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model’s linguistic knowledge.

## introduction the outstanding performance of pre-trained language models (lms) on many nlp benchmarks has provoked curiosity about the reasons behind their effectiveness. to this end, several probes have been proposed to explore their capacity @xcite @xcite . the investigations have clearly highlighted the ability of lms in capturing various types of linguistic knowledge @xcite @xcite @xcite . however, to take full advantage of the encoded knowledge of pre-trained models in specific target tasks, it is usually required to perform a further fine-tuning @xcite . the broad application of fine-tuning has garnered the attention of many researchers to explore its peculiarities. trying to understand the fine-tuning procedure, recent analyses have shown that most of the pre-trained linguistic knowledge is preserved after fine-tuning @xcite . furthermore, by encoding the essential linguistic knowledge in the lower layers, this procedure makes the higher layers taskspecific @xcite . however, @xcite argued that the changes in the probing performance can not be attributed entirely to the modifications a model undergoes with respect to its linguistic knowledge after fine-tuning. while the previous studies focused on the role of the target task as a factor that affects the probing performance of fine-tuned models, we present another important factor in interpreting probing results for such models. our investigations reveal that the conclusions drawn by previous probing studies that investigate the impact of fine-tuning on acquiring or forgetting knowledge might not be entirely reliable unless the size of the fine-tuning dataset is also taken into account. through several experiments, we show that the encoded linguistic knowledge can highly depend on the size of target tasks' datasets. specifically, the larger the task data, the more the probing performance deviates from the pre-trained model, irrespective of the finetuning tasks. to address the overlooked role of data size, we run several experiments by limiting training samples and probing the fine-tuned models. our results indicate that models fine-tuned on large training datasets witness more change in their encoded linguistic knowledge compared to pre-trained bert. however, by reducing fine-tuning training data size (e.g., from 393k in mnli to 7k), the gap between probing scores becomes smaller. moreover, we expand our analysis and evaluate the extent to which large training datasets affect the captured knowledge across layers. the layer-wise results show that the effect of data size is more notable on higher layers, particularly for models trained on larger datasets. we take our analysis a step further and show that the difference in probing performance among different data sizes is due to the total number of optimization steps rather than the diversity of training samples. finally, through a set of experiments, we show that the changes made to the probing performance by a fine-tuning task can be recovered if the model is re-fine-tuned on a task with comparable data size. the findings of this paper can be summarized as follows: ## related work recently, many studies have shown that pre-trained language models, such as bert @xcite , encode certain linguistic knowledge in their internal representations @xcite . for instance, @xcite found that syntactic dependencies can be obtained from bert's token embeddings, suggesting that bert encodes syntactic knowledge in its representations. nevertheless, not all layers behave similarly in capturing linguistic features: lower layers tend to encode surface-level knowledge, middle layers seem to be responsible for syntactic information, and higher layers capture semantic knowledge in their representations @xcite . while models such as bert capture considerable amounts of linguistic features, one still requires to fine-tune them to take full advantage of their potential in specific downstream tasks @xcite . fine-tuning affects bert in various ways; for instance, @xcite found that fine-tuning mainly affects the attention mode of the higher layers and alters the feature extraction mode of the middle and last layers. in addition, finetuning bert on a negation scope task improves the model's attention sensitivity to negation (zhao and bethard, 2020). apart from the changes made to bert's attention, recent work has studied how fine-tuning affects bert's representations and, as a result, its linguistic knowledge. @xcite found that fine-tuning primarily affects the representations in higher layers, and depending on the downstream task, the changes made to lower layers could be either deep or shallow. moreover, on only a small number of downstream tasks, fine-tuning seems to have a positive impact on the probing accuracy @xcite . given the fact that fine-tuning mostly affects higher layers, @xcite showed that after fine-tuning, most of the model's linguistic knowledge is transferred to lower layers to reserve the capacity in the higher layers for task-specific knowledge. studies so far have relied on probing accuracy to explain how fine-tuning affects a model's linguistic knowledge @xcite @xcite . however, given the fact that fine-tuning tasks do not share the same number of samples, concluding to what extent target tasks contribute to the model's linguistic knowledge is not fully reliable. to the best of our knowledge, none of the previous studies have considered the role of data size in fine-tuned models' linguistic knowledge. in this work, we show that the size of the dataset plays a crucial role in the amount of knowledge captured during fine-tuning. by designing different experiments, we analyze the effect of the size of the dataset in-depth. ## experimental setup we have carried out over 600 experiments to study the linguistic features captured during fine-tuning. this allows us to examine how much different factors impact performance on various probing tasks. moreover, varying the sample size lets us understand its importance in analyzing fine-tuned models. in this section, we provide more details on setups, downstream tasks, and probing tasks. ## data size analysis in this section, we first provide insight on the role of target tasks in capturing or forgetting different types of knowledge (e.g., syntactic and semantic) during fine-tuning. then, we investigate the role of datasets' size on linguistic knowledge. ## layer-wise analysis given our observations on the role of data size, we were curious to see how it affects the encoded knowledge in specific layers. as noted by @xcite , bert's layers can be divided into three classes in terms of the linguistic knowledge they capture. to this end, we carry out experiments by probing layers 2, 7, and 11-12 to cover all the three categories. regardless of data size, we can also observe that fine-tuning mainly affects higher layers. our finding is aligned with @xcite that fine-tuning has a more significant impact on higher layers and negligible effects on lower layers. there is also an interesting pattern concerning cola's performance. despite a drop in performance of around 15% from the full to 1k version ( ## fixed iteration analysis given the observations from section 5, we have realized that by training bert on larger datasets, the model's performance deviates substantially from the baseline. however, by reducing the size of training data, the gap between the fine-tuned models and the baseline decreases. this behavior can be either attributed to the diversity of training samples or to the larger number of iterations through which the model is updated. to address this, we repeated the same experiments carried out in section 5 but with fixing the number of iterations on all data sizes. this allows the model to be fine-tuned for an equal number of iterations across different data sizes of a specific task. note that we fine-tuned the full models for just one epoch to avoid a large number of iterations for the 7k and 2.5k models. since sst-2, cola, and mrpc have much smaller datasets, and the number of iterations does not substantially differ across the full, 7k, and 2.5k models, we have dropped them from this scenario. this finding is interesting because, firstly, it indicates that the high variance between baselines and full models is mainly due to the number of times their weights are updated during fine-tuning rather than the diversity of the training samples. secondly, with equal data sizes, the role of target tasks becomes less influential in the linguistic knowledge 7 linguistic knowledge recoverability fine-tuning procedure modifies the encoded linguistic knowledge in the pre-trained model. in this section, we aim at verifying the extent to which these modifications are recoverable. to this end, taking a fine-tuned model on a specific task as our baseline, we further fine-tune the model on another task. we then compare the probing performance of the resulting models with their corresponding baselines. high similarity in probing performance indicates the recoverability of the modifications. we opt for cola and sst-2 as a pair of tasks with different linguistic objectives but with comparable data sizes. also, we experiment with mrpc and qqp, which are similar tasks but with significantly different data sizes (the former's data size is a hundred times larger than the latter's). for instance, considering cola and sst-2 as our finetuning task pair, sst-2 → @xmath0, cola, and sst-2. following our previous experiments, we report the probing results for the bigram shift and semantic odd man out tasks. 5 5 the results are presented in figure 3 . the threequarters of a circle in the figures represent the maximum value in the corresponding probing task. as shown in the figures, the linguistic knowledge is recoverable through re-fine-tuning on a set of pairs with comparable data sizes. in the previous sections, we observed that cola and sst-2 have notably different performances on bigram shift and semantic odd man out. nevertheless, after re-finetuning, both target tasks can recover the knowledge modified by the previous fine-tuning step. on the other hand, for the qqp and mrpc pair, we observe a different behavior in which the data size of qqp highly limits the extent of knowledge recoverability. considering bigram shift, we observe that the final mrpc fine-tuning in the @xmath1. in the reverse setting (@xmath2, the probing performance is negligently affected by mrpc data size, leading to a performance fairly similar to qqp's. 6 our results suggest that the extent of knowledge recoverability is bound to the fine-tuning data size. more specifically, further fine-tuning a fine-tuned model with a comparable data size (e.g., sst-2 → cola and @xmath3.g., cola). however, increasing the data size in one of these tasks decreases the extent of recoverability by the other task. ## conclusion in this paper, we carried out a set of experiments to determine the effect of training data size on the probing performance of fine-tuned models. to begin with, by individually probing all layers, we found out that models fine-tuned on larger datasets deviate more from the base model in terms of their encoded linguistic knowledge. therefore, we argue that comparing the linguistic knowledge of fine-tuned models is valid only if they are trained on datasets of comparable sizes. through layerwise probing analysis, we realized that the number of training samples mainly affects the probing results for the higher layers, while the results remain similar in the lower layers across different target tasks. furthermore, we investigated why data size affects the probing performance of fine-tuned models through training the models with limited training data for the same number of iterations as we trained the full models. we showed that the gap in probing performance between models fine-tuned on different data sizes is due to the number of iterations for which the model is updated during fine-tuning rather than the diversity of the training set. finally, in our last experiment, we showed that the size of a target task's dataset affects the extent to which it can recover the linguistic knowledge previously changed by a different task. we argue that probing accuracy cannot fully represent the linguistic knowledge captured by finetuned models, given that factors, such as the size of the dataset, can highly affect probing accuracy and should be ruled out in any such study. as future work, we plan to evaluate the reliability of existing accuracy and loss-based probes and design more robust metrics for investigating the encoded knowledge in the existing language models. alex warstadt, amanpreet singh, and samuel r. bowman. 2019. neural network acceptability judgments. transactions of the association for computational linguistics, 7:625-641. adina williams, nikita nangia, and samuel bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1112-1122, new orleans, louisiana. association for computational linguistics. zhiyong wu, yun chen, ben kao, and qun liu. 2020. perturbed masking: parameter-free probing for analyzing and interpreting bert. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4166-4176, online. association for computational linguistics. yiyun zhao and steven bethard. 2020. how does bert's attention change when you fine-tune? an analysis methodology and a case study in negation scope. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4729-4747, online. association for computational linguistics.