What Does BERT actually Learn about Event Coreference? Probing Structural Information in a Fine-Tuned D utch Language Model (2023)

Abstract: We probe structural and discourse aspects of coreferential relationships in a fine-tuned Dutch BERT event coreference model. Previous research has suggested that no such knowledge is encoded in BERT-based models and the classification of coreferential relationships ultimately rests on outward lexical similarity. While we show that BERT can encode a (very) limited number of these discourse aspects (thus disproving assumptions in earlier research), we also note that knowledge of many structural features of coreferential relationships is absent from the encodings generated by the fine-tuned BERT model.

## introduction the advent of large language models (llms) has drastically improved performance in the field of natural language processing (nlp) on a large variety of tasks that require thorough syntactic and semantic knowledge @xcite . however, discourse-based tasks, which typically require a deeper understanding of long-distance semantic relationships and dependencies within a given text, remain a tough nut to crack. one of such tasks, event coreference resolution (ecr), aims to determine whether or not two textual events refer to the same real-life or fictional event. while transformer-based architectures have been moderately successful in tackling this problem @xcite , much work remains to be done, especially in lowerresourced language domains. consider the two examples below, which have been taken from a collection of dutch (flemish) newspaper articles: ## experimental setup in our experiments we aim to evaluate a fine-tuned bert model's performance on a set of auxiliary probe tasks in order to gauge whether the model can indirectly encode discourse properties that have proven important predictors in feature-based ecr. ## results and discussion for the classification probe tasks we establish that the fine-tuned model outperforms the baseline pre-trained model in both the prominence and sentiment matching tasks, while showing no improvement when it comes to realis matching. this indicates that by fine-tuning, the bert model does implicitly learn some basic information regarding document structure and can differentiate between the importance of events within a given document and use this information for the classification of coreferential relations between events. while the improvement in the sentiment task is minor, results for prominence show significant improvement over the baseline, showing that the prominence of two events can be a component to consider for future studies in ecr. conversely, the realis and sentiment properties seem to be not directly related to the correct classification of coreferential events within this model. to get a more complete picture of the models' layer-by-layer performance we also calculate spearman's correlation coefficients over different layer performances. correlation coefficients on the promincence (0.146 & 0.720), realis (0.914 & 0.748) and sentiment (0.637 & 0.851) tasks indicate no significant changes in layer performance for the baseline and fine-tuned models as, overall, for all tasks performance increases towards the higher layers. for the regression tasks we see that final layer performance improves for the event distance task in the fine-tuned model, albeit only slightly. it should be noted, though, that the rsme for both tasks is very high, leading us to believe that no significant knowledge regarding event or sentence distance is encoded within the fine-tuned coreference model. similarly to the classification tasks we also calculate spearman's correlation coefficients for the performance on both regression tasks over different layers, showing again no different trends for the ed (0.34 & 0.38) and sd (0 & -0.048) tasks for the baseline and fine-tuned models, respectively. finally, as the raw rmse result scores from the pairwise distance probes are hard to interpret without context, we also compare the rmse for the sd and ed tasks on each layer for both coreferring and non-coreferring mentions to see if the fine-tuned model has implicitly learned something about event and sentence distances in within-document contexts for individual class labels. ## conclusion in this paper we devised a set of rudimentary probes to determine if a fine-tuned dutch bert event coreference model can learn a set of basic characteristics regarding the nature of coreferential relations. we show that the fine-tuned bert model can in fact encode a limited number of these properties. this goes against previous findings that event coreference resolution in transformer-based models is entirely based on outward lexical similarity, rather than the proper discourse mechanisms governing coreferential relations in natural language @xcite . in future research, we aim to further investigate and integrate structural and discourse aspects of coreference in llms, which will hopefully lead to more stable, interpretable and better performing ecr models.