Students Who Study Together Learn Better: On the Importance of Collective Knowledge Distillation for Domain Transfer in Fact Verification (2021)

Abstract: While neural networks produce state-of-the- art performance in several NLP tasks, they generally depend heavily on lexicalized information, which transfer poorly between domains. Previous works have proposed delexicalization as a form of knowledge distillation to reduce the dependency on such lexical artifacts. However, a critical unsolved issue that remains is how much delexicalization to apply: a little helps reduce overfitting, but too much discards useful information. We propose Group Learning, a knowledge and model distillation approach for fact verification in which multiple student models have access to different delexicalized views of the data, but are encouraged to learn from each other through pair-wise consistency losses. In several cross-domain experiments between the FEVER and FNC fact verification datasets, we show that our approach learns the best delexicalization strategy for the given training dataset, and outperforms state-of-the-art classifiers that rely on the original data.

## introduction neural networks have achieved state-of-the-art (sota) performance across many natural language processing (nlp) tasks, usually in supervised settings. however, it has been shown that there are limitations to these methods caused in part by their over-fitting on statistical and lexical nuances (or artifacts) specific to a dataset @xcite , which prevent them from transferring well across domains. a key solution to this problem is to not let these models rely on such dataset artifacts and instead encode the true underlying semantics of the dataset. also, in recent years fact verification has emerged as a critical task with important societal implications @xcite . formally, the task is defined as: given a pair of claim and evidence texts, determine if the evidence supports or rejects the claim, or does not have enough information to reach a conclusion. several fact verification datasets have been proposed recently, based on realworld news articles @xcite , wikipedia based knowledge bases @xcite , fact verification websites @xcite , etc. several transformer networks @xcite based approaches @xcite have achieved sota performance on these tasks. however, as shown in @xcite @xcite , these models are also similarly affected by syntactic and lexical artifacts seen in other nlp tasks. specifically, @xcite shows the presence of such artifacts in the fact extraction and verification (fever) dataset @xcite , along with demonstrating that this limits the ability of the trained models to transfer knowledge to other similar datasets such as the fake news challenge (fnc) dataset @xcite . to mitigate this dependency on such artifacts, they proposed a data distillation (or delexicalization) approach, which replaces some lexical artifacts such as named entities with their type and a unique id to indicate the occurrence of the same artifact in claim and evidence. a key unresolved issue in this direction is how much delexicalization to apply. as indicated in previous work @xcite , delexicalization reduces overfitting. but too much delexicalization may discard critical information, e.g., replacing india with its ne label, say location, may remove contextual information about the country that is necessary for the correct classification of the claim-evidence pair. our work proposes a solution for this problem, with the following contributions: ## results ## discussion and conclusion previous work @xcite has shown that delexicalization is useful in learning domain transferable knowledge. however, the level of delexicalization suitable for each task is unclear. in this work, we provide multiple delexicalization choices to neural network models and encourage them to choose the most appropriate choice. we suspect this approach acts as regularization (through the consistency losses), as well as a form of data noise (because of the imperfect ners), which has been shown to aid in knowledge distillation paradigms @xcite . also, note that the bert models perform better than the decomposable attention (da) @xcite model in most of the cases, especially in the fnc dataset. more importantly, the crossdomain performance gain when using bert with the proposed group learning architecture is higher than that achieved by the decomposable attention model. this is likely caused by three reasons. first, bert has a considerably larger number of parameters than da. second, in applications involving text pairs, the decomposable attention model individually encodes these text pairs before using bidirectional cross attention. instead, bert combines these two stages using the self-attention mechanism that operates jointly over the two concatenated sentences (and separated with the [sep] token). lastly, bert is pretrained on massive amounts of texts related to the datasets used here, whereas da learns its parameters from scratch. further, analyzing the selection made by the gl framework for various random seeds, we observed that when trained on fever and tested on fnc, gl selects the lexicalized student, while in the other cross-domain direction, the common choice is the student delexicalized with oa-ner. we hypothesize this happens for two reasons. first, the training data in the fnc dataset is smaller (40,904 data points) compared to the fever dataset (119,197 training data points). therefore it is more prone to overfitting in the original, lexicalized form. second, as mentioned above, since the fnc dataset is derived from real-world news articles, the number of evidence sentences in the fnc is higher than fever sentences. this means that delexicalized sentences in fnc preserve enough lexical signal for training, even in their delexicalized forms. the opposite observations hold in the other direction (fever to fnc), which caused the lexicalized students to perform better. also, please note that even though we use only four student methods in our experiments to train with each other, this can be extended to any number of methods. however, the correct number of student models (and their corresponding delexicalized datasets) for a given task needs to be empirically determined. our approach demonstrates that: (a) delexicalization helps model generalization, (b) the amount of delexicalization to apply varies from dataset to dataset, and (c) it is possible to learn how much delexicalization to apply through our proposed gl architecture.