Data and Model Distillation as a Solution for Domain-transferable Fact Verification (2021)

Abstract: While neural networks produce state-of-the-art performance in several NLP tasks, they generally depend heavily on lexicalized information, which transfer poorly between domains. We present a combination of two strategies to mitigate this dependence on lexicalized information in fact verification tasks. We present a data distillation technique for delexicalization, which we then combine with a model distillation method to prevent aggressive data distillation. We show that by using our solution, not only does the performance of an existing state-of-the-art model remain at par with that of the model trained on a fully lexicalized data, but it also performs better than it when tested out of domain. We show that the technique we present encourages models to extract transferable facts from a given fact verification dataset.

## introduction neural networks have matched, and in several cases even surpassed, human performance in several supervised learning problems. however, such successes come at a cost. these neural networks typically need a great deal of human support in the form of man power required for curating domain specific datasets. further, it has been shown @xcite @xcite ) that several such models depend heavily on certain statistical nuances found in these datasets, information that transfers poorly between domains. the ideal solution to this problem is the creation of models that do not rely on such statistical nuances in the given datasets, but instead encode the true underlying semantics of the task, that are in turn transferable to other domains. fact verification is the task of verifying the truthfulness of claims by estimating their assertions against credible evidences. specifically, given a pair of claim and evidence statements, they have to be classified into one of the 3 class labels, agree, disagree, or neutral. fact verification datasets, which often constitute real life news articles, have the added advantage of being used in practical problems such as fake news detection. more recently, several neural network models @xcite built on top of the transformers @xcite , have achieved excellent performance in fact verification tasks. however these methods are not devoid of the shortcomings that besiege other neural networks in natural language processing tasks. it has been shown that these approaches depend heavily on lexical artifacts that transfer poorly between domains @xcite @xcite . for example, @xcite observed that out of all the statements containing the phrase 'american author' in the fever dataset @xcite , 91% of them belonged to one class label. further, they demonstrated that neural methods put unnecessary emphasis on such lexical artifacts, which limits their transfer to other fact verification datasets such as the fake news challenge (fnc) @xcite . to mitigate the dependency on such artifacts, @xcite proposed a data distillation (or delexicalization) approach, which replaces some lexical artifacts such as named entities with their type and a unique id to indicate occurrence of the same artifact in claim and evidence. while promising, the risk of this direction is discarding too much information through the delexicalization process. for example, replacing china with its named entity (ne) type (country) in an evidence sentence discards the fact that the text is about an asian country which might be relevant in the context. in this work we propose a solution that combines data distillation with model distillation to reduce the risk of over delexicalization. in particular, we introduce a teacher-student architecture inspired from that of @xcite . in our architecture, the student model is trained on delexicalized data (to take advantage of data distillation), but is also guided by a teacher trained on the original lexicalized data (as a form of model distillation) to mitigate the possibility of discarding too much lexical information. the contributions of our work are as follows: all the software for our proposed approach is open-source and publicly available on github at: https://github. com/clulab/releases/tree/master/ naacl2021-student-teacher . suntwal et al. (2019) demonstrated that named entities are most prone to overfitting for fact verification. based on this observation, we also replace named entities with their type (and a unique id). however, unlike their work, we have observed in early experiments that more fine-grained ne types yield better models. in particular, we utilize the figer named entity recognizer (ner) @xcite to detect and replace named entities with their most specific label returned by the ner. further, we also process the text with the corenlp ner @xcite to delexicalize additional ner classes not covered by figer. we include in this list mentions of date, time, money, number, and ordinal. ## conclusion we present a new strategy to improve domain transfer of fact verification methods, which combines data distillation and model distillation. we show that the performance of existing state-of-the-art models degrades significantly on a cross-domain setting, hence motivating the necessity of robust data distillation techniques such as delexicalization to minimize overfitting on lexical artifacts. we further combine delexicalization with a teacherstudent architecture as a form of model distillation to reduce the risk of over-delexicalization. we hope that this solution will encourage the development of architectures capable of reducing the dependency of models on lexical artifacts in an effort to learn domain transferable knowledge in the task of fact verification.