Claim Verification Using a Multi- GAN Based Model (2021)

Abstract: This article describes research on claim verification carried out using a multiple GAN-based model. The proposed model consists of three pairs of generators and discriminators. The generator and discriminator pairs are responsible for generating synthetic data for supported and refuted claims and claim labels. A theoretical discussion about the proposed model is provided to validate the equilibrium state of the model. The proposed model is applied to the FEVER dataset, and a pre-trained language model is used for the input text data. The synthetically generated data helps to gain information that improves classification performance over state of the art baselines. The respective F1 scores after applying the proposed method on FEVER 1.0 and FEVER 2.0 datasets are 0.65+-0.018 and 0.65+-0.051.

## introduction misleading claims and news are becoming pervasive in our lives. sometimes these are extremely difficult to identify. as a result, they can cause serious problems. this makes the research on claim verification essential. fake news can be broadly classified into three categories @xcite : i) serious fabrications (uncovered in mainstream or participant media, yellow press or tabloids); ii) large-scale hoaxes; and iii) humorous fakes (news satire, parody, game shows). to solve this problem, research on this subject has evolved from knowledge-base oriented methods to sophisticated deep learning-based techniques. ## proposed methodology as described above, three gan units are used. these units are responsible for generating positive samples equation 1, negative samples equation 2 and class labels equation 3. algorithm 1 details the training of the generators and discriminators. the proposed model can handle only supported and refuted claims. d y is trained with both supported and refuted claims, while d p and d n are trained with only supported and refuted claims separately. hence, d y is a more powerful discriminator compared to d p and d n . there is a possibility that d p or d n will assign some sentences generated by g p and g n wrongly. as d y has the global view of both supported and refuted claims, it is better able to classify them. consider a situation: g p generates y p (a synthetic positive claim). in the next step, y p is the input to g y , and g y is generating 1 (positive class label). the output of g y and input of g p is the input to the discriminator state (d y ). if d y classifies y p as real, then no penalty is incurred by g y and g p otherwise both g p and g y are penalized. consider another situation, where g y generates 0 (negative class label) for an input of y p and d y also classifies the y p as fake, then a penalty will be added to g p , not g y . so d y is acting as a global discriminator. equation 4 is the loss function for the generator g y , where π p and π n are the probabilities of positive and negative claims in the dataset. for a gan, achieving equilibrium is very important. in the present context, to find the equilibrium condition, first, we need to find the optimal conditions for discriminators. using the optimal conditions of the discriminators, the minimization conditions for the generator can be obtained. considering the generators (g p , g n , g y ) are fixed, and π p and π n are the probabilities of positive and negative claims in the dataset, at the equilibrium condition the distribution of positive generated data (p gp (x)) and negative generated data (p gn (x)) will follow the equations 5 and 6, where p p (x) and p n (x) are the positive and negative class probability distributions. the optimal discriminator functions d * p (x), d * n (x), d * y (x) can be derived by differentiating equations 1, 2 and 3 @xcite . using jensen-shannon divergence (jsd) (fuglede and topsoe, 2004), we can show that the argmin generators are achieved when the following conditions are satisfied: in most cases, the first few lines of a particular wikipedia page are taken in fever dataset as the evidence. fever training subset has 80,035 supported claims, 29,775 refuted claims, and 35,639 nei claims. the fever 1.0 validation set and test set have 3,333 supported claims, 3,333 refuted claims, and 3,333 nei claims respectively. fever 2.0 has 391 supported claims, 396 refuted claims, and 387 nei claims respectively. for the experiments, we used only supported and refuted claims. ## experiments the workflow of the experiment is given in fig 2 . in the first phase, data is preprocessed as described in section 4.1. this preprocessed data is used as input to the proposed model for training. the supported claim, evidence pairs are input to the positive synthetic data generator subunit, and the refuted claim, evidence pairs are input to the negative synthetic data generator subunit. once the proposed model is trained with the preprocessed data, the model is used for the testing phase using the test dataset. finally, the model's performance is compared with the results of other standard methods and sota models. the steps of the experiments are detailed below. ## results all models are trained with the fever training dataset and tested with fever 1.0 and fever 2.0 test dataset. in tables 2, and 3 detailed results for each of the models are presented. each experiment is repeated five times. the result for fever 1.0 is also compared with previous research work by @xcite . in tables 2, and 3 we see that the f1 score for the proposed method is better than the new baselines and previous research. the gradual change of precision, recall, and f1 score for the fever 1.0 and fever 2.0 is presented in fig. 3 , and fig. 4 . moreover, to visualize the distribution of original and synthetic data, t-sne plots of the positive and negative generated data are shown in figures 5 , and 6 . the perplexity of the t-sne plot is 30, and the learning rate is 120. it can be observed that the distribution of synthetically (a) positive (b) negative (c) label figure 7: different losses (a) cosine (b) manhattan (c) euclidean figure 8: similarity scores for positive data (a) cosine (b) manhattan (c) euclidean figure 9: similarity scores for negative data ## conclusion we propose a multiple gan-based model that employs the gan's synthetic data generation capability to solve claim verification problems. the model generates synthetic data for supported, refuted claims and their class labels using three separate generator discriminator pairs. the synthetic data eventually helps in the fact-checking task for fever 1.0 and fever 2.0 test datasets. the results have shown that the proposed model starts with random data generation, and as the training progresses, it generates synthetic data similar to the original data. different statistical and analytical similarity metrics confirm that the similarity between original data and synthetically generated data increases as the training progresses. this gradual improvement of data quality shows the effectiveness of the model. the proposed model produces an f1 score of 0.65 ± 0.018 and 0.65 ± 0.051 for fever 1.0 and fever 2.0, respectively. dataset quality is a subtle issue, e.g., see @xcite . in the future, this model can be extended to a multi-class classifier, and a similar set of experiments can be carried out on other publicly available standard datasets to test this proposed model's effectiveness across different datasets.