BOUN at S em E val-2021 Task 9: Text Augmentation Techniques for Fact Verification in Tabular Data (2021)

Abstract: In this paper, we present our text augmentation based approach for the Table Statement Support Subtask (Phase A) of SemEval-2021 Task 9. We experiment with different text augmentation techniques such as back translation and synonym swapping using Word2Vec and WordNet. We show that text augmentation techniques lead to 2.5% improvement in F1 on the test set. Further, we investigate the impact of domain adaptation and joint learning on fact verification in tabular data by utilizing the SemTabFacts and TabFact datasets. We observe that joint learning improves the F1 scores on the SemTabFacts and TabFact test sets by 3.31% and 0.77%, respectively.

## introduction recognizing textual entailment (rte) @xcite is one of the core nlp problems for understanding the semantic relations between words and sentences, which is useful for other tasks including question answering (abacha and demner-fushman, 2019), text summarization @xcite , and text classification @xcite . for the rte task, datasets of various sizes @xcite and from different domains @xcite have been introduced. however, these works and datasets are solely focused on textual data without considering structured data such as tables. recently, question answering @xcite ) and textual entailment datasets (wenhu @xcite for tabular data have been introduced. semeval-2021 task 9 addresses the problem of statement verification 1 the grammatical error exists in the given dataset. distance statistics between buildings of ancient buildings and modern buildings to the main water channel (unit: meter). index ancient building ## datasets we use two different table-based fact verification datasets for the experiments: semtabfacts @xcite and tabfact (wenhu chen and wang, 2020). we compare semtabfacts and tabfact in terms of the average size of the tables, average word length of the statements, and the number of examples for each class in semtabfacts @xcite : this dataset consists of tables from articles published in elsevier, which are available on sciencedirect. after filtering complicated examples, five entailed and five refuted statements about these tables are generated by high-quality crowd-sourcing. these statements are further verified by additional crowdsource workers, especially for filtering out ungrammatical sentences. to increase the quality level, @xcite further verified the statements in the development and test sets. the semtabfacts dataset also contains automatically generated statements and unknown classes in the development and test sets for the fact verification and evidence finding tasks. in this study, we target two-way (entailed / refuted) classification without automatically generated statements for the fact verification task. semtabfacts releases tables and statements in xml format. we convert these tables into csv format to properly use in our models. due to cells with multirow and multicolumn features in xml, we could not accurately convert all tables into csv, which might affect our models' overall performance. we manually checked the xml to csv conversion of 50 tables. we identified three errors related to multirow and multicolumn features, and one error that causes a missing column. tabfact (wenhu chen and wang, 2020): this dataset crawls tables from wikipedia articles following previous works on table question answering @xcite . designed input parameters during experimental set-up. experimental bottle id material type material amount, g a inoculum 200.2 b inoculum 200.4 1a inoculum + olive cake 200.4 1b inoculum + olive cake 199.7 augmentation methods sentence original 199.7 is the lowest inoculum compare to all others. wordnet 199.7 is the small inoculum compare to all others. word2vec 199.7 is the lowest inoculum comparisons to all others. back translation 199.7, compared to others is the most low inoculum. figure 2 : generated sentences by different text augmentation methods for the same statement. the table for the original statement is given above with some modifications. complicated tables including multirows, multicolumns, and latex symbols, and large tables with more than 50 rows or 10 columns were filtered out. amazon mechanical turk was used to generate simple and complex statements about tables. the mechanical turk workers also filtered out poor statements that have grammatical errors or vague claims. finally, annotator agreement scores were computed by having the same set of statements labeled by another set of mechanical turk workers. ## experimentation and results we conduct two different experimental setups to compare our results. in both experiments, we finetune all layers of a pretrained tapas model and its classifier head. first, we finetune the tapas model on the semtabfacts dataset with all combinations of different augmentation techniques. second, instead of using augmentation techniques, we finetune the tapas model on tabfact only and then on semtabfacts and tabfact jointly and compare the results on the test sets of tabfact and semtabfacts. in all these experiments, we use the adamw @xcite optimizer with a 5e -5 learning rate and 0.01 weight decay. we set batch size as 8 with 2 accumulation steps, and the number of steps used for linear warm-up is 100 in semtabfacts training and 2000 in tabfact and joint training. we finetune this model over 10 epochs and decide the best model based on the development set. we use the official evaluation metric, which is the macro-average of f1 scores over the tables. in the augmentation steps, we include new augmented statements for each statement and augmentation method to the training data of semtabfacts. the original versions of the development and test sets are used without any augmentation. we observe that different augmentation techniques in semtabfacts can improve f1 scores on the test set as shown in in we further analyzed the errors in terms of table sizes (number of rows x number of columns) and length of the statements. however, our results indicate that there is no significant difference in f1 scores for different table sizes and different lengths of statements. our models have similar performance for small and large tables as well as for short and long statements. ## conclusion in this work, we described our models for the table statement support subtask (phase a) of semeval-2021 task 9. our base model relies on the recently introduced pretrained transformer architecture for tabular data, tapas. we proposed three different augmentation techniques which are based on word-net, word2vec, and back translation. we showed that all combinations of these augmentation techniques except back translation perform better on the test set than methods without augmentation. furthermore, we investigated the domain adaptation and joint learning capabilities of semtab-facts and tabfact. we showed that our best model in terms of development and test f1 for semtab-facts occurs when we trained tapas jointly on the semtabfacts and tabfact datasets. additionally, we illustrated that the joint model achieves better results on the tabfact test set than the model trained only on the tabfact training dataset. as future work, we plan to focus on better preprocessing the semtabfacts dataset and more diverse augmentation techniques by integrating perplexity scores of augmented statements.