QMUL - SDS at SCIVER : Step-by-Step Binary Classification for Scientific Claim Verification (2021)

Abstract: Scientific claim verification is a unique challenge that is attracting increasing interest. The SCIVER shared task offers a benchmark scenario to test and compare claim verification approaches by participating teams and consists in three steps: relevant abstract selection, rationale selection and label prediction. In this paper, we present team QMUL-SDS’s participation in the shared task. We propose an approach that performs scientific claim verification by doing binary classifications step-by-step. We trained a BioBERT-large classifier to select abstracts based on pairwise relevance assessments for each <claim, title of the abstract> and continued to train it to select rationales out of each retrieved abstract based on <claim, sentence>. We then propose a two-step setting for label prediction, i.e. first predicting “NOT_ENOUGH_INFO” or “ENOUGH_INFO”, then label those marked as “ENOUGH_INFO” as either “SUPPORT” or “CONTRADICT”. Compared to the baseline system, we achieve substantial improvements on the dev set. As a result, our team is the No. 4 team on the leaderboard.

## introduction as online content continues to grow at an unprecedented rate, the spread of false information online increases the potential of misleading people and causing harm. where the volume of information shared online is difficult to be managed by human fact-checkers, this leads to an increasing demand on automated fact-checking, which is formulated by researchers as 'the assignment of a truth value to a claim made in a particular context' @xcite . though a body of research focuses on conducting fact-checking in the politics domain, scientific claim verification has also gained increasing interest in the context of the ongoing covid-19 pandemic. the sciver shared task provides a n ei stands for "not_enough_info", c stands for "con-tradict" and s stands for "support". given claim c, our system first retrieves top k tf-idf similarity abstracts out of the corpus, then uses a biobert binary classifier to further identify desired abstracts on top of that. with retrieved abstracts, our system then uses another biobert binary classifier to select rationales. we finally do label prediction in a two-step fashion, i.e. first make verdicts on "enough_info" or not and, if positive, then make verdicts on "support" or not. this paper presents and analyses team qmul-sds's participation in the sciver shared task. in particular, we explore creative approaches of solving the challenge with limited resources. figure 1 provides an overview of our system. while many other systems make use of external datasets, e.g. fever @xcite , our system focuses on efficient use of the scifact dataset @xcite . furthermore, in the interest of keeping the efficiency of our system, we limit our model choices to the size of roberta-large @xcite , ruling out for example @xcite and t5 @xcite , which were used in other participating systems. more specifically, our system mainly uses roberta @xcite and biobert @xcite . the latter is pretrained on biomedical text and therefore is very close to our target domain. with improved pipeline design, our system shows competitive performance with limited computing resources, achieving the 6th position in the task and ranked 4th when distinct teams are considered. 1 2 related work ## approach following the convention of automated factchecking systems @xcite and the verisci baseline system, we explore novel ways of tackling the challenge by handling the three subtasks: abstract retrieval, rationale selection and label prediction. ## results we perform various experiments on the scifact dataset to identify the best models and techniques to be submitted to the task. unless explicitly specified, models are trained on the scifact's train set and evaluated on the scifact's dev set. ## discussion and future work our intuitive step-by-step binary classification system achieves substantial improvements over the baseline without demanding additional data or extra large models. an improved evidence retrieval module has made the main contributions to the performance boost. our system makes an effort to improve the abstract retrieval module after applying a scalable traditional information retrieval weighting scheme, tf-idf. instead of handling it as a re-ranking task and manually selecting thresholds @xcite , we formulate it as a binary classification task, which makes better use of the available training data and decreases the false positive rate effectively. when applying a similar approach to ratio- nevertheless, this problem is inevitable for a pipeline system that has multiple machine learning modules, as errors in each of the modules will accumulate throughout the pipeline. a better system design is desired such that it tackles the challenge in a more systematic way. a promising approach is to train a model to learn three subtasks in a multitask learning manner so that it may optimise for better overall performance. ## conclusions in this paper, we proposed a novel step-by-step binary classification approach for the sciver shared task. our submission achieved an f1 score of 55.35% on the test set, ranking 6th among all the submissions and 4th among all the teams. we show that (1) concerning evidence retrieval, a classification based approach is better than a ranking based approach with manual thresholds; (2) two-step binary label prediction has better performance than three-way label prediction with limited training data; (3) a more systematic design of automated fact-checking system is desired.