QADYNAMICS : Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering (2023)

Abstract: Zero-shot commonsense Question-Answering (QA) requires models to reason about general situations beyond specific benchmarks. State-of-the-art approaches fine-tune language models on QA pairs constructed from CommonSense Knowledge Bases (CSKBs) to equip the models with more commonsense knowledge in a QA context. However, current QA synthesis protocols may introduce noise from the CSKBs and generate ungrammatical questions and false negative options, which impede the model’s ability to generalize. To address these issues, we propose QADYNAMICS, a training dynamics-driven framework for QA diagnostics and refinement. Our approach analyzes the training dynamics of each QA pair at both the question level and option level, discarding machine-detectable artifacts by removing uninformative QA pairs and mislabeled or false-negative options. Extensive experiments demonstrate the effectiveness of our approach, which outperforms all baselines while using only 33% of the synthetic data, even including LLMs such as ChatGPT. Moreover, expert evaluations confirm that our framework significantly improves the quality of QA synthesis. Our code and model checkpoints are available at https://github.com/HKUST-KnowComp/QaDynamics.

## introduction the advent of various commonsense question-answering (qa) benchmarks @xcite has demonstrated that pre-trained language models (ptlms) @xcite can achieve extraordinary performances when fine-tuned on these benchmarks. however, these neural systems have been criticized for only learning surface-level correlations and lacking general semantic reasoning abilities, which often require implicit commonsense * equal contribution knowledge @xcite . to reliably assess the resilience of qa models across diverse domains, the zero-shot commonsense qa task has been proposed to evaluate the generalizable reasoning ability of a qa model @xcite without supervision signals from any qa benchmarks. @xcite introduced a technique for tackling this task by fine-tuning a ptlm on qa pairs synthesized from knowledge triples in common-sense knowledge bases (cskbs). the head and relation of a triple were transformed into a question using natural language templates, with the tail serving as the answer. distractors, or negative examples, were tails from triples sampled from the same cskb using pre-defined strategies, such as keyword or embedding proximity filtering. however, the primary obstacle hindering further progress in this method is the quality of the synthetic qa dataset. this issue arises because manually-curated cskbs often contain subtle but strong annotation artifacts @xcite , which could provide easy backdoors for the model to perform exceptionally well on synthetic test sets but fail to generalize on heldout qa benchmarks. additionally, the current qa synthesis process results in a significant number of ungrammatical questions, and the negative sampling strategy used to create distractors is not entirely effective in preventing false-negative options, as evidenced by @xcite . despite the existence of dataset filtering algorithms, such as adversarial filtering @xcite for negative option selection, they have been shown to be less effective compared to random selection baselines @xcite . this is because they only focus on model uncertainty in the final predictions, which is not effective enough for synthetic data that contains a plethora of noise and imbalanced examples (appx. §a.1). instead of leveraging data filtering based on model uncertainty in the final predictions, we draw inspiration from @xcite and employ training dynamics as a more precise indicator that studies instance learnability across all training steps. while the vanilla training dynamics regard each data instance as a whole without considering the learnability of each option or choice, we propose qadynamics, a training dynamics-driven framework for synthetic qa diagnostic and refinement that favors choice-level diagnosis. specifically, our approach proposes a novel schema that offers greater flexibility in deriving the training dynamics of multiple-choice qa with an arbitrary number of options, thus accommodating the varying number of choices in different commonsense qa benchmarks. qadynamics then analyzes the training dynamics of each option, greedily drops the easy distractor to reduce the impact of cskb artifacts, and eliminates qa pairs containing mislabeled or false-negative options according to the confidence gap between all options ( §3). extensive experiments showcase the efficacy and data efficiency of our proposed framework, surpassing all previous zero-shot csqa baselines while only leveraging 33% of training data and even outperforming gpt3.5 @xcite and chat-gpt ( §4.4). further expert evaluations confirm the effectiveness of our proposed method in enhancing the quality of the synthetic qa set ( §4.5). ## qadynamics this section outlines our proposed framework, qa-dynamics, which consists of four steps: (1) calculate the training dynamics for each option in a qa pair. (2) refine the qa pair by eliminating the easy distractor. (3) filter out qa pairs that may be mislabeled or contain false-negative distractors. (4) train the model using marginal ranking loss. ## conclusions in this paper, we propose qadynamics, a training dynamics-empowered framework for data-efficient zero-shot commonsense qa that jointly considers the learning difficulty at both the qa and option levels. our framework, on average, achieves state-of-the-art performance by surpassing large language models and all baselines significantly with only 33% of training data. further expert evaluations showcase that our proposed method effectively eliminates poor-quality qa entries in the synthetic dataset.