S tanford ML ab at S em E val 2022 Task 7: Tree- and Transformer-Based Methods for Clarification Plausibility (2022)

Abstract: In this paper, we detail the methods we used to determine the idiomaticity and plausibility of candidate words or phrases into an instructional text as part of the SemEval Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. Given a set of steps in an instructional text, there are certain phrases that most plausibly fill that spot. We explored various possible architectures, including tree-based methods over GloVe embeddings, ensembled BERT and ELECTRA models, and GPT 2-based infilling methods.

## introduction the internet is filled with instructional texts from websites like wikihow that detail how to perform a variety of tasks (from tying a bow tie to building a deck of stairs). with this increase of quantity and use of instructional texts, it has become increasingly important for them to be clear and unambiguously worded. to this end, we evaluate whether lightweight and neural models are capable of detecting which phrases most plausibly fit into a given series of instructions. the current revision process requires that a reader potentially identify something wrong about the content, and they report it to the website for alterations. a system such as the one described could automatically identify candidates that are unlikely to exist and report them for human verification. in this paper, we describe our lightweight and transformer-based models that rank the plausibility of candidate phrases given some previous context. ## experimental setup we merge the provided train and dev sets, perform random 75:25 splits of the merged data to use for training and validation. we noted little difference in performance between different random splits. predictions were evaluated on accuracy for subtask a and spearman correlation for subtask b. although we also calculated other metrics such as macro-averaged f1 for subtask a and mean squared error for subtask b, we standardized on accuracy and spearman correlation for consistency in comparing results. in particular, since the classes were close to balanced for subtask a, using accuracy was not a big issue in overfitting to certain classes. ## results as shown in after switching to bert-like models, we generally achieve a significant improvement in classification accuracy. in particular, electra achieves the highest accuracy across all of our models, with a dev set accuracy of 0.465. an exception is roberta, with which we achieve an accuracy close to that of linear discriminant analysis. the ilm-based models ultimately failed to improve upon the accuracy of the lightweight models. while the ilm approach might initially seem to be the most promising given the task, it seems that the model's ability to generate the top few most likely options did not correlate with its ability to compare relatively more unlikely potential infills. the model was also pre-trained on a stories database, which may not reflect the context appropriate for instructional texts like wikihow articles. for the regression subtask, we only experiment with bert with a regression head. we find that bert achieves a spearman correlation of 0.149 on the dev dev set. our final evaluation results on both subtasks are shown in ## conclusion bert-like models are the highest performing models for this type of instructional clarifications task. because bert has been trained to predict masked tokens, it is naturally better at finding words or phrases that most plausibly fit with the surrounding context. simple glove word embedding models were unable to learn to the requisite complexity of this meta-linguistic task and were unable to break the level of 0.389 accuracy. meanwhile, ilm-based approaches seemed promising, but in practice failed to accommodate phrases or long words. this method is more effective at generating text and not necessarily determining how plausible an infill sounds in that context. in terms of future work, we believe that our system may be improved by the usage of large pretrained models such as spanbert @xcite , which are trained using tasks involving multi-token span prediction, which may be more fit to the given task. despite our lack of results, we believe that the infilling approach is still promising, and hope that it can be adapted to this task going forward.