niksss at S em E val-2022 Task7:Transformers for Grading the Clarifications on Instructional Texts (2022)

Abstract: This paper describes the 9th place system description for SemEval-2022 Task 7. The goal of this shared task was to develop computational models to predict how plausible a clarification made on an instructional text is. This shared task was divided into two Subtasks A and B. We attempted to solve these using various transformers-based architecture under different regime. We initially treated this as a text2text generation problem but comparing it with our recent approach we dropped it and treated this as a text-sequence classification and regression depending on the Subtask.

## introduction instructional texts which are in the form of stepby-step instruction to achieve a particular goal are sometimes ambiguous to make out what is being talked about. to ensure that instructions describe clearly enough what steps must be followed, some clarifications are made in the places of ambiguity. this task @xcite revolves around automating the grading of a particular clarification made on the instructions into plausible implausible and neutral (subtask a) and on a finer scale where it is required to rank potential clarifications from 1 to 5 @xcite . previous work, related to this involves a shared task @xcite which was a binary classification task, in which systems had to predict whether a given sentence in context requires clarification or not. this shared task uses the same dataset that is the wikihowtoimprove dataset (anthonio et al., 2020) but with some variations. instead of a binary classification task, this task is shaped as a cloze task in which, clarifications are presented as possible fillers and systems have to score how well each filler plausibly fits in a given context. a data instance can be seen in figure 1 . seeing the performance of bert over bilstms in @xcite , we decided to build upon that and explore the use of other transformers-based models. ## error analysis after examining the predictions from the submitted model, we saw that the model struggled significantly in distinguishing between neutral and either of plausible/implausible clarifications as there's a very slight difference between them. this problem increases further when we created a feature vector from the same sentence with changed filler word as it leads to a very slight change in the vector, hence leading the model confused to distinguish between individual classes having almost similar feature distribution. the performance can also be attributed to the distribution of the labels in development set and our submitted model might have overfitted to the development set, leading to a further decrease in performance in the test set. ## conclusion we developed a system to classify the clarification made on instructional text into varying levels of plausibility using a transformer based language model with a limited attention span only taking a limited context around the filler. the recent advancement in transformer based models, such as bigbird @xcite and longformer (beltagy et al., 2020) which can take up longer context into consideration are more preferred for a task like this.