D uluth NLP at S em E val-2022 Task 7: Classifying Plausible Alternatives with Pre–trained ELECTRA (2022)

Abstract: This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. Given an instructional text with an omitted token, the task requires models to classify or rank the plausibility of potential fillers. To solve the task, we fine–tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausibility. This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%. However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A.

## introduction instructional texts (e.g., how to guides) describe how to accomplish a given goal and are integral to our daily lives. one popular source is wikihow foot_0 which is an online platform that allows users to collaborate to create and maintain such guides. this kind of documentation must be clear, and if it is not then this is a key reason that prompts revisions of underspecified instructions in wikihow @xcite . one important problem for nlp is to determine if a given instructional text is in need of clarification or revision. semeval-2021 task 7 @xcite extends this problem by requiring models to score five possible fillers based on how well they can plausibly fit a given context. task 7 includes two subtasks. subtask a classifies the possible fillers as implausible, neural, and plaus-bile. subtask b requires systems to rank the fillers on a scale of 1 to 5, where a higher score means more plausible. we only participated in subtask a and used a variety of bert-based methods. ## task data the training, development and test data were supplied by the organizers of semeval-2022 task 7 @xcite . the dataset is based on the wik-ihowtoimprove corpus @xcite which consists of edits of 2.5 million sentences from wikihow. the authors show that edits are primarily made to clarify instructional texts and that the distinction between older and revised versions of sentences can be modelled computationally. each instance in the data is divided into an article title, a section header, and a sentence nested between a previous and future context. each instance also includes five potential fillers, each of which is annotated as implausible, neutral, or plausible, which serves as the basis of subtask a. there is a also a plausibility score of 1 to 5 which is the basis of subtask b (which we did not participate in). ## methodology previous context sentence followup context state what you have contributed to the company. by doing it this way, it is going to show that you have done your job and been an asset, thus the raise is welldeserved. * p: if you believe your [...] of time working at this company warrants a raise or promotion, say that as well. fillers: a. continuity b. window c. abundance d. length e. appreciation it is best to tell them all the reasons you believe you deserve this increase. an all weather strategy often keeps you always afloat compared to one planned for normal market behavior. planning for a failure is always better than failing to plan p: uncertainties of [...] can be classified into four levels fillers: a. public opinion b. future markets c. the future d. this sort e. their futures level one gives a fairly clear view of the future, and an inkling of what to expect. since wikis are often volunteerdriven projects, wikigifts can go a long way in showing someone how much you appreciate their efforts. find or create awards specific to that wiki. p: on wikihow, for example, you can make award templates on [...] and post them on people's talk pages. fillers: a. books b. facebook c. graph d. wiki-how e. earth publicize that you gave the wikigift. roberta and electra. bert (bidirectional encoder representational from transformers) @xcite , is a transformer-based language model trained using masked language modeling (mlm) to predict the masked tokens based on the surrounding context. in mlm, a given percentage of the tokens of an input sequence is masked, and bert is tasked to predict the orginal tokens. with the mlm approach, bert was able to produce good results when transferred to downstream nlp tasks, becoming the new benchmark for other pre-trained models. the authors of the electra paper @xcite , however, note that the mlm approach only learns from the masked tokens (about 15%) of any given example, thus requiring substantial compute resources to train a language model using mlm. the next model we used was roberta, which is essentially a replication of bert @xcite which adjusts key hyperparameters and uses larger amounts of data during pre-training. roberta improves upon bert when trained longer, using larger mini-batches over more data. similarly, we experimented with electra @xcite , a pre-trained model that uses replaced token detection as a pre-training objective. this distinguishes real inputs from plausible but synthetically generated ones coming from a small masked language model. the authors argue that electra improves compute efficiency during pre-training, and can match or exceed the performance of bert and its variants when fine-tuned on downstream tasks. the electra model includes two transformer models, a generator and a discriminator. the generator emulates a small masked language model by predicting the original token of a masked-out token. like the masked language model in bert, some samples of the input sequence to the generator are replaced with @xcite . the predicted results of the generator are fed as inputs to the discriminator. for each token in the sequence, the discriminator predicts whether it is the original or the generated one. this means that the discriminator is able to learn from all input tokens for any given example, with the model loss calculated over all the tokens. this is what sets electra apart from bert, and a major reason for electra's greater compute efficiency. electra addresses a drawback of masked language models, which is that the masked tokens are only used during pre-training and are omitted during fine-tuning. this pre-train finetune mismatch contributes to a loss in performance. replaced token detection, in which electra distinguishes between real tokens and their plausible fakes, is easily transferable to fine-tuning. this is particularly true for subtask a of semeval-2022 task 7, which requires a model to classify how fillers will plausibly fit an omitted token. ## system description this section introduces duluthnlp's approach which relied on bert, roberta, and electra. we discuss our fine-tuning process, and then how class weights were tuned. ## experimental results in this section we present the results of our models on both the development data and the test data as used in the official evaluation scored by the task organizers. we used the logistic regression model from scikit-learn foot_1 as a baseline method. it incorporates binarized ngram counts @xcite .this obtained an accuracy of 34.8% on the development data for subtask a. as shown in however, with tuned class weights, all three of the pre-trained models achieved accuracy above 50%. electra and roberta obtained nearly identical scores of 55.2% versus 55.6%. we decided to use electra as our official evaluation ## error analysis the classes predicted by our model on the development data prior to the official evaluation were skewed to neutral, as discussed earlier. we observed this with various different pre-trained models including bert, roberta, and electra with various different hyperparameter settings. despite our best efforts the duluthnlp system never reached accuracy above 45%. we addressed this by adding class weights to crossentropyloss function used in our models. when we assigned class weights of [1.5, 0.03, 0.7] for the implausible, neutral, and plau-sible labels, the wrongly predicted scores for the neutral label reduced to 28%, as shown in what is curious, though, is the difficulty in correctly classifying the implausible label. this represents 38% of the actual labels but is 14% of the predicted labels. even after class weights are set as described above, only 18% of the labels are predicted to be implausible, which is still a difference of 20% from the actual implausible labels. we achieved further performance gains on the devset across all the models by highlighting the filler in each data instance with a special "[]" symbol, and this forms the basis for our results in ## ethical considerations the training of large language models has a dark side: demands for large amounts of compute power and the corresponding energy consumption. training bert with the masked language model requires a lot of computational resources.this raises concerns over the accessibility, cost, and environmental impact of such methods @xcite . whilst we experimented with three bertvariants, we sought to limit model fine-tuning to the base models (bert-base, roberta-base, and electra-base), which require less compute resources than their larger versions. the electra model, which we used for our official evaluation test results, is computationally efficient, which partly informed our choosing it over the other models for use as our official method during the evaluation stage. the accuracy of our models are, at best, a little above 50%. this means that roughly half of any of these predictions may be wrong. this is clearly not accurate enough to deploy in a real setting without potentially causing harm. it is easy to imagine the negative impacts of automatically clarified instructions that prove to be inaccurate. similarly, the test data and the training set are from the same sample distribution, and we cannot guarantee that our model will achieve similar results for any out-of-distribution test data. in other words, our model, reliant as it is on the contextual representations provided by the pre-trained models, cannot perform well on a completely new task or distribution. we relied on large language models which were trained on very large corpora. such text may include stereotypes and biases which are then carried over into the resulting model @xcite . this locks the model to older, less-inclusive understandings that may not reflect more modern views of gender, race, or other questions of identify. to minimize the potential harms of such misrepresentations it would be best if candidate predictions were verified by a human editor. however, if models like these were deployed and used to auto-suggest revisions to wikihow, they may constrain the choices of human editors @xcite . this could create a mindset that uncritically accepts the framed options of the model as legitimate @xcite . the reviewer may then abandon their own edits because the auto-suggestion seems to provide an answer. one way to minimize such risk is the deployment of reflection machines @xcite , a decision support system that will compel users to give reasons for accepting or rejecting suggestions from the model.