Nowruz at S em E val-2022 Task 7: Tackling Cloze Tests with Transformers and Ordinal Regression (2022)

Abstract: This paper outlines the system using which team Nowruz participated in SemEval 2022 Task 7 “Identifying Plausible Clarifications of Implicit and Underspecified Phrases” for both subtasks A and B. Using a pre-trained transformer as a backbone, the model targeted the task of multi-task classification and ranking in the context of finding the best fillers for a cloze task related to instructional texts on the website Wikihow. The system employed a combination of two ordinal regression components to tackle this task in a multi-task learning scenario. According to the official leaderboard of the shared task, this system was ranked 5th in the ranking and 7th in the classification subtasks out of 21 participating teams. With additional experiments, the models have since been further optimised. The code used in the experiments is going to be publicly available.

## introduction oxford dictionary defines cloze test as "a test of readability or comprehension in which a person is required to supply words which have been deliberately omitted from a passage" (oxford university press, 2022). in the context of nlp, a cloze format task is one in which the context is one or more sentences with masked spans and the model is expected to predict a suitable filler for each span. cloze-format datasets have become popular in nlp recently as they are relatively easy to create automatically and provide high quality resources for model training @xcite . 0 national organization for development of exceptional talents semeval 2022 task 7 is framed as a cloze task in which the goal is to rank or classify fillers within a given context based on their suitability. the texts are taken from actual articles on an instructional website and the masked spans are placed at locations of edits made by users. we participated in this shared task in both ranking and classification parts and developed a transformer-based model that utilises both classification and regression components at the top layer. the code and the data used in these experiments are publicly available. ## related work cloze tasks have been a subject of interest in natural language understanding (nlu) in recent years, especially within the context of reading comprehension, story understanding, and summarisation @xcite @xcite . there is evidence that cloze tasks can be used to effectively pretrain or finetune language models in order to perform few shot learning @xcite . wikihow is a community-edited open domain repository that hosts how-to articles on a variety of different subjects. it is possible to track edits made by users and compare different versions. there have been some recent computational works exploring this resource, including modelling of revision requirements @xcite , and the effect of edits on fluency @xcite and vagueness @xcite . ## system description figure 1 shows the overall architecture of our model. a pre-trained transformer sits at the base of the architecture. different variations of this transformer are tried and reported in sec 5. these include bert @xcite , roberta @xcite , deberta (v1 & 3) @xcite , and t5 1 @xcite in different configurations. at the bottom of the network, there are two components, one for regression and the other for classification. at training time, both of these are trained in tandem using ordinal regression and a combined loss. the loss function ensures that the ranks of the labels are kept consistent across these task (sec. 3.6). 1 since t5 is originally an encoder-decoder language model, we only use its pre-trained encoder for our @xmath0 ## dataset and experimental setup the shared task is based on @xcite , where instructional texts from the website wikihow foot_2 are used to create training instances for a cloze task. the deletions in the dataset are based on actual edits made by online users and they are assumed to represent certain types of clarifications to make a point more clear or disambiguate a sentence. based on the actual edits, alternative fillers are automatically extracted and added to build a number of possible fillers. these texts along with the fillers were later annotated by humans and given plausibility scores from 1 to 5. there is also a separate type of annotation available in which there are 3 labels with discrete plausibility values. the shared task was organised in two separate tracks of classification and regression, depending on what kind of annotation was used for modelling the task. resolved pattern is one of the four following categories: implicit reference, added compound, metonymic reference, and fused head which indicates the relationship of the fillers with the context. article title is the name of the article from which the paragraph is selected. section header is the section from which the article is selected. previous context is a few sentences before the sentence that contains the filler. sentence is the sentence that contains the filler. we obtained our best results when we used a custom formatting using which we can feed all the features to the model as textual input. ## results and discussion the official test results for both subtasks are presented in we have since performed extensive analyses on the effect of hyperparameters on different varia-tions of our models, and since we do not have access to the true labels in the test set, we report our best results on the dev set. as can be seen in ## conclusion in this work we developed a set of transformerbased models powered with ordinal regression to tackle an nlp cloze task as part of the semeval 2022 shared task 7. the goal was to assign suitability scores or labels to several different provided fillers given each context and masked span. using a combined architecture based on ordinal regression that used training labels from both subtasks, we developed and trained models with a multi-task learning objective. the proposed system was ranked 4th out of 21 teams in both tracks of the shared task. in the subsequent analyses in the post evaluation phase, we have showed the effectiveness of this architecture in addressing this task. we compared different variations of our models and explored the effects of hyperparameters on model performance. the code and analyses are going to be publicly available.