TTCB System Description to a Shared Task on Implicit and Underspecified Language 2021 (2021)

Abstract: In this report, we describe our transformers for text classification baseline (TTCB) submissions to a shared task on implicit and underspecified language 2021. We cast the task of predicting revision requirements in collaboratively edited instructions as text classification. We considered transformer-based models which are the current state-of-the-art methods for text classification. We explored different training schemes, loss functions, and data augmentations. Our best result of 68.45% test accuracy ( 68.84% validation accuracy), however, consists of an XLNet model with a linear annealing scheduler and a cross-entropy loss. We do not observe any significant gain on any validation metric based on our various design choices except the MiniLM which has a higher validation F1 score and is faster to train by a half but also a lower validation accuracy score.

## introduction a shared task on implicit and underspecified language 2021 is the first installment of predicting revision requirements in collaboratively edited instructions @xcite based on the wikihow-toimprove dataset @xcite . the dataset consists of sentences and their revisions if any. there are 5 rule-based revision types which are pronoun replacement, 'do' verb replacement, verbal phrase compliment insertion, adverbial and adjectival modifier insertion, and logical quantifier or modal verb insertion. the task is to determine whether a given sentence with its corresponding context paragraph needs any revision based on the aforementioned revision types. previous work @xcite ) compares bert @xcite and bilstm on the full wikihowtoimprove dataset which has 2.7 millions sentences. the previous experiment integrates 4.25 millions of unrevised sentences from wikihow to ## experimental setup our input is a simple concatenation of a sentence and its context paragraph. we tried different context lengths and found that 128 yields the best result. from figure 1 , the mean input length is only 62.58 with the standard deviation of 36.00. this is from the shared task dataset which is the subset of the original wikihowtoimprove dataset and has 45,909 sentences in total (39,187 sentences in the training set.). the statistics suggest setting the context length less than 200 to be cost-effective and there are only 1,632 training instances (around 4%) having their input lengths longer than 128 with the maximum length of 770. all of our experiments were done in the google colab setting. we used only base models for all transformers. we used the batch size of 8 and the learning rate of 1e-5 for all experiments. we considered linear annealing scheduler since other schedulers, such as reducelr scheduler, cosine annealing scheduler, or cosine annealing scheduler with restart, do not provide any significantly different results. also, adding a warm-up step does not make any difference too. we trained the model for 2 for xlnet. ## conclusion this report describes our baseline systems for a shared task on implicit and underspecified language 2021, predicting revision requirements in wikihow. our best result is from the xlnet model with a linear annealing scheduler and a cross-entropy loss. we do not observe any significant gain on any validation metric based on our various design choices. the cost-sensitive loss might help only when performing data augmentation. minilm is comparable to xlnet but at a half computation cost. we summarize the results as finetuning transformerbased language models for text classification only provides incremental improvements even though better language models consistently lead to better results. also, the accuracies at most ∼ 70% are not very practical. this suggests a big challenge for the language models in the context of implicit and underspecified language. we release our training code as an unofficial baseline for the challenge. there are many possible future directions. first, we have not considered any advanced loss functions, such as triplet loss @xcite , for our siamese training experiments. second, recent work on predicting revisions in wikihow @xcite depicts a promising integration of syntactic preprocessing and sentence embedding training. nevertheless, more data analysis is needed to pinpoint what a particular model should learn.