OCHADAI - KYOTO at S em E val-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction (2021)

Abstract: We propose an ensemble model for predicting the lexical complexity of words and multiword expressions (MWEs). The model receives as input a sentence with a target word or MWE and outputs its complexity score. Given that a key challenge with this task is the limited size of annotated data, our model relies on pretrained contextual representations from different state-of-the-art transformer-based language models (i.e., BERT and RoBERTa), and on a variety of training methods for further enhancing model generalization and robustness: multi-step fine-tuning and multi-task learning, and adversarial training. Additionally, we propose to enrich contextual representations by adding hand-crafted features during training. Our model achieved competitive results and ranked among the top-10 systems in both sub-tasks.

## introduction predicting the difficulty of a word in a given context is useful in many natural language processing (nlp) applications such as lexical simplification. previous efforts @xcite @xcite have focused on framing this as a binary classification task, which might not be ideal, since a word close to the decision boundary is assumed to be just as complex as one further away @xcite . to alleviate this issue, semeval-2021 task 1 @xcite formulates this task as a regression task, where a model should predict the complexity value of words (subtask 1) and mwes (subtask 2) in context. this paper describes the system developed by the ochadai-kyoto team for semeval-2021 task 1. given that a key challenge in this task is the limited size of annotated data, we follow best practices from recent work on enhancing model generalization and robustness, and propose a model 2 task description semeval-2021 task 1 provides participants with an augmented version of the complex dataset @xcite , a multi-domain english dataset with sentences containing words and mwes annotated on a continuum scale of complexity, in the range of [0,1]. easier words and mwes are assigned lower complexity scores, while the more challenging ones are assigned higher scores. this corpus contains a balanced number of sentences from three different domains: bible @xcite , europarl @xcite and biomedical @xcite . the task is to predict the complexity value of single words (subtask 1) and mwes (subtask 2) in context. the statistics of the corpus are presented in ## system overview we focus on exploring different training techniques using bert and roberta, given their superior performance on a wide range of nlp tasks. each text encoder and training method used in our model are detailed below. ## conclusion in this paper, we have presented the implementation of the ochadai-kyoto system submitted to the semeval-2021 task 1. our model ranked 7th out of 54 participating teams on subtask 1, and 8th out of 37 teams on subtask 2. we proposed an ensemble model that leverages pretrained representations, multi-step fine-tuning, multi-task learning and adversarial training. we also proposed to enrich contextual representations by incorporating hand-crafted features during training. in future efforts, we plan to further improve our model to better handle data-constraint and domain-shift scenarios.