RG PA at S em E val-2021 Task 1: A Contextual Attention-based Model with R o BERT a for Lexical Complexity Prediction (2021)

Abstract: In this paper we propose a contextual attention based model with two-stage fine-tune training using RoBERTa. First, we perform the first-stage fine-tune on corpus with RoBERTa, so that the model can learn some prior domain knowledge. Then we get the contextual embedding of context words based on the token-level embedding with the fine-tuned model. And we use Kfold cross-validation to get K models and ensemble them to get the final result. Finally, we attain the 2nd place in the final evaluation phase of sub-task 2 with pearson correlation of 0.8575.

## introduction lcp is an augmented version of complex word identification(cwi) @xcite , predict complexity score for each target word in a sentence. the dataset is a multi-domain english dataset annotated with a 5-point likert scale (1-5). the annotation model in complex addresses complexity as a continuum instead of a binary feature. previous studies of cwi treat the task as a binary classification, predict a complexity label @xcite for a set of target words in a sentence. in this paper, we exploratory data analysis(eda) for the dataset, and found that the distribution of the single task and multi task dataset are very inconformity, so should bulid two models for every dataset great than one model for merge two task dataset. several key technologies as follows: previous approaches to cwi typically refer to binary identification of complex words, two shared tasks on cwi topic have been organized so far. semeval-2016 task 11 @xcite and bea workshop 2018 @xcite . the two tasks approache a number of different model to classification, ranging from traditional machine learning classifiers such as support vector machines (svm), decision trees, random forest, and maximum entropy classifiers to deep learning classifiers, such as recurrent neural networks. a wide range of features were used such as word embeddings, word and character n-grams, word frequency, zipfian frequency distribution, word length, morphological, syntactic, semantic, and psycholinguistic. bert is a new language representation model, and stands for bidirectional encoder representations from transformers @xcite . since bert appear, fine-tuned pre-trained model with just one additional output layer to create stateof-the-art models for a wide range of tasks. a number of transformers series models are proposed, such as gpt-2, roberta @xcite , xlm, distilbert, xlnet. in this papaer we focus on use roberta to slove the tasks. ## system overview we propose a roberta with attention based model to solve the lcp task, figure 1 outlines our proposed model framework. first, use byte-pair encoding (bpe) to tokenize the input sentent, which is an effective subword technique to relieve the out-of-vocabulary (oov) problem. for every roberta hidden layer, we apply token pooling that is average the target words tokens embedding as the target vector. then we calculate the attention between target vector and context tokens vector which are sentence tokens masked the target tokens, after that, context tokens embedding multiply attention weight as the context vector, then concatenate the context vector and target vector. second, pooling the roberta last 12 layers context vector and target vector, finally, connect the mlp layer to predict the lcp score. ## conclusion this paper presents a method to predicting lexical complexity, which apply roberta-large as the backbone language model. first fine-tune backbone model for corpus classification. then bulid model with attention based context representation. make vector recall for multi task data augmentation. finally, we carry out a multi-model average ensemble strategy to enhance the model performance. in the future, we will exploit better model for text representation, and utilizing data augmentation for all task.