Semi-Supervised Joint Estimation of Word and Document Readability (2021)

Abstract: Readability or difficulty estimation of words and documents has been investigated independently in the literature, often assuming the existence of extensive annotated resources for the other. Motivated by our analysis showing that there is a recursive relationship between word and document difficulty, we propose to jointly estimate word and document difficulty through a graph convolutional network (GCN) in a semi-supervised fashion. Our experimental results reveal that the GCN-based method can achieve higher accuracy than strong baselines, and stays robust even with a smaller amount of labeled data.

## introduction accurately estimating the readability or difficulty of words and text has been an important fundamental task in nlp and education, with a wide range of applications including reading resource suggestion @xcite ), text simplification @xcite , and automated essay scoring @xcite . a number of linguistic resources have been created either manually or semi-automatically for nonnative learners of languages such as english @xcite , french @xcite , and swedish @xcite , often referencing the common european framework of @xcite . however, few linguistic resources exist outside these major european languages and manually constructing such resources demands linguistic expertise and efforts. this led to the proliferation of nlp-based readability or difficulty assessment methods to automatically estimate the difficulty of words and texts (vajjala and meurers, 2012; wang and andersen, 2016; alfter and volodina, 2018; vajjala and rama, 2018; 1 our code is at https://github.com/akkikiki/ diff_joint_estimate layer0 layer1 w1 w2 w3 w4 figure 1 : overview of the proposed gcn architecture which recursively connects word w i and document d j to exploit the recursive relationship of their difficulty. @xcite . however, bootstrapping lexical resources with difficulty information often assumes the existence of textual datasets (e.g., digitized coursebooks) annotated with difficulty. similarly, many text readability estimation methods @xcite assume the existence of abundant lexical or grammatical resources annotated with difficulty information. individual research studies focus only on one side, either words or texts, although in reality they are closely intertwined-there is a recursive relationship between word and text difficulty, where the difficulty of a word is correlated to the minimum difficulty of the document where that word appears, and the difficulty of a document is correlated to the maximum difficulty of a word in that document (figure 2 ). we propose a method to jointly estimate word and text readability in a semi-supervised fashion from a smaller number of labeled data by leveraging the recursive relationship between words and documents. specifically, we leverage recent developments in graph convolutional networks @xcite and predict the difficulty of words and documents simultaneously by modeling those as nodes in a graph structure and recursively inferring their embeddings using the convolutional layers (figure 1 ). our model leverages not only the supervision signals but also the recursive nature of word-document relationship. the contributions of this paper are two fold: ## task definition given a set of words w and documents d, the goal of the joint readability estimation task is to find a function f that maps both words and documents to their difficulty label f : w ∪ @xmath0. documents here can be text of an arbitrary length, although we use paragraphs as the basic unit of prediction. this task can be solved as a classification problem or a regression problem where @xmath1. we use six cefrlabels representing six levels of difficulty, such as @xmath2, a2, b1, b2, c1, c2 (highest)} for classification, and a real-valued readability estimate β ∈ r inspired by the item response theory @xcite for regression foot_0 . the β for each six cefr level are @xmath3.38, @xmath4.67, @xmath5.21, @xmath6.21, @xmath7.67, and @xmath8.38. words and documents consist of mutually exclusive unlabeled subsets w u and d u and labeled subsets w l and d l . the function f is inferred using the supervision signal from w l and d l , and potentially other signals from w u and d u (e.g., relationship between words and documents). ## exploiting recursive relationship by graph convolutional networks we first show how the readability of words and documents are recursively related to each other. we then introduce a method based on graph convolutional networks (gcn) to capture such relationship. ## conclusion in this paper, we proposed a gcn-based method to jointly estimate the readability on both words and documents. we experimentally showed that gcn achieves higher accuracy by capturing the recursive difficulty relationship between words and documents, even when using a smaller amount of labeled data. gcns are a versatile framework that allows inclusion of diverse types of nodes, such as subwords, paragraphs, and even grammatical concepts. we leave this investigation as future work.