I nsert GNN : A Hierarchical Graph Neural Network for the TOEFL Sentence Insertion Problem (2024)

Abstract: The integration of sentences poses an intriguing challenge within the realm of NLP, but it has not garnered the attention it deserves. Existing methods that focus on sentence arrangement, textual consistency, and question answering have been shown to be inadequate in addressing this issue. To bridge this gap, we introduce InsertGNN which conceptualizes the problem as a graph and employ a hierarchical Graph Neural Network (GNN) to comprehend the interplay between sentences. Our approach was rigorously evaluated on a TOEFL dataset, and its efficacy was further validated on the expansive arXiv dataset using cross-domain learning. Thorough experimentation unequivocally establishes InsertGNN’s superiority over all comparative benchmarks, achieving an impressive 70% accuracy—a performance on par with average human test scores.

## introduction and related work sentence insertion (si), initially introduced by @xcite , stands as a crucial task for evaluating human linguistic prowess. however, with the advent of deep learning (dl), it has received scant attention over the past decade compared to other nlp domains like machine translation and text generation. as a result, the realm of dl lacks a standardized solution tailored to this particular challenge. to address this void, we curate a dataset sourced from the toefl exam, an english proficiency test that archives test-takers' comprehensive accuracy scores. through this dataset, we investigate whether contemporary nlp techniques can surpass the performance of nonnative english speakers 1 . sentence ordering (so) and question answering (qa) constitute the two closely intertwined subdomains that share relevance with si. on the 1 our compiled dataset is accessed at https://anonymous.4open.science/r/ toefl-sentence-insertion-dataset-899d/readme.md one hand, so initiates its approach by employing multi-layer perceptrons (mlps) to facilitate pairwise order ranking @xcite but can inadvertently propagate errors to a significant degree. to mitigate this concern, the pointer network (pn) @xcite integrates an attention mechanism to enhance model capacity, building on subsequent advances such as the incorporation of attention and the deepening of the architecture @xcite . however, the direct applicability of the pn framework to si is hindered by the differing nature of its input. as a remedy, recent studies solve so by plugging a coherence verifier @xcite or through a non-autoregressive manner @xcite ). another line @xcite presents an unsupervised graph method that approaches the problem through the lens of sentence coherence and similarity within text. this method claims superiority when applied to the supervised entity grid and unsupervised entity graph scenarios. yet, this approach, while effective, relies on a non-parametric structure and necessitates the construction of all potential graphs containing the extracted sentence. meanwhile, qa provides a more universal pathway for our context. notable works @xcite have established qa as a potent framework. in our scenario, the removed sentence and the paragraph at large can be likened to the question and context, respectively. these components are seamlessly amalgamated and channeled through a network to produce the sought-after position. however, this linear fusion poses challenges for cutting-edge models such as transformers in comprehending the inherent logic connecting the concatenated paragraph and its designated slots. in this paper, we represent si as a directed graph, where each node represents a sentence, and the edge depicts their relative potential order. this pattern imitates the way people tackle si, that is, ## discussion it is worth mentioning that the si problem shares similarities with existing pretraining objectives like bert's next sentence prediction (nsp) and al-bert's sentence-order prediction (sop). specifically, nsp concatenates two masked sentences as inputs during pretraining. sometimes they correspond to sentences that were next to each other in the original text, sometimes not. the model then has to predict if the two sentences were following each other or not. sop primary focuses on inter-sentence coherence and is designed to address the ineffectiveness of the next sentence prediction (nsp) loss proposed in the original bert. ## conclusion in the paper, we propose a new sentence insertion framework called insertgnn and build a toefl benchmark data set. strong empirical evidence demonstrates the effectiveness of insertgnn over existing language language models like chatgpt in this interesting nlp task.