Incremental Neural Lexical Coherence Modeling (2020)

Abstract: Pretrained language models, neural models pretrained on massive amounts of data, have established the state of the art in a range of NLP tasks. They are based on a modern machine-learning technique, the Transformer which relates all items simultaneously to capture semantic relations in sequences. However, it differs from what humans do. Humans read sentences one-by-one, incrementally. Can neural models benefit by interpreting texts incrementally as humans do? We investigate this question in coherence modeling. We propose a coherence model which interprets sentences incrementally to capture lexical relations between them. We compare the state of the art in each task, simple neural models relying on a pretrained language model, and our model in two downstream tasks. Our findings suggest that interpreting texts incrementally as humans could be useful to design more advanced models.

## introduction coherence describes the semantic relation between elements of a text. it distinguishes a text as either a unified whole or a collection of unrelated sentences. lexical coherence represents the cohesive effect achieved by lexical relations @xcite . earlier work mainly focuses on capturing lexical relations using external resources @xcite . @xcite introduce a graph model, the latest model for lexical coherence, to represent lexical relations between sentences on the graph. it encodes sentences as nodes and lexical relations between sentences as edges. this model, nevertheless, considers lexical items independently. recent neural models adopt a modern machine learning-based technique @xcite , the transformer @xcite . it relates all items simultaneously to capture semantic relations in sequences. more recently, large-scale pretrained language models, transformerbased models pretrained on the massive amounts of text, have led to significant improvements in a range of nlp tasks @xcite . however, the transformer processes texts in a way which is different from the way humans do it. psycholinguistic experiments show that humans read texts incrementally @xcite @xcite . @xcite claim that nlp systems which follow this theory should interpret texts incrementally, too. do neural models benefit from both pretrained language models and incremental sentence processing? to investigate this question, we propose a coherence model which interprets sentences incrementally to capture lexical relations. for the ongoing sentence being read, our model first captures a semantic centroid vector which represents the centroid of preceding sentences. the centroid vector is computed as averaged representations of sentences. the model then measures semantic similarity between the centroid vector and the current sentence. our model iterates this procedure for all sentences. algorithm 1 incremental processing module. 1: procedure ipm(sent list) we evaluate our model on two tasks: assessing discourse coherence and automated essay scoring. we compare our model with the state of the art in each task and two variants of a simple baseline relying on a pretrained language model: the first baseline encoding sentences individually, and the second baseline encoding a whole text at once foot_0 . modeling lexical coherence has proven to be effective in diverse nlp applications like summarization @xcite , translation @xcite , and discourse parsing @xcite . we believe that our study for lexical coherence can be beneficial in these applications. ## our model figure 1 shows our model architecture. our model takes sentence representations using a pretrained language model. the model then feeds sentences into the lexical coherence module to produce the semantic centroid vector and the semantic similarity vector. we concatenate the two vectors to generate a model output through a feed-forward network. ## conclusions we propose a coherence model which encodes sentences individually using a pretrained language model and interprets sentences incrementally. the simple baseline, which encodes a whole text at once unlike humans do, outperforms our model on gcdc which includes informal texts such as online reviews. however, our model outperforms this model on toefl whose texts are organized better. our findings suggest that it could be useful to constrain models to be exposed limited information as humans do to design more advanced neural models with a pretrained language model. prompt 1 agree or disagree: it is better to have broad knowledge of many academic subjects than to specialize in one specific subject. prompt 2 agree or disagree: young people enjoy life more than older people do. prompt 3 agree or disagree: young people nowadays do not give enough time to helping their communities. prompt 4 agree or disagree: most advertisements make products seem much better than they really are. prompt 5 agree or disagree: in twenty years, there will be fewer cars in use than there are today. prompt 6 agree or disagree: the best way to travel is in a group led by a tour guide. prompt 7 agree or disagree: it is more important for students to understand ideas and concepts than it is for them to learn facts. prompt 8 agree or disagree: successful people try new things and take risks rather than only doing what they already know how to do well.