Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction (2020)

Abstract: Training recurrent neural networks on long texts, in particular scholarly documents, causes problems for learning. While hierarchical attention networks (HANs) are effective in solving these problems, they still lose important information about the structure of the text. To tackle these problems, we propose the use of HANs combined with structure-tags which mark the role of sentences in the document. Adding tags to sentences, marking them as corresponding to title, abstract or main body text, yields improvements over the state-of-the-art for scholarly document quality prediction. The proposed system is applied to the task of accept/reject prediction on the PeerRead dataset and compared against a recent BiLSTM-based model and joint textual+visual model as well as against plain HANs. Compared to plain HANs, accuracy increases on all three domains. On the computation and language domain our new model works best overall, and increases accuracy 4.7% over the best literature result. We also obtain improvements when introducing the tags for prediction of the number of citations for 88k scientific publications that we compiled from the Allen AI S2ORC dataset. For our HAN-system with structure-tags we reach 28.5% explained variance, an improvement of 1.8% over our reimplementation of the BiLSTM-based model as well as 1.0% improvement over plain HANs.

## introduction automatic prediction of the quality of scientific and other texts is a new topic within the field of deep learning. deep learning has been successfully applied to many natural language processing (nlp) problems including text classification, as well as many computer vision applications including document structure analysis. these successes suggest automatic quality assessment of scientific documents, while still highly ambitious, is feasible for scientific study. sequential deep learning models, particularly recurrent neural networks (rnns), long short-term memories (lstms) and their variants, have been particularly successful for applications that require the encoding and/or generation of relatively short sequences of text, typically at most a few sentences. applications include (short) text classification @xcite , entailment @xcite and neural machine translation (mt) @xcite . newer attention-based models, particularly the transformer model @xcite are even more apt at using all of the possible context when encoding sentences, further improving performance. transformers are also used to build general sentence embeddings with the bert model @xcite . in comparison, the accurate classification of full documents remains challenging. to be effective, a deep learning model for longer text should fulfill the following three criteria: plain sequential models such as rnns and lstms model text as unstructured word sequences. this causes problems on longer texts because of the vanishing gradient and exploding gradient problem @xcite , which hampers trainability. gradient bounding methods including gradient clipping @xcite , can help to reduce these problems, but provide no solution for docu-ments with thousands of words. transformers and bert are not a good match for long texts either, as these models have a computational cost that grows quadratically with sentence length. arguably, bagof-word models, including models performing average pooling over word embeddings accomplish trainability and computational efficiency. however, their computational cheapness is achieved at the price of making very strong statistical independence assumptions that harm prediction quality. a group of models does fulfill all three criteria: hierarchical versions of sequential models, in particular hierarchical attention networks (hans) @xcite . hans produce hierarchical text encodings using a hierarchical stacking of lstmswith attention, for the sentence and text level. this massively increases parallelization while simultaneously reducing the number of steps the gradient signal needs to be back-propagated during training, increasing learnability. han text encodings can still take much context into account at every level in the representation, thanks to the use of lstms. while hans are highly effective in forming adequate representations of longer texts, they are still deficient in the use of structure information inherent in the text. the reason is simple: these models have only one ((bi)lstm) encoding sub-model per level in the hierarchy. this sub-model is used to encode all the inputs at that level, without access to relevant structure context. in this work we observe that this problem can be tackled by adding xml-like structure-tags at the beginning and end of each input sentence. the effectiveness of our approach is demonstrated on two tasks: a paper accept/reject prediction on the peer-read dataset @xcite . ## related work multiple methods have been proposed to estimate the quality of scientific papers. the most common approach is to use the citation counts as a measure of quality, to be predicted by models. @xcite proposed one of the first models which used both the papers content in the form of the paper title, abstract and keywords as well as bibliometric information. notably they used automated scripts to retrieve bibliometric information, even so their final corpus is still relatively small, containing 3788 papers. limited recent research is available on the subject of predicting the quality of papers with deep learning using the textual content. @xcite combine visual and textual content using a cnn and lstm respectively. the authors make use of the wikipedia and the arxiv datasets. the authors propose a joint model that classifies the quality of papers. to generate textual embeddings, the authors use a bi-directional lstm model similar to the one proposed by the same authors in @xcite . the input to the model is the word embeddings of a paper, obtained using glove, and the output is a textual embedding. some recent work focuses on predicting the number of citations from the paper text augmented with review text. to do so, @xcite created a dataset of abstracts and reviews from the iclr and nips conferences: 1739 abstracts with a total of 7171 reviews for iclr and 384 abstracts with 1119 reviews for nips. @xcite collect a dataset of 3427 papers with 12260 reviews. both papers show improvement in the results from using the review information. hierarchical sequential models hierarchical versions of sequential models have already been pioneered in the literature for a long time in the form of hierarchical rnns @xcite . more recently however, use of lstms instead of rnns and use of attention resulted in the now popular han model @xcite , which was successfully applied to sentiment ## models in this work we use and refine state-of-the-art textbased deep learning models for text classification and regression tasks: accept/reject prediction and number of citations prediction respectively. our contributions focus on hans, foot_0 which we show for these tasks to be competitive with models that use a flat bilstm encoder at their core @xcite . figure 1a shows a diagram of our han model with structure-tags added to the input, and figure 1b shows a diagram of the bilstm-based model, our baseline for comparison. as can be seen from the diagrams, both models use a bilstm at the text level that works on embeddings computed for the sentences of the text. however, while han uses the sequential order to compute an embedding, the baseline model averages word vectors, disregarding order, similar to bag-of-word representations. we also use a second baseline model: average word embeddings (awe), which simply encodes text by the average word embedding. ## accept/reject prediction on peerread the first scholarly document quality prediction task we test our methods on is accept/reject prediction on arxiv papers from the peerread dataset @xcite . this dataset is chosen because of the large amount of earlier work in the literature reporting results on it, allowing comparison against the state-of-the-art on a well studied task. the full peerread dataset holds 14784 papers in total, each of which contains implicit or explicit accept/reject labels. furthermore, peerread contains different subsets of papers. the largest subset consists of arxiv papers (11778) in three computerscience sub-domains: foot_2 machine learning (cs.lg), computation and language (cs.cl), artificial intelligence (cs.ai), and has only accept/reject labels; this is the dataset that we use. a part of the papers also include reviews (3006 papers) and a subset of the latter also contains aspect scores (586 papers). however, of these papers with reviews, the large majority is from nips (2420 papers), and those papers are all accepted. as the arxiv portion is relatively larger, and accept/reject labeled, most work has focused on the task of accept/reject prediction for the papers in this set. ## number of citations prediction the second task we test our models on is number of citations prediction. a key advantage of this task is that large datasets can be obtained relatively easy by leveraging public sources such as the semantic scholar database. in contrast, obtaining accept/reject labels in large quantities typically requires having an agreement with publishers, and even then because of legal problems, it is hard to obtain and publish such data. 6 yet, how useful it is to predict the number of citations? more specifically: is the number of citations of a paper predictive of its quality? intuitively one would expect this to be the case at least to some extent. figure 3 shows histograms of the numbers of citations of articles from the peerread datasets for accepted and rejected papers. 7 while there are some differences between the two domains, the main trend is the same in both cases: for rejected papers, the counts are peaked around zero citations and quickly decrease to one or zero for high citation counts. in contrast, the number of citations for accepted papers is two to three times higher on average, depending on the domain. finally, we formally computed correlation in the form of the spearman rank-order correlation coefficient (ρ) and associated p-value for both domains. for both domains, the value of ρ is high and the p-value extremely close to zero, which indicates significant correlation can be concluded at all p-levels of significance for a two-sided test. these histograms and numbers prove that there is indeed a strong correlation between acceptance/rejection and the number of citations. therefore it makes sense to consider the number of citations as an imperfect but nonetheless useful proxy for the quality of scholarly documents. ## conclusion this work showed the usefulness of han and rich context tags to the processing of scientific documents. consistent improvements in prediction quality were obtained for both accept/reject estimation and number of citations prediction for han when adding structure-tags. a strong and significant correlation between accept/reject labels and number of citations was demonstrated, signaling the usefulness of the latter as a measure of scholarly document quality. with more training data, as available on the citation-score prediction task, han with structure-tags outperforms the strong and recently proposed scholarly document quality prediction models that we compared to in this study.