Training recurrent neural networks on long texts, in particular scholarly documents, causes problems for learning. While hierarchical attention networks (HANs) are effective in solving these problems, they still lose important information about the structure of the text. To tackle these problems, we propose the use of HANs combined with structure-tags which mark the role of sentences in the document. Adding tags to sentences, marking them as corresponding to title, abstract or main body text, yields improvements over the state-of-the-art for scholarly document quality prediction : substantial gains on average against other models and consistent improvements over HANs without structure-tags . The proposed system is applied to the task of accept/reject prediction on the PeerRead dataset and compared against a recent BiLSTM-based model and joint textual+visual model . It gains 4.7\% accuracy over the best of both models on the computation and language domain and loses 2.4\% against the best of both on the machine learning domain . Compared to plain HANs, accuracy increases on both domains, with 1.5\% and 2\% respectively . We also obtain improvements when introducing the tags for prediction of the number of citations for 88k scientific publications that we compiled from the Allen AI S2ORC dataset. For our HAN-system with structure-tags we reach 28.5\% explained variance, an improvement of  1.0\% over HANs without structure-tags .
Training recurrent neural networks on long texts, in particular scholarly documents, causes problems for learning. While hierarchical attention networks (HANs) are effective in solving these problems, they still lose important information about the structure of the text. To tackle these problems, we propose the use of HANs combined with structure-tags which mark the role of sentences in the document. Adding tags to sentences, marking them as corresponding to title, abstract or main body text, yields improvements over the state-of-the-art for scholarly document quality prediction . The proposed system is applied to the task of accept/reject prediction on the PeerRead dataset and compared against a recent BiLSTM-based model and joint textual+visual model as well as against plain HANs . Compared to plain HANs, accuracy increases on all three domains. On the computation and language domain our new model works best overall, and increases accuracy 4.7\% over the best literature result . We also obtain improvements when introducing the tags for prediction of the number of citations for 88k scientific publications that we compiled from the Allen AI S2ORC dataset. For our HAN-system with structure-tags we reach 28.5\% explained variance, an improvement of 1.8\% over our reimplementation of the BiLSTM-based model as well as 1.0\% improvement over plain HANs .
D
: substantial gains on average against other models and consistent improvements over HANs without structure-tags
R
. It gains 4.7\% accuracy over the best of both models on the computation and language domain and loses 2.4\% against the best of both on the machine learning domain
as well as against plain HANs
R
both domains, with 1.5\% and 2\% respectively
all three domains. On the computation and language domain our new model works best overall, and increases accuracy 4.7\% over the best literature result
A
1.8\% over our reimplementation of the BiLSTM-based model as well as
R
over HANs without structure-tags
improvement over plain HANs