  Social scientists have shown that up to 50% if the content posted to a news
article have no relation to its journalistic content. In this study we propose
a classification algorithm to categorize user comments posted to a new article
base don their alignment to its content. The alignment seek to match user
comments to an article based on similarity off content, entities in discussion,
and topic. We proposed a BERTAC, BAERT-based approach that learn jointly
article-comment embeddings and infers the relevance class of comments. We
introduce an ordinal classification loss that penalizes the difference between
the predicted and true label. We conduct a thorough study to show influence of
the proposed loss on the learning process. The results on five representative
news outlets show that our approach can learn the comment class with up to 36%
average accuracy improvement compering to the baselines, and up to 25%
compering to the BA-BC model. BA-BC is out approach that consists of two models
aimed to capture dis-jointly the formal language of news articles and the
informal language of comments. We also conduct a user study to evaluate human
labeling performance to understand the difficulty of the classification task.
The user agreement on comment-article alignment is "moderate" per
Krippendorff's alpha score, which suggests that the classification task is
difficult.
