A Case Study of In-House Competition for Ranking Constructive Comments in a News Service (2021)

Abstract: Ranking the user comments posted on a news article is important for online news services because comment visibility directly affects the user experience. Research on ranking comments with different metrics to measure the comment quality has shown “constructiveness” used in argument analysis is promising from a practical standpoint. In this paper, we report a case study in which this constructiveness is examined in the real world. Specifically, we examine an in-house competition to improve the performance of ranking constructive comments and demonstrate the effectiveness of the best obtained model for a commercial service.

## introduction in online news services, the user comments posted on news articles function as a type of useful content known as user-generated content (ugc). figure 1 shows examples of comments posted on yahoo! japan news, a japanese news portal. 1 by reading these comments along with the article, users can obtain supplementary information such as other users' opinions, experiences, and simplified explanations of the article. there is a limit, however, on the number of comments that can be displayed on a page, and as users typically do not have the time or inclination to read through all the comments, ideally they should be ranked in some way. prioritizing the comments for display is directly linked to user satisfaction, so improving this ranking is an important issue for such services. there have already been multiple studies on comment ranking in online news services and discussion forums @xcite @xcite . all of these studies have utilized user feedback (e.g., "like"-button clicks in figure 1 ) as their ranking metrics. although such user feedback is easy to obtain, this type of measurement has two drawbacks: (i) user feedback does not always satisfy the service provider's needs, such as to create a fair place (i.e., a news space that is neutral), and (ii) user feedback will be biased by where comments appear in a comment thread (also known as "position bias" @xcite ). a typical example for (i) can be seen in political comments, where the "goodness" of the comment tends to be decided on the basis of the political views of the majority of the users rather than on its quality. a typical example of (ii) can be illustrated by a case where earlier comments tend to receive more feedback since they are displayed at the top of the page, which implies later comments will be ignored irrespective of their quality. to resolve this issue, @xcite introduced a metric representing a comment's constructiveness (see section 2 for details), which has also been studied in argument analysis @xcite . interestingly, they found empirical evidence that the constructiveness has no correlation with the user feedback, which has been commonly used for ranking comments. this implies that we need to consider the constructiveness rather than the user feedback to avoid unfavorable situations (i) and (ii) in real services. in this paper, we take their study one step further towards practical application. specifically, in collaboration with yahoo! japan news, we report a case study of deploying a model that ranks constructive comments in a commercial service. the characteristic unique point of our study is that we aim to improve the ranking quality through an inhouse competition. as represented by @xcite , the machine learning competition platform, it has become common to improve a model's performance through a competition format. this kind of experiment has also been conducted in various research areas through shared-task workshops, with the wmt translation task @xcite , tac text analysis task @xcite , and ntcir information retrieval task (kato and liu, 2019) being well-known examples. following this trend, we also aim to improve the ranking performance through a competition format. as this kind of work conducted within a company towards a commercial service is rarely released in the form of an academic paper, we expect our findings to become valuable knowledge for practitioners in the field. we clarify the novelty of our study against other previous studies in section 7. our main contributions are as follows: • we report the details of the in-house competition (i.e., constructive comment ranking task) conducted in a commercial news service, yahoo! japan news, where we obtained a new model with a 2.73% improvement in performance (ndcg) compared to the baseline (section 3). we also administer a participant survey and discuss positive and negative opinions relating to this competition (section 6). ## preliminaries constructiveness: we use the concept of constructiveness to prioritize comments that provide insight and encourage healthy discussion. according to the dictionary (oxford, 2020), the term "constructive" is defined as "having or intended to have a useful or beneficial purpose." however, this dictionary definition is a bit too generic to determine whether a comment is constructive or not. to avoid individual variation as much as possible, we need a more specific definition for our task. thus, we follow a previous study @xcite on constructiveness, where a questionnaire administered to 100 people clarified the detailed conditions for constructive comments. yjccr dataset: we use (part of) the yj constructive comment ranking (yjccr) dataset, which was created by @xcite . the yjccr dataset consists of more than 100k japanese comments labelled with a constructiveness score (c-score), which is a graded numeric score representing the level of constructiveness for ranking comments. the c-score was defined as the number of crowdsourced workers who judged a comment as constructive in response to a yes-or-no (binary) question. as a consequence, the c-score indicates how many people think that a comment is constructive with the goal of sufficiently satisfying as many users as possible. the detailed settings of the crowdsourcing were as follows. the task was prepared with questions referencing a news article and its comments extracted from yahoo! japan news and conducted on a crowdsourcing service. the workers were asked to read the definition of constructiveness and then judge whether each comment was con-structive. to ensure reliability, only the results of serious workers who correctly answered qualitycontrol questions that were randomly included in each task were kept. ten workers were used for each comment in the dataset, so a c-score of 8, for example, means that eight workers judged a comment as constructive. the reliability of this annotation was confirmed with krippendorff's alpha, which was "moderate agreement." the comments in figure 1 are actual ones in the yjccr dataset. the lower comment has a high score (9) because it includes a constructive opinion with some reasoning, whereas the upper comment has a low score (0) since it includes offensive content (see appendix c for more examples). ## in-house competition task: the competition task consisted of ranking comments based on their degree of constructiveness, that is, the c-score defined in section 2. specifically, given that we have training data with triples {(a, x, y)} consisting of a news article a, a comment x on the article, and its corresponding c-score y, the task is to predict the ranking of comments for every article in the test dataset {(a, x)}, where the c-scores are unknown. the goal of this task is to create a model that predicts the correct ranking from the training data as closely as possible. the competition was held for about six weeks (dec. 13, 2018 -jan. 23, 2019), and a dozen employees related to the comment ranking service were made aware of it. the information shared among them included not only the dataset but also sample code consisting of a simple feature extraction, model creation, and evaluation pipeline in order to reduce the burden on the participants. we also prepared a leaderboard to display the latest evaluation results for submitted models. the participants reported their evaluation results on the leaderboard and were able to update them any number of times during the competition period. dataset: the training dataset consisted of a combination of the above-mentioned public dataset yjccr and a new dataset of long comments created for this study. we used 49,215 comments (9,845 articles with five comments each) from the yjccr dataset, each comment having a c-score assigned by crowdsourcing. while this dataset only contained comments up to 125 characters in length, we noticed in our preliminary experiments that long comments tended to be incorrectly determined as constructive despite having a bigger impact on visibility than short ones. for that reason, we additionally extracted long comments (from 126 up to the maximum of 400 characters) posted to the articles in yjccr and created a long comment dataset with c-scores assigned by crowdsourcing in the same way as for yjccr, as described in section 2. the resulting combination of the above two datasets yielded 59,120 comments (9,845 articles with an average of six comments). we split it into 80% training data, 10% validation data, and 10% test data to form the competition dataset. evaluation: we used normalized discounted cumulative gain (ndcg) @xcite , which is a widely used evaluation measure for ranking tasks. in this competition, we adopted a variant defined as ndcg@@xmath0, which was also used in the yahoo! learning to rank challenge @xcite . this ndcg@k computes how close the top k comments predicted by a model are to the correct ranking, where r i is the true c-score of the comment with predicted rank i, and z k is a normalization term. to simplify the evaluation process, we set the average value of ndcg@k, i.e., 1 k k @xmath1, as the main measure in the competition, where k is the number of comments included in the article. furthermore, to particularly encourage the performance improvement for long comments, we extracted a dataset consisting of only long comments (305 articles, 917 comments) from the test data and used its ndcg@k value as a supplementary measure. this was meant to reduce the effect of submitting sloppy methods that merely determined long comments to be constructive. from here on we call the normal measure ndcg and the one for long comments ndcg-l. submitted models: eight individuals participated in the competition and submitted 14 models dur-ing the competition period (before the deadline). figure 2 shows the total number of submissions across the competition period. we can see that the number of submissions was low during the initial period of the competition but increased significantly at the start of the year (beginning of work), a period where time is relatively more available (jan. 9, 2019), and on the day of the deadline (jan. 23, 2019). moreover, after the submission deadline had passed, several participants continued to work on the task and created an additional four models. we included these additional models when carrying out our analysis, although only the models submitted before the deadline were eligible for internal awards. we obtained a wide variety of models created by the participants' trials and errors, but due to space limitations, we only discuss in detail the four highest-performing models, which were model-4, model-11, model-14, and model-17. the following list includes the summary of each model with its detailed settings and features (see appendix a for their hyperparameter settings). ## model ensemble to further improve the performance, we considered using an ensemble of the models submitted in the competition. for ease of implementation, we focused on unsupervised ensemble methods that combine predicted scores. assuming practical use, we only used the models that could accurately (or stably) reproduce their leaderboard performance, resulting in ensembles of 12 models. ## towards practical use to determine if the submitted models can be used in the running service, we carried out a qualitative evaluation from the perspective of service, not just constructiveness. specifically, we prepared the comment lists ranked by candidate models for each news article and asked three experts in the comment service to rank them. we instructed the experts to evaluate them on the basis of "which list should be provided as a service" rather than "which list is constructive," as the goal of this evaluation was to improve the service quality. as an evaluation measure, we calculated the micro-average of the ranks by the experts over the evaluation data prepared separately from the competition data. we used 104 articles (each having 3,406 comments on average) for the first evaluation and 66 articles (each having 3,888 comments on average) for the second evaluation. 3 baseline vs. naive methods: we first examined whether the constructiveness ranking model baseline is useful compared to other naive methods, which was confirmed by @xcite in terms of automatic evaluation (ndcg) only. specifically, we compared the four models described below in terms of human evaluation. average rank feedback 2.61 latest 3.42 length 2.20 baseline (c-score) 1.77 likes/dislikes. this model has been used in the service. one of the reasons model-17 performed better than the others seems to be related to the fact that it had a full neural structure (as explained in section 3), which implies "robustness" (or expressiveness of the model) thanks to a lot of parameters, as in neyshabur et al. ( 2017 )'s study. in fact, the evaluators reported that model-17 had few critical errors compared to the other models. although model-4 and model-11 performed well in ## participant survey and future issues after the competition, we collected opinions from the participants through an optional survey. we discuss certain positive and negative opinions in detail below (see appendix b for other opinions). positive opinions: the most popular opinion was that the number of model submissions was greater than initially expected. according to the participants, this was mainly due to the game element of the competition, i.e., publicly competing against other participants. in other words, the fun of the task was an implicit incentive to encourage submissions. as a result, we were able to use a wide variety of models for the ensemble experiment (section 4), which seems to have contributed to the performance improvement. another interesting opinion was about disclosure of the modeling methods. in this competition, the participants were encouraged to include model descriptions such as structures and features when reporting their evaluation results on the leaderboard. this information helped the participants make improved models, which contributed to the best performance of single models (section 3). other positive opinions were related to the improved knowledge and skills acquired by the participants. negative opinions: one major negative opinion was about the leaderboard system, where the participants individually posted their own results pertaining to the evaluation tool and test data. this setting allowed the participants to purposefully design models effective only on the test data, although we confirmed that they actually used the validation data for fine-tuning. to hold a competition on a larger scale, we should prepare an automatic evaluation system with private test data. such a setting is relatively common in strict competitions such as kaggle, while most test datasets tend to be publicly available in research communities (under research ethics). another insightful opinion was to make an incentive for exploring new directions, since it is valuable to obtain findings in unknown/rare directions, even if the results are not superior. in addition, model diversity can contribute to the ensemble performance, as discussed above. we suggest preparing a special prize for novelty in order to encourage exploring different directions. constructiveness has been introduced in argument analysis frameworks @xcite @xcite . the purpose of these studies was to classify constructive comments, whereas @xcite recently expanded their tasks to a ranking one. they created a new dataset for ranking constructive comments on a news service and showed that the commonly used method that ranks comments by user feedback does not contribute to constructiveness in terms of automatic evaluation (ndcg). our study has value as a deployment report of their approach, and we also confirmed that constructiveness performed better than user feedback for ranking comments in terms of human evaluation by experts. aside from constructiveness and user feedback, we may consider hate speech detection @xcite @xcite and sentiment analysis @xcite as alternative approaches for analyzing the quality of comments on the basis of their content. although these approaches are useful for other tasks, they do not directly solve our task, namely, ranking constructive comments. for example, the simple comment "great!" is positive and is not hate speech, but it is not suitable as a top-ranked comment in our task. shared tasks and competitions: there have been many competitions in various research areas through shared-task workshops, such as the wmt translation task @xcite , tac text analysis task @xcite , and ntcir information retrieval task @xcite . their purpose to find good models for a specific task is almost the same as ours, and the main difference (ignoring the task) is that the competition in our work was conducted within a company. as this kind of work towards a commercial service is rarely released in the form of an academic paper, we expect that our findings will become valuable knowledge for practitioners in this field. as for "learning to rank" tasks, there have also been several competitions such as the internet mathematics 2009 @xcite , the yahoo! learning to rank challenge @xcite , and the personalized web search challenge @xcite . their tasks are basically to rank pages in terms of relevance to a search query, which is common in the information retrieval field. in contrast, our task is to rank comments in terms of constructiveness. it has value in the sense of applying the concept of argument analysis in the real world. a unique aspect of our work is the ensemble of submitted models in the competition. although there have been many studies on model ensembles @xcite @xcite , the models for prior ensemble experiments were basically prepared by either random initialization or a researcher's preference, which is different from our competition setting. the most closely related study involves the concept of "resource by collaborative contribution (rbcc)" @xcite , which collaboratively creates a large-scale dataset for named entity recognition by using the predicted labels of submitted models in a shared task, although their purpose and task were completely different from ours. we believe our findings in a commercial service will be useful for future ensemble studies. ## conclusion we reported a case study of an in-house competition for ranking constructive comments. our experimental results showed that the competition format is effective for testing various model structures, and that ensembling submitted models can further improve the ranking performance. moreover, we confirmed that the submitted models were practically useful in a service perspective evaluation. samples ('neg') = 5, and loss function ('loss') = 'softmax'.