  Recently, research on explainable recommender systems (RS) has drawn much
attention from both academia and industry, resulting in a variety of
explainable models. As a consequence, their evaluation approaches vary from
model to model, which makes it quite difficult to compare the explainability of
different models. To achieve a standard way of evaluating recommendation
explanations, we provide three benchmark datasets for EXplanaTion RAnking
(denoted as EXTRA), on which explainability can be measured by ranking-oriented
metrics. Constructing such datasets, however, presents great challenges. First,
user-item-explanation interactions are rare in existing RS, so how to find
alternatives becomes a challenge. Our solution is to identify nearly duplicate
or even identical sentences from user reviews. This idea then leads to the
second challenge, i.e., how to efficiently categorize the sentences in a
dataset into different groups, since it has quadratic runtime complexity to
estimate the similarity between any two sentences. To mitigate this issue, we
provide a more efficient method based on Locality Sensitive Hashing (LSH) that
can detect near-duplicates in sub-linear time for a given query. Moreover, we
plan to make our code publicly available, to allow other researchers create
their own datasets.
