  Distant supervision (DS) is a well established technique for creating
large-scale datasets for relation extraction (RE) without using human
annotations. However, research in DS-RE has been mostly limited to the English
language. Constraining RE to a single language inhibits utilization of large
amounts of data in other languages which could allow extraction of more diverse
facts. Very recently, a dataset for multilingual DS-RE has been released.
However, our analysis reveals that the proposed dataset exhibits unrealistic
characteristics such as 1) lack of sentences that do not express any relation,
and 2) all sentences for a given entity pair expressing exactly one relation.
We show that these characteristics lead to a gross overestimation of the model
performance. In response, we propose a new dataset, DiS-ReX, which alleviates
these issues. Our dataset has more than 1.5 million sentences, spanning across
4 languages with 36 relation classes + 1 no relation (NA) class. We also modify
the widely used bag attention models by encoding sentences using mBERT and
provide the first benchmark results on multilingual DS-RE. Unlike the competing
dataset, we show that our dataset is challenging and leaves enough room for
future research to take place in this field.
