Ddup - towards a deduplication framework utilising apache spark.
Dedoop: Efficient Deduplication with Hadoop