Ddup - towards a deduplication framework utilising apache spark.
Spark: Cluster computing with working sets