80+ Free Big Data Resources to Satisfy Your Knowledge Appetite - part 2
Tags: datascience, resources, distributedsystems, machinelearning
Published: 2019-12-29 19:12:20

This is a continuation of the resources I listed in part 1

{% link https://dev.to/tariqabughofa/80-free-big-data-resources-to-satisfy-your-knowledge-appetite-part-1-4361 %}


This part includes the following four categories:

- [Machine Learning & Algorithms in Big Data](#ml)
- [Data Processing Systems](#process)
- [Real-time Processing](#realtime)
- [Graph Processing](#graph)


### <a name="ml"></a>Machine Learning and Algorithms in Big Data

<a href="https://engineering.fb.com/core-data/recommending-items-to-more-than-a-billion-people/" target="_blank">Recommending items to more than a billion people</a>: An article about collaborative filtering at Facebook.

<a href="https://h2o-release.s3.amazonaws.com/h2o/rel-xia/3/docs-website/h2o-docs/booklets/SparklingWaterBooklet.pdf" target="_blank">Machine Learning with Sparkling Water</a>: Using H2O the machine learning framework with Apache Spark.

<a href="https://stanford.edu/~rezab/sparkworkshop/slides/xiangrui.pdf" target="_blank">MLlib</a>: Scalable Machine Learning library on Apache Spark from Stanford/Databricks.

<a href="https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf" target="_blank">TensorFlow</a>: the famous large-scale machine learning library.

Large-scale parallel collaborative filtering for the Netflix prize: an algorithm that for large scale recommendations of Netflix movies.

### <a name="process"></a>Data Processing Systems

<a href="https://medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8" target="_blank">Airflow</a>: a workflow management system by AirBnB.

Oozie: a workflow management system for Hadoop by Yahoo!.

<a href="https://sameeragarwal.github.io/blinkdb_eurosys13.pdf" target="_blank">BlinkDb</a>: analytics on large scale data from Berkeley.

<a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35650.pdf" target="_blank">FlumeJava</a>: a library for developing parallel data pipelines from Google.

<a href="https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf" target="_blank">MapReduce</a>: the google framework behind Hadoop.

<a href="https://pdfs.semanticscholar.org/fe1c/c4e034ad4d3a54a5aa0a53a24b6f564298c4.pdf" target="_blank">Pig</a>: an engine that supports <a href="http://infolab.stanford.edu/~olston/publications/sigmod08.pdf" target="_blank">PigLatin</a> a procedural dataflow language for Hadoop from Yahoo.

<a href="https://event.cwi.nl/lsde/papers/hive.pdf" target="_blank">Hive</a> <a href="http://infolab.stanford.edu/~ragho/hive-icde2010.pdf" target="_blank">(resource#2)</a>: A data warehouse on top of Hadoop.

<a href="https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf" target="_blank">The Dataflow Model</a>: the model behind Google Cloud Dataflow which provides simplified stream and batch processing.

<a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41378.pdf" target="_blank">MillWheel</a>: stream processing engine from Google.

<a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41318.pdf" target="_blank">Photon</a>: A tool to join data streams at Google.

<a href="https://d0.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf" target="_blank">Kinesis</a>: stream processing engine from Amazon.

<a href="http://asterios.katsifodimos.com/assets/publications/flink-deb.pdf" target="_blank">Apache Flink</a> <a href="https://www.ververica.com/blog/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink" target="_blank">(resource#2)</a>: stream and batch processing engine from TU Berlin.

<a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/trill-vldb2015.pdf" target="_blank">Trill</a>: incremental data analytics engine from Microsoft.

<a href="http://notes.stephenholiday.com/Kafka.pdf" target="_blank">Kafka</a>: the famous distributed messaging system from LinkedIn.

<a href="https://people.eecs.berkeley.edu/~alig/papers/spark-cacm.pdf" target="_blank">Apache Spark</a>: the famous <a href="https://people.csail.mit.edu/matei/papers/2013/sosp_spark_streaming.pdf" target="_blank">stream</a> and <a href="https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf" target="_blank">batch</a> processing engine. It uses distributed memory abstractions: <a href="https://www.kdnuggets.com/2017/08/three-apache-spark-apis-rdds-dataframes-datasets.html" target="_blank">RDDs, Dataframes, and Datasets</a>. Since Spark 2 was released, it moved to <a href="https://cs.stanford.edu/~matei/papers/2018/sigmod_structured_streaming.pdf" target="_blank">structured streaming</a> [(resource#2)](https://databricks.com/blog/2017/01/19/real-time-streaming-etl-structured-streaming-apache-spark-2-1.html) <a href="https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html" target="_blank">(3)</a> <a href="https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html" target="_blank">(4)</a> and the <a href="https://amplab.cs.berkeley.edu/wp-content/uploads/2015/03/SparkSQLSigmod2015.pdf" target="_blank">SparkSQL</a> library was introduced to allow SQL queries over Spark Dataframes. The whole <a href="https://databricks.com/blog/category/engineering" target="_blank">Databricks blog</a> is a great resource for the project.

<a href="https://cs.stanford.edu/~matei/papers/2016/sigmod_sparkr.pdf" target="_blank">SparkR</a>: a Spark library to write processing application in R.

<a href="http://www.istc-cc.cmu.edu/publications/papers/2013/grades-graphx_with_fonts.pdf" target="_blank">GraphX</a> <a href="https://amplab.cs.berkeley.edu/wp-content/uploads/2014/09/graphx.pdf" target="_blank">(resource#2)</a>: distributed graph processing with Spark's RDDs.

<a href="https://cs.stanford.edu/~matei/papers/2016/grades_graphframes.pdf" target="_blank">GraphFrames</a>: distributed graph processing with Spark's Dataframes.

<a href="https://www.snappydata.io/snappy-industrial" target="_blank">SnappyData</a> <a href="http://cidrdb.org/cidr2017/papers/p28-mozafari-cidr17.pdf" target="_blank">(resource#2)</a>: a transaction datastore on top of Spark.

### <a name="realtime"></a>Real-time Processing

<a href="https://engineering.linkedin.com/blog/2018/11/samza-1-0--stream-processing-at-massive-scale" target="_blank">Samza</a> <a href="https://martin.kleppmann.com/papers/kafka-debull15.pdf" target="_blank">(resource#2)</a> <a href="http://www.vldb.org/pvldb/vol10/p1634-noghabi.pdf" target="_blank">(3)</a> <a href="https://martin.kleppmann.com/papers/samza-encyclopedia.pdf" target="_blank">(4)</a>: Stream processing engine from LinkedIn.

<a href="https://cs.brown.edu/courses/csci2270/archives/2015/papers/ss-storm.pdf" target="_blank">Storm</a>: real-time data processing engine from Twitter.

<a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/heron_icde-1.pdf" target="_blank">Heron</a>: the new Storm from Twitter.

<a href="https://research.fb.com/wp-content/uploads/2016/11/realtime_data_processing_at_facebook.pdf" target="_blank">Real-time data processing at facebook</a>.

<a href="https://tech.ebayinc.com/engineering/announcing-pulsar-real-time-analytics-at-scale/" target="_blank">Pulsar</a>: real-time data processing engine from eBay.

### <a name="graph"></a>Graph Processing

<a href="https://stanford.edu/~rezab/papers/wtf_overview.pdf" target="_blank">WTF</a>: the who to follow service at Twitter.

<a href="https://www.vldb.org/pvldb/vol9/p1281-sharma.pdf" target="_blank">GraphJet</a>: real-time recommendation graph engine at Twitter.

<a href="https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p135-malewicz.pdf" target="_blank">Pregel</a>: large-scale graph processing engine at Google.

<a href="http://www.vldb.org/pvldb/vol8/p1804-ching.pdf" target="_blank">Giraph</a>: open source implementation of Pregel by Facebook.