Building a Raspberry Pi Hadoop / Spark Cluster
Tags: raspberrypi, showdev, hadoop, java
Published: 2019-07-08 14:07:03

> _Pro tip: if you're only looking for how to configure Hadoop and Spark to run on a cluster, [start here](#networking)._

### <a name="top">Table of Contents</a>

1. [Motivation and Background](#introduction)
    - What is Apache Hadoop?
    - Map-Reduce and Parallelisation
    - What is Apache Spark?
2. [Hardware](#hardware)
    - About Raspberry Pi
    - Choosing a Pi Model
    - Power over Ethernet (PoE) and Networking
    - Disk Space
    - Cost and Overall Specs
    - Installing the PoE HATs
3. [Software](#software)
    - Operating System
    - Networking
    - Securing the Cluster
4. [Hadoop & Spark](#hadoopspark)
    - Single-Node Setup
    - Cluster Setup
5. [Conclusion](#conclusion)

## <a name="introduction">Motivation and Background</a>

"Big Data" has been an industry buzzword for nearly a decade now, though agreeing on what that term means and what the field of Big Data Analytics encompasses have been points of contention. Usage of Big Data tools like The Apache Software Foundation's [Hadoop](https://hadoop.apache.org/) and [Spark](https://spark.apache.org/) (H&S) software has been met with [scorn](https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html) and [praise](https://www.researchgate.net/publication/281403776_Big_Data_And_Hadoop_A_Review_Paper) alike. The truth is that, like any other tools, H&S are only helpful if you know when and how to use them appropriately.

[Many companies](https://www.eweek.com/enterprise-apps/how-apache-spark-helped-eight-companies-grow-their-businesses) use H&S for data storage and analysis. Developers with these skills are in demand, but until recently, it was difficult to get the necessary experience with cluster computing without a big investment of time and/or money. The wide availability of small, cheap single-board computers like the brand-new [Raspberry Pi 4](https://www.raspberrypi.org/) ($35 starting price) has all but eliminated these barriers to developing your own cluster.

In this guide, __I will teach you how to build a networked cluster of Raspberry Pis__. We'll get them communicating with each other through a network switch, install HDFS, and have Spark running distributed processing jobs via YARN across the whole cluster. This guide is a full-featured introduction to the hardware and software involved in setting up an H&S cluster, and can scale (just like H&S!) to any number or size of machines.

### What is Apache Hadoop?

The Apache Hadoop Distributed Filesystem (HDFS) is a distributed (networked) filesystem which can be run on off-the-shelf, consumer-grade hardware. Running on networked commodity hardware means that HDFS is both scalable and cost-effective. HDFS splits files into "blocks" (typically 64MB in size) and stores multiple copies of these blocks across different "DataNodes" on the same rack, different racks in the same data storage centre, or across multiple data storage centres.

<figure>
  <img src="https://thepracticaldev.s3.amazonaws.com/i/mtjl1qm6z9h4gxlmcsgq.PNG" alt="A simplified diagram of the hardware architecture of a Hadoop Distributed File System (HDFS) installation."/>
<figcaption>
HDFS Hardware Architecture Diagram
</figcaption>
</figure>

Data split and stored in this way has several advantages:

1. __Fault tolerance.__ Redundant copies are made of each block and stored across multiple physical locations. This means that if a disk fails (as is to be expected with large volumes of commodity hardware), all of the blocks on that node can be sourced from other locations. HDFS can also make additional copies of the lost blocks so that the desired number of redundant blocks is always maintained.

    > In older versions of Hadoop, the single NameNode was a potential vulnerability for the entire system. Since the NameNode holds all of the information about the filesystem and changes made to it, a failed NameNode compromises the whole cluster. In newer releases, [a cluster can have multiple NameNodes](https://fr.hortonworks.com/blog/hdfs-metadata-directories-explained/), eliminating this single point of failure.

2. __Parallelisation.__ Data split into blocks is ripe for parallel processing with a [map-reduce](https://en.wikipedia.org/wiki/MapReduce) analysis pipeline. A file split into 100 blocks across 100 similarly-powered machines can be processed in roughly 1/100th of the time it would take to process the same file on a single machine.

3. __Data Locality.__ In-situ processing of blocks also eliminates the need to transfer large data files across the network. This can drastically reduce network bandwidth requirements. Time and computing power that would have otherwise been spent copying files from a repository can instead be used to process the data locally: _"moving computation is cheaper than moving data"_.

4. __Large datasets.__ Because HDFS breaks files into blocks, the size of any individual file is limited only by the total amount of storage available across the network. HDFS is POSIX-based but [relaxes some POSIX requirements to allow fast streaming of data](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html), among other benefits. HDFS can support hundreds of networked nodes and tens of millions of files.

HDFS speeds up data processing by distributing the I/O (read/write) disk latency across all disks on the network. Instead of reading a file from a single location on a single disk, it's read simultaneously from multiple points by multiple machines. Those same machines can then operate on that same file in parallel, increasing processing speeds by orders of magnitude.

__One point to note__ for HDFS is that, to facilitate data coherency, files written to HDFS are [immutable](https://www.linkedin.com/pulse/20140528113353-16837833-6-benefits-of-programming-with-immutable-objects-in-java). This means that, for a file to be edited, it must be downloaded from HDFS to a local file system, changed, and uploaded back to HDFS. This workflow is a bit different from what most users are accustomed to, but it is necessary for the high-throughput data access provided by HDFS. HDFS is not meant to replace your normal, interactive filesystem. It should be used as a repository for your Big Data, which won't change regularly, but which needs to be processed quickly and easily. It's a "write one time, read many times" file system.

> You can read much more about the nitty-gritty architectural details of HDFS [here](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html), if you're interested.

### Map-Reduce and Parallelisation

The distributed nature of the data stored on HDFS makes it ideal for processing with a __map-reduce__ analysis framework. Map-reduce (also "MapReduce", "Map-Reduce", etc.) is a programming technique where, as much as possible, parallelisable tasks are performed concurrently, followed by any non-parallelisable "bottlenecks". Map-reduce is a general framework for analysis and is not a particular algorithm. (In Big Data circles, however, it is sometimes synonymous with Apache's __Hadoop MapReduce__, which is discussed in detail below.)

<figure>
  <img src="https://thepracticaldev.s3.amazonaws.com/i/2hzhe1o9g837vov865up.PNG" alt="A diagram showing an example map-reduce analysis task: counting the frequency of letters in a database of words."/>
<figcaption>
An Example Map-Reduce Pipeline
</figcaption>
</figure>

Some data analysis tasks are _parallelisable_. For instance, if we wanted to find the most common letters among all of the words in a particular database, we might first want to count the number of letters in each word. As the frequency of letters in one word don't affect the freqency of letters in another, the two words can be counted separately. If you have 300 words of roughly equal length and 3 computers to count them, you can divvy up the database, giving 100 words to each machine. This approach is very roughly 3x as fast as having a single computer count all 300 words. Note that tasks can also be parallelised across CPU cores.

> Note: there is some overhead associated with splitting data up into chunks for parallel analysis. So if those chunks cannot be _processed_ in parallel (if only one CPU core on one machine is available), a parallelised version of an algorithm will usually run _more slowly_ than its non-parallelised counterpart.

Once each machine in the above example has analysed all of its 100 words, we need to synthesise the results. This is a _non-parallelisable_ task. A single computer needs to add up all of the results from all of the other machines so that the results can be analysed. Non-parallelisable tasks are _bottlenecks_, because no further analysis can even be started until they are complete.

__Common parallelisable tasks include:__

- filtering (ex: remove invalid or incomplete data)
- transformation (ex: format string data, interpret strings as numeric)
- streaming calculations (ex: sum, average, standard deviation, etc.)
- binning (ex: frequency, histogramming)

__Common non-parallelisable tasks include:__

- aggregation (ex: collecting _partial_ results into a single _global_ result)
- [text parsing](https://scicomp.stackexchange.com/a/1423) (ex: regular expressions, syntax analysis)
- visualisation (ex: creating summary plots)
- mathematical modelling (ex: linear regressions, machine learning)

_Sorting_ data is an example of an algorithm which doesn't fit nicely into either of the above categories. Although the entire dataset necessarily needs to be collected into one location for complete _global_ sorting, sorting small collections of data which themselves are already _locally_ sorted is much faster and easier than sorting the equivalent amount of unsorted data. Sorting data in this way is essentially both a _map_ and a _reduce_ task.

> Parallelisation is not appropriate for all tasks. Some algorithms are [inherently sequential](https://cs.stackexchange.com/q/19643/86266) (aka. [P-complete](https://en.wikipedia.org/wiki/P-complete)). These include _n-body problems_, the _circuit value problem_, [_Newton's Method_](https://stackoverflow.com/a/18774268/2925434) for numerically approximating the roots of a polynomial function, and _hash-chaining_, which is widely used in cryptography.

### What is Apache Spark?

When HDFS was [first released](https://en.wikipedia.org/wiki/Apache_Hadoop) in 2006, it was coupled with a map-reduce analysis framework called -- creatively enough -- [Hadoop MapReduce](https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html) (usually just "MapReduce"). Both HDFS and MapReduce were inspired by research at Google, and are Apache counterparts to Google's ["Google File System"](https://en.wikipedia.org/wiki/Google_File_System) and ["MapReduce"](https://en.wikipedia.org/wiki/MapReduce), the latter of which Google was granted a patent for (which has been criticised).

Hadoop MapReduce is the original analysis framework for working with data stored on HDFS. MapReduce executes map-reduce analysis pipelines (described above), reading data from HDFS before the "map" tasks, and writing the result back to HDFS after the "reduce" task. This behaviour in particular is one of the reason's why [Apache Spark](https://spark.apache.org/), widely seen as a successor to MapReduce, offers a [speedup of 10-100x](https://www.scnsoft.com/blog/spark-vs-hadoop-mapreduce), relative to MapReduce.

> _Hadoop MapReduce works with the HDFS to process “process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.”_ [[ source ]](https://medium.com/@jayhuang75/a-simple-mapreduce-in-go-42c929b000c5)

<a href="https://www.edureka.co/blog/apache-spark-vs-hadoop-mapreduce" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/yylxwm1zanpdhef21ivh.jpg" alt="Chart showing Hadoop vs. Spark performance on a logistic regression as a function of iterations"/><figcaption>Hadoop vs. Spark performance on a logistic regression</figcaption></figure></a>

Spark offers at least four primary advantages over MapReduce:

1. __Spark minimises unnecessary disk I/O.__ Spark offers several improvements over MapReduce in an effort to read from and write to disk as little as possible.* While MapReduce [writes every intermediate result to disk](https://www.quora.com/What-makes-Spark-faster-than-MapReduce), Spark tries to pipeline results as much as possible, only writing to disk when the user demands it, or at the end of an analysis pipeline. Spark will also cache data which is used for multiple operations in memory, so it doesn't need to be read from the disk multiple times. For these reasons, Spark is sometimes said to have "in-memory processing". (This is a bit of a misleading term, though, as both MapReduce and Spark necessarily process data in RAM.)

    > \* In general, [reading from and writing to a CPU's cache](https://linustechtips.com/main/topic/34636-what-is-the-bandwith-speed-of-l1l2-and-l3-cache/?do=findComment&comment=10911961) is an order of magnitude faster than RAM, and RAM is a few orders of magnitude faster than SSD (which is [faster still](https://gist.github.com/jboner/2841832) than a conventional hard disk).

2. __Spark provides abstractions for fault-tolerant processing.__ The primary data structure provided by Spark is the [Resilient Distributed Dataset (__RDD__)](https://cs.stanford.edu/~matei/papers/2012/nsdi_spark.pdf):
    - __Resilient__ -- [Spark keeps a _lineage_](https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-lineage.html) of how a given RDD is constructed from any "parent" RDDs. If any RDD is corrupted or lost, it can easily be recreated from its lineage graph (aka. its [_logical execution plan_](https://www.quora.com/What-is-an-RDD-Lineage)).
    - __Distributed__ -- An RDD may physically exist in different pieces over several machines. Spark cleanly abstracts away the distributed nature of the files stored on HDFS. The same code which reads and processes a single file stored on a single machine can be used to process a distributed file, broken into chunks and stored over many different physical locations.
    - __Dataset__ -- [RDDs can store](https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd.html) simple objects like `String`s and `Float`s, or more complex objects like tuples, records, custom `Object`s, and so on. These datasets are inherently parallelisable.

    > RDDs are immutable collections of data, so they are __thread safe__: they can be processed in parallel without the programmer having to worry about [race conditions](https://en.wikipedia.org/wiki/Race_condition) or other multithreading pitfalls. (The logic for making files stored on HDFS immutable is similar.)

    > RDDs are __lazily evaluated__. The sequence of calculations which must be traced to construct an RDD are not performed until the RDD needs to be _used_ -- printed to the terminal, plotted, written to a file, etc. This reduces the amount of unnecessary processing that's performed by Spark.

3. __Spark has a different processing model.__ Spark uses a [Directed Acyclic Graph (__DAG__)](https://data-flair.training/blogs/dag-in-apache-spark/) processing model, rather than a simple, two-step map-reduce pipeline:

    <a href="https://blog.daglabs.com/an-introduction-to-the-blockdag-paradigm-50027f44facb" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/nlsudiklj7tu793evsqx.png" alt="A simple diagram of a Directed Acyclic Graph (DAG)."/><figcaption><br/>Diagram of a Directed Acyclic Graph (DAG)</figcaption></figure></a>

    What this means is that Spark takes a holistic view of the entire processing pipeline and attempts to optimise the process globally. While MapReduce will (1) read data from disk, (2) perform a "map" operation, (3) perform a "reduce" operation, (4) write data to disk, Spark is more flexible about what is completed when. As long as forward progress is made toward the final result, _maps_ and _reduces_ may be performed in parallel or at different times on different chunks of data. The DAG is a more general version of MapReduce's map-reduce pipeline -- it can also be viewed as the implementation in code of the idealised _lineage graph_ of an RDD.

4. [__Spark eliminates JVM boot times.__](https://medium.com/@Nim94Sha/apache-spark-ready-get-set-stream-75bbb2dae9c) By booting up a Java Virtual Machine (JVM) on each DataNode at startup (as opposed to when a new job is executed), Spark eliminates the time required to load `*.jar`s, parse any configuration files, and so on. MapReduce opens a new JVM each time a new task is run, and all of this startup overhead is incurred on every job. It may only take a few seconds each time, but it adds up.

Additionally, Spark has built-in support for complex analytics like machine learning ([MLlib](https://spark.apache.org/mllib/)) and graph processing ([GraphX](https://spark.apache.org/graphx/)) and can be used interactively in the [Python, R, Scala, Java, and SQL](https://en.wikipedia.org/wiki/Apache_Spark#Spark_Core) shells

The above features make Spark much faster, more fault-tolerant, and more feature-rich than MapReduce. With all of the benefits Spark has over MapReduce, essentially [the only time](https://stackoverflow.com/q/30299129/2925434) you should prefer to use the latter is when you have legacy MapReduce code to maintain. Or if you don't care about processing time. Or if you don't want to spend time learning Spark.

There are other distributed data storage / distributed analysis pipelines, but Apache Hadoop and Spark are free, fast, fault-tolerant, and relatively easy to install and understand. Equipped with this foundational understanding of H&S, we can now move on to constructing our own Raspberry Pi H&S cluster.

[[ back to top ]](#top)

## <a name="hardware">Hardware</a>

### About Raspberry Pi

The [Raspberry Pi Foundation](https://en.wikipedia.org/wiki/Raspberry_Pi) have been developing and releasing single-board computers since early 2012. Originally conceived of as a way to [inspire kids to get into electronics and programming](https://www.bbc.co.uk/blogs/thereporters/rorycellanjones/2011/05/a_15_computer_to_inspire_young.html), like the [DIY "kit" computers](https://en.wikipedia.org/wiki/List_of_early_microcomputers) of the 1970s and 80s, the popularity of the Raspberry Pi [has far exceeded the expectations of the Foundation's creators](https://www.raspberrypi.org/blog/ten-millionth-raspberry-pi-new-kit/), selling 12.5 million units to become the [third most popular personal computer ever](https://www.theverge.com/circuitbreaker/2017/3/17/14962170/raspberry-pi-sales-12-5-million-five-years-beats-commodore-64), beating the Commodore 64, but [still trailing](https://www.raspberrypi.org/magpi/raspberry-pi-sales/) the Apple Macintosh and the Windows PC. Yes, Raspberry Pis have been used to introduce a new generation to computing, but they're also used for home automation, manufacturing, robotics, digital cameras, [and more](https://en.wikipedia.org/wiki/Raspberry_Pi#Reception_and_use).

<a href="https://www.modmypi.com/download/Raspberry-Pi-Comparison_r3.pdf" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/6m8qlhi6vq6kb8zce9x0.png" alt="A comparison chart showing different versions and models of Raspberry Pi."/><figcaption>Raspberry Pi Comparison Chart (click to open PDF)</figcaption></figure></a>

There are many different models of Raspberry Pi available today, and they fall into four families: Model A, Model B, Zero, and Compute. The __Model B__ family are the flagship, full-featured Raspberry Pis. Model Bs all have Ethernet connectivity, which no other family of Pi has. Every Model B has an MSRP (Manufacturer's Suggested Retail Price) of $35 US. Model Bs all have the exact same form factor: 85.6mm x 56.5mm x 17.0mm, weighing in at only 17g.

> Model B Pis are _literally_ credit-card-sized! And about as "thick" as a keyboard key is wide. They're roughly the size and weight of a deck of cards.

<a href="https://www.theengineeringprojects.com/2018/07/introduction-to-raspberry-pi-3-b-plus.html" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/lz6rdu489o8cimmrg4fs.jpg" alt="Detailed diagram of a Raspberry Bi 3 B+, indicating ports and hardware components."/><figcaption>Raspberry Pi 3 B+ Diagram</figcaption></figure></a>

There are different _versions_ of Raspberry Pi, as well. Version 1 included a Model B as well as a Model B+, which saw the inclusion of two additional USB ports, a switch from SD to microSD storage, and a change in the GPIO (General-Purpose Input/Output) header on the board which allowed for the addition of __Pi HATs__ (Hardware Attached on Top). HATs connect to the GPIO pins and add additional features to the Pi: allowing PoE (Power over Ethernet), cooling fans, LCD screens, and more.

Version 1 also included a Model A and A+. The A series are physically smaller, cheaper ($20-$25), and have reduced specs (only 1 USB port, no Ethernet, less RAM). Model Zero Pis are even more stripped-down and even cheaper ($5-$10). Details on all of these models of Pi can be seen in the table above, or found in [various](https://www.rs-online.com/designspark/rel-assets/dsauto/temp/uploaded/Raspberry_Pi_Comparison_A3_Final.png?w=815) [comparison](http://socialcompare.com/fr/comparison/raspberrypi-models-comparison) [tables](https://www.mbtechworks.com/hardware/raspberry-pi-model-comparison.html) online.

### Choosing a Pi Model

We want to network our Pis to build a Hadoop cluster, so we are restricted to models which have either Wi-Fi and/or Ethernet hardware. Of the Pis which are currently available, this limits us to the Pi 3 Model B, Pi 3 Model B+, Pi 3 Model A+, or the Pi Zero Wireless. All of these Pis have WiFi connectivity, but only the Model B and B+ have Ethernet.

> Note that the Version 4 Pis were just released a few weeks ago, but they're selling out as quickly as stores can stock them. If you can, I recommend using the Version 4 Pis over the Version 3 ones. They have faster processors and the ability to add more RAM (up to 4GB).

<a href="https://opensource.com/life/16/10/which-raspberry-pi-should-you-choose-your-project" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/h7wymzk78bqhdxi2qkql.jpg" alt="An image showing a grid of different models of Raspberry Pi."/><figcaption>Raspberry Pi Models</figcaption></figure></a>

If we network the Pis using WiFi, we'll need a wireless router. We'll also need to buy [power supplies](https://www.raspberrypi.org/products/type-c-power-supply/) for each Pi. I'm going to build an eight-Pi cluster. Eight power supplies will be bulky. I'll have to carry around all of these, plus a multi-outlet, if I want to show off my cluster to anyone. To eliminate this bulk, we can instead power them through PoE. This takes care of both the networking and the power supply for each Pi, but it is more expensive. This choice restricts me to only the Model B or the Model B+.

Finally, to eliminate any possible network latency with the Pis (and since all Models B/B+ are the same price) I'm going to chose the Pi 3 Model B+, which has the largest bandwidth Ethernet support (~300Mbit/s). For my 8-Pi cluster, the total cost of the Pis is about __$280__.

### Power over Ethernet (PoE) and Networking

Raspberry Pis Model B+ can be powered via a traditional DC power supply (micro-USB). These  cost [about $8 each](https://www.pishop.us/product/raspberry-pi-15w-power-supply-us-white/?src=raspberrypi) (so $64 total for my 8-Pi cluster) and would require each Pi to have access to its own electrical outlet. You can buy a multi-outlet ("power strip") online [for about $9](https://amzn.to/2Jm9PoE).

Alternatively, Pis B+ can also be powered via Ethernet, through a technology known, creatively, as Power over Ethernet, or PoE. PoE requires a network switch which supports PoE; these are generally more expensive than those which don't support PoE. Non-PoE, 8-port network switches can be bought [for as little as $10 on Amazon](https://amzn.to/2J9cvqW), and gigabit (1000Mbps) switches are available for as little as $15. An 8-port network switch with PoE support on all ports runs [about $75](https://amzn.to/2FRIv0N). Networking via Ethernet requires 8 short Ethernet cables ([about $14 for a 10-pack](https://amzn.to/2XnCfDF)). If you opt for the non-PoE route, you could also network your Pis with a good, cheap wireless router ([about $23](https://amzn.to/327snBw) on Amazon). 

At minimum, then, we're looking at $64 + $9 + $10 + $14 = __$97__ in networking and power supply costs for a wired (Ethernet) cluster with DC power supplies, or $64 + $9 + $23 = __$96__ in networking and supply costs for a WiFi cluster with DC power supplies.

<figure><img src="https://thepracticaldev.s3.amazonaws.com/i/bsdvkjhuzojxxxfkrhqi.jpg" alt="Photograph of a Raspberry Pi cluster using Power over Ethernet."/><figcaption>Raspberry Pi cluster using Power over Ethernet</figcaption></figure>

The cluster I'm building will be used for demonstrations, so it needs to be portable. Eight power supplies plus a multi-outlet is not ideal, so I opted for the PoE option. This eliminates all of that extra bulk, but it comes at a cost. A PoE Pi cluster requires 8 PoE HATs ([about $21 each](https://www.pishop.us/product/raspberry-pi-poe-hat/?src=raspberrypi)), plus the Ethernet cables (10 for $14), plus a PoE-enabled, 8-port network switch (about $75). This means that a PoE Pi cluster will incur at least __$257__ in networking and power supply costs, or over 2.5x as much as a wireless cluster.

<a href="https://www.c4labs.com/product/8-slot-stackable-cluster-case-raspberry-pi-3b-and-other-single-board-computers-color-options/?attribute_pa_cloudlet-case-color=clear" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/i1g2olcvgybthhhzb47p.jpg" alt="Photograph of a C4Labs Cloudlet Cluster Case, holding a Raspberry Pi cluster"/><figcaption>C4Labs Cloudlet Cluster Case, holding a Raspberry Pi cluster</figcaption></figure></a>

I have one more small complication, too. I bought [a special carrying case](https://www.c4labs.com/product/8-slot-stackable-cluster-case-raspberry-pi-3b-and-other-single-board-computers-color-options/?attribute_pa_cloudlet-case-color=clear) from C4Labs for my Pi cluster ($55), so the only network switch I could order from Amazon UK (due to space restrictions) is the [TRENDnet V1.1R](https://amzn.to/328kLis), at a cost of about $92. So my total networking and power supply cost (plus this case) is __$329__.

> Note that none of the above quoted prices include shipping. Also, the C4Labs case comes with cooling fans which are meant to draw their power from the GPIO 5V pins on the Pis. As these will be occupied by the PoE HAT, I bought [some USB jacks](https://www.amazon.co.uk/gp/product/B07R1X6B59/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1) and [a soldering iron](https://www.amazon.co.uk/gp/product/B07R55S2H4/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1) so I can solder the cooling fan wires to the USB jacks. The fans can then draw their power from the Pis via USB. These two purchases added about another __$32__ to the cluster cost.

### Disk Space

Raspberry Pis don't come with on-board storage space. They do come with microSD slots, though, so microSD cards can be used to hold the OS and data for the Pi. High-rated micro SD cards tend to cost [around $17.50 for 128GB models](https://amzn.to/2Xrcqma), though most of these limit read speed to about 80MB/s. As of this writing, the cheapest 128GB SD cards with read speeds of up to 100MB/s cost [around $21 each](https://amzn.to/2FOFuxV) or __$168__ for the entire cluster.

### Cost and Overall Specs

In summary, the total cost of my 8-Pi cluster is roughly

- $280 for all eight Raspberry Pis Model B+
- $274 for all of the power supply and networking paraphernalia
- $168 for all of the microSD cards
- $55 for a nice carrying case
- $32 for USB jacks and a soldering iron

...or about $810 total. Shipping all of this stuff to Ireland cost about another ~$125, bringing the total cost of my cluster to about $935. Note that this doesn't include the cost of a keyboard, mouse, monitor, or HDMI cable, all of which I borrowed from other projects.

Selected specs of this 8-machine cluster are as follows:

- CPU: 8 x 1.4GHz 64-bit quad-core ARM Cortex-A53 Processors
- RAM: 8 x 1GB LPDDR2 SDRAM (max 1066 Mbps data rate)
- Ethernet: Gigabit Ethernet (max 300 Mbps), PoE support
- Storage: 8 x 128GB microSD storage (max 80MB/s read speed)
- Ports: 8 x HDMI ports, 8 x 4 x USB 2.0 ports

Not too bad for under $1000!

### Installing the PoE HATs

For the most part, Raspberry Pis are plug-and-play. But since we're installing a HAT, we have to do a tiny bit of hardware work. The Raspberry Pi website [gives instructions here](https://www.raspberrypi.org/products/poe-hat/) for how to install the PoE HATs on top of a Raspberry Pi -- it's pretty easy:

1. Locate the [40-pin GPIO header](https://www.raspberrypi.org/documentation/usage/gpio/) and the nearby [4-pin PoE header](https://raspberrypi.stackexchange.com/q/84752/103781) on the top of the Raspberry Pi, and the corresponding 40-pin and 4-pin slots on the underside of the PoE HAT, but __don't put anything together yet__.

    ![20190705_101601.jpg](https://thepracticaldev.s3.amazonaws.com/i/piq1a3itlki75b817etc.jpg)
    
    When the PoE HAT is flipped right-side-up, the pins on the Pi should align with the slots on the HAT:
    
    ![20190705_101641.jpg](https://thepracticaldev.s3.amazonaws.com/i/xhz1huedr1vgdywa8fjk.jpg)

2. Locate the spacers and screws that came packaged with the PoE HAT. There should be at least 4 spacers and at least 8 screws.

    ![20190705_104024.jpg](https://thepracticaldev.s3.amazonaws.com/i/pepp8oeqt6d37nikc85l.jpg)

3. Screw the spacers to the HAT from the top.

    ![20190705_105715.jpg](https://thepracticaldev.s3.amazonaws.com/i/7fkdbqytrcvqp6sgzmb1.jpg)

4. At this point, if you are only assembling a single Pi, you'll  attach the Pi to the HAT by aligning the spacers to the holes on the Pi and screwing into the spacers from the underside of the Pi, while making sure to carefully insert the Pi GPIO and PoE pins into the GPIO and PoE slots on the HAT.

    ![20190705_105809.jpg](https://thepracticaldev.s3.amazonaws.com/i/hudokf7xq9lr4iyra5bn.jpg)

    For my cluster, however, I'm assembling all of the Pis within a C4Labs Cloudlet / Cluster case, so the Pis need to be attached to the plexiglas mounts. I remove the protective paper from the plexiglas, and add the metal spacers and screws as shown in the following photograph:
    
    ![20190705_105936.jpg](https://thepracticaldev.s3.amazonaws.com/i/zwlj1d4c5kt11jd97125.jpg)

    Make sure to use the _longer_ screws, because they need to make it through the plexiglas, past the metal spacers, through the Pi circuit board, and into the PoE HAT spacers:
    
    ![20190705_110032.jpg](https://thepracticaldev.s3.amazonaws.com/i/o0nxrwsr4mwalap26max.jpg)

    Once everything is screwed together, the Pi should be securely attached to the plexiglas mount:
    
    ![20190705_110237.jpg](https://thepracticaldev.s3.amazonaws.com/i/sj55s6n3rasn3bj1rivf.jpg)

...that's it! PoE HAT successfully installed.

__Be very careful__ if you attempt to take the PoE HAT off after you've pushed it down onto the Pi pins. It is very easy to wrench those pins off of the circuit board while trying to remove the PoE HAT. If that happens, you'll have to solder the pins back onto the Pi or switch to using a microUSB power supply instead (or buy a new Pi).

[[ back to top ]](#top)

## <a name="software">Software</a>

### Operating System

There are [dozens](https://www.makeuseof.com/tag/7-operating-systems-you-can-run-with-raspberry-pi/) of [operating systems](https://www.fossmint.com/operating-systems-for-raspberry-pi/) available for the [Raspberry Pi](https://www.techrepublic.com/pictures/choosing-a-raspberry-pi-os-heres-the-definitive-list/). Some of the more notable ones include:

- [__Raspbian__](https://www.raspberrypi.org/downloads/raspbian/) -- a Debian-based OS developed and maintained by the Raspberry Pi Foundation (the Foundation's recommended OS for the system)

    ![raspbian.jpg](https://thepracticaldev.s3.amazonaws.com/i/d947kckorec5s1t9fq9c.jpg)

- [__Windows IoT Core__](https://docs.microsoft.com/en-us/windows/iot-core/windows-iot-core) -- Windows' Internet of Things (IoT) operating system, for small devices; looks and feels similar to [Windows 10](https://www.microsoft.com/en-gb/software-download/windows10)

    ![windowsiotcore.jfif](https://thepracticaldev.s3.amazonaws.com/i/u0qua1jmpx51hmcho7q0.png)

- [__Ubuntu Core__](https://ubuntu.com/core) -- A stripped-down version of one of the most popular Linux distributions, [Ubuntu](https://ubuntu.com/)

    ![ubuntucore.png](https://thepracticaldev.s3.amazonaws.com/i/ytnhi77i77nc768lvd72.png)

- [__Android__](https://www.android.com/) -- [The most popular operating system](https://en.wikipedia.org/wiki/Android_(operating_system)) in the world; installed on over 2 billion devices, mostly mobile phones

    ![Android-on-Raspberry-Pi.png](https://thepracticaldev.s3.amazonaws.com/i/amutk0z7irccc4lv5e8h.png)

- [__RaspBSD__](https://www.raspbsd.org/) -- An offshoot of the Berkeley Software Distribution (BSD) of UNIX for the Raspberry Pi; macOS is also BSD-based
- [__Chromium OS__](https://en.wikipedia.org/wiki/Chromium_OS) -- Google's browser-based operating system
- [__RetroPie__](https://retropie.org.uk/) -- an OS for turning your Raspberry Pi into a retro-gaming machine (no copyrighted games allowed!)
- [__RISC OS Pi__](https://www.raspberrypi.org/blog/risc-os-for-raspberry-pi/) -- a Pi-optimised version of [RISC OS](https://en.wikipedia.org/wiki/RISC_OS), an operating system developed specifically for Reduced Instruction Set Chips (RISC), like the AMD series used in Raspberry Pi

As fun as it would be to install RetroPie, I think I'll stick with Raspbian, as it's the OS that's recommended by The Raspberry Pi Foundation, and has been [specially optimised for the low-performance hardware](https://en.wikipedia.org/wiki/Raspbian) of the Raspberry Pi.

Raspbian's [__NOOBS__](https://www.raspberrypi.org/downloads/noobs/) (New Out Of the Box Software) installer is the easiest way to get Raspbian onto a Raspberry Pi. Just download it, and follow [the instructions](https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up) listed on Raspberry Pi's website. You'll need to [format the microSD card as FAT (__not ExFAT__!)](https://www.raspberrypi.org/documentation/installation/sdxc_formatting.md), extract the NOOBS `*.zip` archive, and copy the files within it to your newly-formatted microSD card. Then, insert the microSD card into the Pi; there's a slot on the underside of the Pi, on the side opposite the USB ports:

![20190705_110156.jpg](https://thepracticaldev.s3.amazonaws.com/i/xbatddkk0lfovv0rgdmv.jpg)

![20190705_110209.jpg](https://thepracticaldev.s3.amazonaws.com/i/sjbnkdo7ax51chca5okr.jpg)

Power the Pi by plugging an Ethernet cable into it, with the other end of the cable in the PoE-enabled network switch. Plug in the network switch and turn it on. After a second or two, the Pi should power up: the yellow light next to the Ethernet port, and the red light near the microSD card should both light up.

![20190705_122348.jpg](https://thepracticaldev.s3.amazonaws.com/i/do7okzp99nl2zbrvgj2n.jpg)

Then, connect the Pi to a monitor using the HDMI port on the side opposite the GPIO header.

![20190705_134246b.jpg](https://thepracticaldev.s3.amazonaws.com/i/biytxqqmwzwzmcy1j51h.jpg)

Connect a mouse and a keyboard to the Pi via USB, and follow the on-screen instructions in the OS install wizard (it should look similar to the screenshot below).

<a href="https://www.raspberrypi.org/documentation/installation/noobs.md" target="_blank"><figure><img src="https://thepracticaldev.s3.amazonaws.com/i/j5n1dhrt9wu1kvhzbr82.png" alt="Image showing NOOBS installer running on a Raspberry Pi."/><figcaption>NOOBS Installer on a Raspberry Pi</figcaption></figure></a>

Once you've successfully installed your OS of choice on a single Pi, you can simply [clone the microSD card](https://beebom.com/how-clone-raspberry-pi-sd-card-windows-linux-macos/) to install the same OS on the other Pis. There's no need to go through the install wizard multiple times. Later on in this tutorial, I'll explain how to easily run a particular command (including installing software with `apt`) simultaneously across all Pis on the cluster, so you don't need to repeat the same manual tasks over and over like a monkey at a typewriter -- this is why we have technology, isn't it?

### <a name="networking">Networking</a>

#### Configuring Static IP Addresses

To facilitate easy networking of the Pis, I'm going to set static IP addresses for each Pi on the network switch. I'll number the Pis 1-8 (inclusive) according to their positions on the network switch and in the carrying case. When looking at the ports on the Pis (the "front" of the case), #1 is the rightmost Pi and #8 is the leftmost Pi.

To enable user-defined, static IP addresses, I edit the file `/etc/dhcpcd.conf` on each Pi and uncomment / edit the lines:

```
interface eth0
static ip_address=192.168.0.10X/24
```

...where `X` should be replaced by `1` for Pi #1, `2` for Pi #2, etc. After this change has been made on a particular Pi, I reboot the machine. Once this is done for all eight Pis, they should all be able to `ping` each other at those addresses.

I've also installed `nmap` on Pi #1 so that the status of all eight Pis can be easily checked:

```bash
$ sudo nmap –sP 192.168.0.0/24
```

...will show the status of all seven other Pis, not including the one on which the command is run. The "`N hosts up`" shows how many Pis are properly configured according to the above, currently connected, and powered on.

#### Enabling `ssh`

To enable `ssh` on each Pi, we need to follow [these instructions](https://www.raspberrypi.org/documentation/remote-access/ssh/) (reproduced here to avoid link rot):

> As of the November 2016 release, Raspbian has the SSH server disabled by default. It can be enabled manually from the desktop:
>
> 1. Launch `Raspberry Pi Configuration` from the `Preferences` menu
> 2. Navigate to the `Interfaces` tab
> 3. Select `Enabled` next to `SSH`
> 4. Click `OK`
>
> Alternatively, [`raspi-config`](https://www.raspberrypi.org/documentation/configuration/raspi-config.md) can be used in the terminal:
>
> 1. Enter `sudo raspi-config` in a terminal window
> 2. Select `Interfacing Options`
> 3. Navigate to and select `SSH`
> 4. Choose `Yes`
> 5. Select `Ok`
> 6. Choose `Finish`
>
> Alternatively, use systemctl to start the service
>
> ```bash
>  $ sudo systemctl enable ssh
>  $ sudo systemctl start ssh
> ```
>
> When enabling SSH on a Pi that may be connected to the Internet, you should change its default password to ensure that it remains secure. See the [Security page](https://www.raspberrypi.org/documentation/configuration/security.md) for more details.

#### Hostnames

Initially, all of the Pis are known as `raspberrypi`, and have a single user, `pi`:

```bash
$ hostname
raspberrypi

$ whoami
pi
```

This has the potential to become very confusing if we're constantly moving back and forth between the different Pis on the network. To simplify this, we're going to assign each Pi a hostname based on its position in the case / on the network switch. Pi #1 will be known as `pi1`, Pi #2 as `pi2`, and so on.

To accomplish this, two files must be edited: `/etc/hosts` and `/etc/hostname`. Within those files, there should be only one occurrence each of `raspberrypi`, which is the default hostname. We change each of those to `piX` where `X` is the appropriate number 1-8. Finally, in `/etc/hosts` only, we also add the IPs for all the other Pis at the end of the file, like:

```
192.168.0.101 pi1
192.168.0.102 pi2
...
192.168.0.108 pi8
```

This must be done manually on each Pi.

Once the above tasks are finished for the appropriate Pi, we reboot that particular Pi. Now, when the terminal is opened, instead of:

```bash
pi@raspberrypi:~ $
```

...you should see

```bash
pi@piX:~ $
```

...where `X` is the index of that Pi on the cluster. We do this on each Pi and reboot each of them after it's been done. From now on in these instructions, the command prompt will be abbreviated to simply `$`. Any other code is output or the text of a file.

#### Simplifying `ssh`

To connect from one Pi to another, having followed only the above instructions, would require the following series of commands:

```bash
$ ssh pi@192.168.0.10X
pi@192.168.0.10X's password: <enter password – 'raspberry' default>
```

Granted, this is not too much typing, but if we have to do it very often, it could become cumbersome. To avoid this, we can set up `ssh` aliases and passwordless `ssh` connections with public/private key pairs.

_ssh aliases_

To set up an `ssh` alias, we edit the `~/.ssh/config` file on a particular Pi and add the following lines:

```
Host piX
User pi
Hostname 192.168.0.10X
```

...replacing `X` with 1-8 for each of the eight Pis. Note that this is done on a single Pi, so that one Pi should have eight chunks of code within `~/.ssh/config`, which look identical to the above except for the `X` character, which should change for each Pi on the network. Then, the `ssh` command sequence becomes just: 

```bash
$ ssh piX
pi@192.168.0.10X's password: <enter password>
```

This can be simplified further by setting up public/private key pairs.

_public/private key pairs_

On each Pi, run the following command:

```bash
$ ssh-keygen –t ed25519
```

This will generate a public / private key pair within the directory `~/.ssh/` which can be used to securely `ssh` without entering a password. One of these files will be called `id_ed25519`, this is the _private key_. The other, `id_ed25519.pub` is the _public key_. No passphrase is necessary to protect access to the key pair. The public key is used to communicate with the other Pis, and the private key never leaves its host machine and should never be moved or copied to any other device.

Each public key will need to be concatenated to the `~/.ssh/authorized_keys` file on every other Pi. It's easiest to do this once, for a single Pi, then simply copy the `authorized_keys` file to the other Pis. Let's assume that Pi #1 will contain the "master" record, which is then copied to the other Pis.

On Pi #2 (and #3, #4, etc.), run the following command:

```bash
$ cat ~/.ssh/id_ed25519.pub | ssh pi@192.168.0.101 'cat >> .ssh/authorized_keys'
```

This concatenates Pi #2's public key file to Pi #1's list of authorized keys, giving Pi #2 permission to `ssh` into Pi #1 without a password (the public and private keys are instead used to validate the connection). We need to do this for each machine, concatenating each public key file to Pi #1's list of authorized keys. We should also do this for Pi #1, so that when we copy the completed `authorized_keys` file to the other Pis, they all have permission to `ssh` into Pi #1, as well. Run the following command on Pi #1:

```bash
$ cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
```

Once this is done, as well as the previous section, `ssh`-ing is as easy as:

```bash
$ ssh pi1
```

...and that's it! Additional aliases can be configured in the `~/.bashrc` file to shorten this further (see below) though this is not configured on our system:

```bash
alias p1="ssh pi1" # etc.
```

_replicate the configuration_

Finally, to replicate the passwordless `ssh` across all Pis, simply copy the two files mentioned above from Pi #1 to each other Pi using `scp`:

```bash
$ scp ~/.ssh/authorized_keys piX:~/.ssh/authorized_keys
$ scp ~/.ssh/config piX:~/.ssh/config 
```

You should now be able to `ssh` into any Pi on the cluster from any other Pi with just `ssh piX`.

#### Ease of Use

Finally, a few miscellaneous ease-of-use-type enhancements for the cluster.

_get the hostname of every Pi except this one_

To get the hostname of a Pi, you can just use:

```bash
$ hostname
pi1
```

...to get the hostname of all _other_ Pis on the cluster, define the function:

```bash
function otherpis {
  grep "pi" /etc/hosts | awk '{print $2}' | grep -v $(hostname)
}
```

> The `-v` flag tells `grep` to _invert_ the selection. Only lines which _don't match_ the `hostname` are returned.

...in your `~/.bashrc`, then, `source` your `~/.bashrc` file with:

```bash
$ source ~/.bashrc
```

(Note that, whenever you edit `~/.bashrc`, for those changes to take effect, you must `source` the file or log out and log back in.) Then, you can call the new function on the command line with:

```bash
$ otherpis
pi2
pi3
pi4
...
```

Note that this function relies on you having listed all of the IPs and hostnames of the Pis within the `/etc/hosts` file.

_send the same command to all Pis_

To send the same command to each Pi, add the following function to the `~/.bashrc` of the Pi from which you want to dictate commands (I add this to Pi #1's `~/.bashrc` and then copy the `~/.bashrc` file to all other Pis using the instructions below):

```bash
function clustercmd {
  for pi in $(otherpis); do ssh $pi "$@"; done
  $@
}
```

This will run the given command on each other Pi, and then on this Pi:

```bash
$ clustercmd date
Tue Apr  9 00:32:41 IST 2019
Tue Apr  9 05:58:07 IST 2019
Tue Apr  9 06:23:51 IST 2019
Tue Apr  9 23:51:00 IST 2019
Tue Apr  9 05:58:57 IST 2019
Tue Apr  9 07:36:13 IST 2019
Mon Apr  8 15:19:32 IST 2019
Wed Apr 10 03:48:11 IST 2019
```

...we can see above that all of our Pis have different system times. Let's fix that.

_synchronise time across cluster_

Simply tell each Pi to install the package `htpdate` and the date will be updated. That's it:

```bash
$ clustercmd "sudo apt install htpdate -y > /dev/null 2>&1"

$ clustercmd date
Wed Jun 19 16:04:22 IST 2019 
Wed Jun 19 16:04:18 IST 2019 
Wed Jun 19 16:04:19 IST 2019
Wed Jun 19 16:04:20 IST 2019 
Wed Jun 19 16:04:48 IST 2019 
Wed Jun 19 16:04:12 IST 2019 
Wed Jun 19 16:04:49 IST 2019
Wed Jun 19 16:04:25 IST 2019
```

Notice how all the dates and times are accurate to within a minute now. If we reboot all of the Pis, they'll be even more closely aligned:

```bash
$ clustercmd date
Wed Jun 19 16:09:28 IST 2019 
Wed Jun 19 16:09:27 IST 2019 
Wed Jun 19 16:09:29 IST 2019
Wed Jun 19 16:09:31 IST 2019 
Wed Jun 19 16:09:28 IST 2019 
Wed Jun 19 16:09:30 IST 2019 
Wed Jun 19 16:09:24 IST 2019
Wed Jun 19 16:09:28 IST 2019
```

They're now aligned to within 10 seconds. Note that it takes a few seconds to run the command across the cluster. To precisely synchronise all clocks to a remote server (a respected standard like `time.nist.gov` or `google.com`) you can do:

```bash
$ clustercmd sudo htpdate -a -l time.nist.gov
```

...this will take a few minutes, because the program slowly adjusts the clock in intervals of < 30ms so there are no "jumps" in any system file timestamps. Once this command finishes, the clocks will be synchronised (remember, it takes a second or two to communicate across the network, so they're still "off" by two or three seconds):

```bash
$ clustercmd date
Wed Jun 19 16:36:46 IST 2019 
Wed Jun 19 16:36:47 IST 2019 
Wed Jun 19 16:36:47 IST 2019
Wed Jun 19 16:36:48 IST 2019 
Wed Jun 19 16:36:49 IST 2019 
Wed Jun 19 16:36:49 IST 2019 
Wed Jun 19 16:36:49 IST 2019
Wed Jun 19 16:36:49 IST 2019
```

_reboot the cluster_

I add the following function to my `~/.bashrc` on Pi #1:

```bash
function clusterreboot {
  clustercmd sudo shutdown -r now
}
```

This function makes it easy to reboot the entire cluster. Another one lets me shut down the entire cluster without rebooting:

```bash
function clustershutdown {
  clustercmd sudo shutdown now
}
```

_send the same file to all Pis_

To copy a file from one Pi to all others, let's add a function called `clusterscp` (cluster secure copy) to the `~/.bashrc` file of any particular Pi:

```bash
function clusterscp {
  for pi in $(otherpis); do
    cat $1 | ssh $pi "sudo tee $1" > /dev/null 2>&1
  done
}
```

Then we can copy all of the ease-of-use functions we've defined in `~/.bashrc` to every other Pi on the cluster:

```bash
$ source ~/.bashrc && clusterscp ~/.bashrc
```

### Securing the Cluster

You might only be building a Hadoop cluster for fun, but I'm building one to do some data analysis for work. Since my cluster is going to hold proprietary data, cluster security is paramount.

[There is a lot we can do](https://medium.com/@jasonrigden/hardening-ssh-1bcb99cd4cef) to secure a computer against unauthorised access. The easiest (and most effective) thing that can be done (after setting up passwordless `ssh` following the instructions in the above section "Simplifying `ssh`") is to disable password-based authentication. This means that users can only log into the system if they have a public key corresponding to one of the private keys on one of the Pis. While it is possible (in theory) to crack these keys, [ed25519 is the most secure public/private key pair algorithm to date](https://security.stackexchange.com/a/90083/183193) and will likely not be breakable within the foreseeable future.

#### Disable Password-Based Authentication & `root` Login

To disable password-based authentication, we edit the file `/etc/ssh/sshd_config`. There are 6 options we want to change in particular. Search for the following keys in this file and change them so that they match the examples below. Make sure that none of the below lines begins with a '`#`', this comments out the line so that it is ignored by the system.

```
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no
X11Forwarding no
PrintMotd no
```

The above commands will, respectively: prevent hackers from trying to login as `root` (so they'll need to know that the username is `pi`); [disallow password-based authentication](https://superuser.com/a/374234/728488) (so they'll need the corresponding ed25519 public key to the particular private key on a given Pi); disable X11 (GUI) forwarding among / from / to the Pis (everything must be done command-line); and disable the "message of the day" (MOTD), which can sometimes include compromising information. Edit `/etc/ssh/sshd_config` and make the above changes on a particular Pi, then copy those changes to all other Pis with...

__WARNING: BE EXTREMELY CAREFUL WITH THE FOLLOWING STEPS.__ If you make a typo in `sshd_config`, then restart the `ssh` service on a Pi, it will throw an error and you will not be able to `ssh` into that Pi any more. You will need to manually (connect the HDMI cable to the video output) connect to a Pi to reconfigure!

```bash
$ clusterscp /etc/ssh/sshd_config
```

...and restart the `ssh` service (reload the configuration) across all Pis with:

```bash
$ clustercmd sudo service ssh restart
```

To remove the MOTD entirely, delete the file `/etc/motd`:

```bash
$ clustercmd sudo rm /etc/motd
```

#### Monitor `ssh` Activity with `fail2ban`

Another big thing we can do to monitor and protect the integrity of the cluster is install a program called `fail2ban`. `fail2ban` "logs and temporarily ban[s] IPs based on possible malicious activity". To install this program on every machine in the cluster, run the command:

```bash
$ clustercmd sudo apt install fail2ban –y
```

Then, copy the configuration file on a single Pi:

```bash
$ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
```

...and edit the lines within `jail.local` near the spot that starts with `[sshd]`. Note that `[sshd]` appears near the top of the file in a commented block – ignore that. `[sshd]` should be configured as follows:

```
[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s
```

Once you've finished editing `jail.local`, copy it to all other Pis on the cluster with:

```bash
$ clusterscp /etc/fail2ban/jail.local
```

...and restart the `fail2ban` service on all Pis with:

```bash
$ clustercmd sudo service fail2ban restart
```

[[ back to top ]](#top)

## <a name="hadoopspark">Hadoop & Spark</a>

### Single-Node Setup

#### Hadoop

Apache Hadoop v2.7+ [requires Java 7+](https://wiki.apache.org/hadoop/HadoopJavaVersions) to run. When I installed Raspbian on my Pis, I used [NOOBS](https://www.raspberrypi.org/downloads/noobs/) v3.0.1 (2019-04-08). This installs Raspbian and also Java 1.8.0_65 (Java 8 by HotSpot aka. Oracle). So all Pis have an acceptable version of Java installed by default. Next, we can download and install Hadoop.

I'll start by building a single-node setup on the master node (Pi #1), then I'll the worker nodes to create a multi-node cluster. On Pi #1, get Hadoop with the commands:

```bash
$ cd && wget https://bit.ly/2wa3Hty
```

(This is a shortened link to `hadoop-3.2.0.tar.gz`)

```bash
$ sudo tar -xvf 2wa3Hty -C /opt/
$ rm 2wa3Hty && cd /opt
$ sudo mv hadoop-3.2.0 hadoop
```

Then, make sure to change the permissions on this directory:

```bash
$ sudo chown pi:pi -R /opt/hadoop
```

Finally, add this directory to the `$PATH` by editing `~/.bashrc` and putting the following lines at the end of the file:

```bash
export JAVA_HOME=$(readlink –f /usr/bin/java | sed "s:bin/java::")
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
```

...and edit `/opt/hadoop/etc/hadoop/hadoop-env.sh` to add the following line: 

```bash
export JAVA_HOME=$(readlink –f /usr/bin/java | sed "s:bin/java::") 
```

You can verify that Hadoop has been installed correctly by checking the version:

```bash
$ cd && hadoop version | grep Hadoop
Hadoop 3.2.0
```

#### Spark

We will download Spark in a similar manner to how we downloaded Hadoop, above:

```bash
$ cd && wget https://bit.ly/2HK6nTW
```

(This is a shortened link to `spark-2.4.3-bin-hadoop2.7.tgz`)

```bash
$ sudo tar –xvf 2HK6nTW –C /opt/
$ rm 2HK6nTW && cd /opt
$ sudo mv spark-2.4.3-bin-hadoop2.7 spark
```

Then, make sure to change the permissions on this directory:

```bash
$ sudo chown pi:pi -R /opt/spark
```

Finally, add this directory to your `$PATH` by editing `~/.bashrc` and putting the following lines at the end of the file:

```bash
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
```

You can verify that Spark has been installed correctly by checking the version:

```bash
$ cd && spark-shell --version
... version 2.4.3 ... Using Scala version 2.11.12 ...
```

#### HDFS

To get the Hadoop Distributed File System (HDFS) up and running, we need to modify some configuration files. All of these files are within `/opt/hadoop/etc/hadoop`. The first is __`core-site.xml`__. Edit it so it looks like the following:

```xml
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://pi1:9000</value>
  </property>
</configuration>
```

The next is __`hdfs-site.xml`__, which should look like: 

```xml
<configuration>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///opt/hadoop_tmp/hdfs/datanode</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///opt/hadoop_tmp/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration> 
```

...this configures where the DataNode and NameNode information is stored and also sets the replication (the number of times a block is copied across the cluster) to 1 (we will change this later). Make sure that you also create these directories with the commands:

```bash
$ sudo mkdir -p /opt/hadoop_tmp/hdfs/datanode
$ sudo mkdir -p /opt/hadoop_tmp/hdfs/namenode
```

...and adjust the owner of these directories:

```bash
$ sudo chown pi:pi -R /opt/hadoop_tmp
```

The next file is __`mapred-site.xml`__, which should look like: 

```xml
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
```

...and finally __`yarn-site.xml`__, which should look like: 

```xml
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration> 
```

Once these four files are edited, we can format the HDFS (__WARNING: DO NOT DO THIS IF YOU ALREADY HAVE DATA IN THE HDFS! IT WILL BE LOST!__):

```bash
$ hdfs namenode -format -force
```

...we then boot the HDFS with the following two commands:

```bash
$ start-dfs.sh
$ start-yarn.sh
```

...and test that it's working by creating a temporary directory:

```bash
$ hadoop fs -mkdir /tmp

$ hadoop fs -ls /
Found 1 items
drwzr-xr-x   - pi supergroup          0 2019-04-09 16:51 /tmp
```

...or by running the command `jps`:

```bash
$ jps
2736 NameNode
2850 DataNode
3430 NodeManager
3318 ResourceManager
3020 SecondaryNameNode
3935 Jps
```

This shows that the HDFS is up and running, at least on Pi #1. To check that Spark and Hadoop are working together, we can do:

```bash
$ hadoop fs -put $SPARK_HOME/README.md /

$ spark-shell
```

...this will open the Spark shell, with prompt `scala>`:

```scala
scala> val textFile = sc.textFile("hdfs://pi1:9000/README.md")
...

scala> textFile.first()
res0: String = # Apache Spark
```

#### Hide `execstack` Warning

While running the Hadoop commands above, you may get a warning like...

> "You have loaded library... which might have disabled stack guard."

...the reason this happens on our Raspberry Pis is because of a mismatch between the 32-bit runtime the Hadoop binaries were built for, and the 64-bit Raspbian version we're running. To ignore these warnings, change the line

```bash
# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"
```

...in `/opt/hadoop/etc/hadoop/hadoop-env.sh` to

```bash
export HADOOP_OPTS="-XX:-PrintWarnings –Djava.net.preferIPv4Stack=true"
```

This will hide the warnings in the future. (Alternatively, you could also download the Hadoop source code and build from scratch.)

#### Hide `NativeCodeLoader` Warning

Another warning you may see is that `util.NativeCodeLoader` is "`Unable to load [the] native-hadoop library for your platform`". This warning cannot be resolved easily. It's due to the fact that Hadoop is compiled for 32-bit architectures and the version of Raspbian we have is 64-bit. We could recompile the library from scratch on the 64-bit machine, but I'd rather just not see the warning. To do this, we can add the following lines to the bottom of our `~/.bashrc` file:

```bash
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA" 
```

This will prevent those `NativeCodeLoader` warnings from being printed.

### Cluster Setup

At this point, you should have a single-node cluster and that single node acts as both a master and a worker node. To set up the worker nodes (and distribute computing to the entire cluster), we must take the following steps...

#### Create the Directories

Create the required directories on all other Pis using: 

```bash
$ clustercmd sudo mkdir -p /opt/hadoop_tmp/hdfs
$ clustercmd sudo chown pi:pi –R /opt/hadoop_tmp
$ clustercmd sudo mkdir -p /opt/hadoop
$ clustercmd sudo chown pi:pi /opt/hadoop
```

#### Copy the Configuration

Copy the files in /opt/hadoop to each other Pi using:

```bash
$ for pi in $(otherpis); do rsync –avxP $HADOOP_HOME $pi:/opt; done
```

This will take quite a long time, so go grab lunch.

When you're back, verify that the files copied correctly by querying the Hadoop version on each node with the following command:

```bash
$ clustercmd hadoop version | grep Hadoop
Hadoop 3.2.0
Hadoop 3.2.0
Hadoop 3.2.0
...
```

#### Configuring Hadoop on the Cluster

It's difficult to find a good guide for installing Hadoop across a networked cluster of machines. [This link](https://www.linode.com/docs/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/) points to the one I followed, mostly without incident. To get HDFS running across the cluster, we need to modify the configuration files that we edited earlier. All of these files are within `/opt/hadoop/etc/hadoop`. The first is __`core-site.xml`__. Edit it so it looks like the following:

```xml
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://pi1:9000</value>
  </property>
</configuration>
```

The next is __`hdfs-site.xml`__, which should look like: 

```xml
<configuration>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/opt/hadoop_tmp/hdfs/datanode</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/opt/hadoop_tmp/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>4</value>
  </property>
</configuration> 
```

The next file is __`mapred-site.xml`__, which should look like: 

```xml
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>256</value>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>128</value>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>128</value>
  </property>
</configuration> 
```

...and finally __`yarn-site.xml`__, which should look like: 

```xml
<configuration>
  <property>
    <name>yarn.acl.enable</name>
    <value>0</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>pi1</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>900</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>900</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>64</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
</configuration>
```

Make these changes to these files, then remove all old files from all Pis. I'm assuming you're working your way through this tutorial step-by-step and haven't yet added any big, important data to the single-node Pi #1 cluster. You can clean up all the Pis with:

```bash
$ clustercmd rm –rf /opt/hadoop_tmp/hdfs/datanode/*
$ clustercmd rm –rf /opt/hadoop_tmp/hdfs/namenode/*
```

__If you don't do this__, you may have errors with the Pis not recognizing each other. If you can't get the DataNodes (Pi #2-#8) to start, or communicate with the NameNode (Pi #1), cleaning the files above might be the solution (it was for me).

Next, we need to create two files in `$HADOOP_HOME/etc/hadoop/` which tell Hadoop which Pis to use as worker nodes and which Pi should be the master (NameNode) node. Create a file named `master` in the aforementioned directory and add only a single line:

```
pi1
```

Then, create a file named `workers` (__NOT `slaves`__, as was the case in previous versions of Hadoop) in the same directory and add all of the other Pis:

```
pi2
pi3
pi4
...
```

Then, you'll need to edit `/etc/hosts` again. On any Pi, remove the line which looks like

```
127.0.0.1 PiX
```

...where `X` is the index of that particular Pi. Then, copy this file to all other Pis with:

```bash
$ clusterscp /etc/hosts
```

We can do this now because this file is no longer Pi-specific. Finally, reboot the cluster for these changes to take effect. When all Pis have rebooted, on Pi #1, run the command:

__WARNING: DO NOT DO THIS IF YOU ALREADY HAVE DATA IN THE HDFS! IT WILL BE LOST!__

```bash
$ hdfs namenode -format -force
```

...we then boot the HDFS with the following two commands:

```bash
$ start-dfs.sh && start-yarn.sh
```

We can test the cluster by putting files in HDFS from any Pi (using `hadoop fs -put`) and making sure they show up on other Pis (using `hadoop fs -ls`). You can also check that the cluster is up and running by opening a web browser and navigating to http://pi1:9870. This web interface gives you a file explorer as well as information about the health of the cluster.

<figure><img src="https://thepracticaldev.s3.amazonaws.com/i/5rrkjy0stc0xkzlk2upw.png" alt="Screenshot of the Hadoop web UI running in the Chromium browser on port 9870 of Pi #1."/><figcaption>Hadoop web UI running on port 9870.</figcaption></figure>

<figure><img src="https://thepracticaldev.s3.amazonaws.com/i/hf1pupsq87cdq4gtntpc.png" alt="Hadoop web UI showing DataNode statistics."/><figcaption>Hadoop web UI showing DataNode statistics.</figcaption></figure>

<figure><img src="https://thepracticaldev.s3.amazonaws.com/i/bp7e3bn5pbq8q3oqnp4q.png" alt="Hadoop web UI showing file browser."/><figcaption>File browser in Hadoop web UI.</figcaption></figure>

#### Configuring Spark on the Cluster

Spark will run fine on a single machine, so we may trick ourselves into thinking we're using the full power of the Hadoop cluster when in reality we're not. Some of the configuration we performed above was for Hadoop YARN (Yet Another Resource Negotiator). This is a "resource negotiator" for HDFS, which orchestrates how files are moved and analysed around the cluster. For Spark to be able to communicate with YARN, we need to configure two more environment variables in Pi #1's `~/.bashrc`. Previously, we defined

```bash
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
```

...in `~/.bashrc`. Just beneath this, we will now add two more environment variables:

```bash
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
```

`$HADOOP_CONF_DIR` is the directory which contains all of the `*-site.xml` configuration files that we edited above. Next, we create the Spark configuration file:

```bash
$ cd $SPARK_HOME/conf
$ sudo mv spark-defaults.conf.template spark-defaults.conf
```

...and we add the following lines to the end of this file:

```
spark.master            yarn
spark.driver.memory     465m
spark.yarn.am.memory    356m
spark.executor.memory   465m
spark.executor.cores    4
```

The meaning of these values is explained at [this link](https://www.linode.com/docs/databases/hadoop/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/). But note that the above is __very machine-specific__. The configuration above works fine for me with the Raspberry Pi 3 Model B+, but it may not work for you (or be optimal) for a less- or more-powerful machine. Spark enforces hard lower bounds on how much memory can be allocated, as well. Where you see `465m` above, that's the minimum configurable value for that entry -- any less and Spark will refuse to run.

> A Raspberry Pi 3 Model B+ uses between 9-25\% of its RAM while idling. Since they have 926MB RAM in total, Hadoop and Spark will have access to at most about 840MB of RAM per Pi.

Once all of this has been configured, reboot the cluster. Note that, when you reboot, you should NOT format the HDFS NameNode again. Instead, simply stop and restart the HDFS service with:

```bash
$ stop-dfs.sh && stop-yarn.sh

$ start-dfs.sh && start-yarn.sh
```

Now, you can submit a job to Spark on the command line:

```bash
pi@pi1:~ $ spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_2.11-2.4.3.jar 7
OpenJDK Client VM warning: You have loaded library /opt/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2019-07-08 14:01:24,408 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-07-08 14:01:25,514 INFO spark.SparkContext: Running Spark version 2.4.3
2019-07-08 14:01:25,684 INFO spark.SparkContext: Submitted application: Spark Pi
2019-07-08 14:01:25,980 INFO spark.SecurityManager: Changing view acls to: pi
2019-07-08 14:01:25,980 INFO spark.SecurityManager: Changing modify acls to: pi
2019-07-08 14:01:25,981 INFO spark.SecurityManager: Changing view acls groups to: 
2019-07-08 14:01:25,981 INFO spark.SecurityManager: Changing modify acls groups to: 
2019-07-08 14:01:25,982 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(pi); groups with view permissions: Set(); users  with modify permissions: Set(pi); groups with modify permissions: Set()
2019-07-08 14:01:27,360 INFO util.Utils: Successfully started service 'sparkDriver' on port 46027.
2019-07-08 14:01:27,491 INFO spark.SparkEnv: Registering MapOutputTracker
2019-07-08 14:01:27,583 INFO spark.SparkEnv: Registering BlockManagerMaster
2019-07-08 14:01:27,594 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2019-07-08 14:01:27,596 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
2019-07-08 14:01:27,644 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-e5479834-d1e4-48fa-9f5c-cbeb65531c31
2019-07-08 14:01:27,763 INFO memory.MemoryStore: MemoryStore started with capacity 90.3 MB
2019-07-08 14:01:28,062 INFO spark.SparkEnv: Registering OutputCommitCoordinator
2019-07-08 14:01:28,556 INFO util.log: Logging initialized @10419ms
2019-07-08 14:01:28,830 INFO server.Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-07-08 14:01:28,903 INFO server.Server: Started @10770ms
2019-07-08 14:01:28,997 INFO server.AbstractConnector: Started ServerConnector@89f072{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-07-08 14:01:28,997 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
2019-07-08 14:01:29,135 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1b325b3{/jobs,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,137 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@b72664{/jobs/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,140 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@34b7b8{/jobs/job,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,144 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1e8821f{/jobs/job/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,147 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@b31700{/stages,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,150 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@165e559{/stages/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,153 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1ae47a0{/stages/stage,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,158 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a54d{/stages/stage/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,161 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1ef722a{/stages/pool,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,165 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1d9b663{/stages/pool/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,168 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14894fc{/storage,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,179 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@567255{/storage/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,186 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@362c57{/storage/rdd,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,191 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4ee95c{/storage/rdd/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,195 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1c4715d{/environment,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,200 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@a360ea{/environment/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,204 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@148bb7d{/executors,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,209 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@27ba81{/executors/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,214 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@336c81{/executors/threadDump,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,217 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@156f2dd{/executors/threadDump/json,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,260 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1e52059{/static,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,265 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@159e366{/,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,283 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1dc9128{/api,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,288 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@c4944a{/jobs/job/kill,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,292 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@772895{/stages/stage/kill,null,AVAILABLE,@Spark}
2019-07-08 14:01:29,304 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://pi1:4040
2019-07-08 14:01:29,452 INFO spark.SparkContext: Added JAR file:/opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar at spark://pi1:46027/jars/spark-examples_2.11-2.4.3.jar with timestamp 1562590889451
2019-07-08 14:01:33,070 INFO client.RMProxy: Connecting to ResourceManager at pi1/192.168.0.101:8032
2019-07-08 14:01:33,840 INFO yarn.Client: Requesting a new application from cluster with 7 NodeManagers
2019-07-08 14:01:34,082 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (900 MB per container)
2019-07-08 14:01:34,086 INFO yarn.Client: Will allocate AM container, with 740 MB memory including 384 MB overhead
2019-07-08 14:01:34,089 INFO yarn.Client: Setting up container launch context for our AM
2019-07-08 14:01:34,101 INFO yarn.Client: Setting up the launch environment for our AM container
2019-07-08 14:01:34,164 INFO yarn.Client: Preparing resources for our AM container
2019-07-08 14:01:35,577 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2019-07-08 14:02:51,027 INFO yarn.Client: Uploading resource file:/tmp/spark-ca0b9022-2ba9-45ff-8d63-50545ef98e55/__spark_libs__7928629488171799934.zip -> hdfs://pi1:9000/user/pi/.sparkStaging/application_1562589758436_0002/__spark_libs__7928629488171799934.zip
2019-07-08 14:04:09,654 INFO yarn.Client: Uploading resource file:/tmp/spark-ca0b9022-2ba9-45ff-8d63-50545ef98e55/__spark_conf__4579290782490197871.zip -> hdfs://pi1:9000/user/pi/.sparkStaging/application_1562589758436_0002/__spark_conf__.zip
2019-07-08 14:04:13,226 INFO spark.SecurityManager: Changing view acls to: pi
2019-07-08 14:04:13,227 INFO spark.SecurityManager: Changing modify acls to: pi
2019-07-08 14:04:13,227 INFO spark.SecurityManager: Changing view acls groups to: 
2019-07-08 14:04:13,228 INFO spark.SecurityManager: Changing modify acls groups to: 
2019-07-08 14:04:13,228 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(pi); groups with view permissions: Set(); users  with modify permissions: Set(pi); groups with modify permissions: Set()
2019-07-08 14:04:20,235 INFO yarn.Client: Submitting application application_1562589758436_0002 to ResourceManager
2019-07-08 14:04:20,558 INFO impl.YarnClientImpl: Submitted application application_1562589758436_0002
2019-07-08 14:04:20,577 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1562589758436_0002 and attemptId None
2019-07-08 14:04:21,625 INFO yarn.Client: Application report for application_1562589758436_0002 (state: ACCEPTED)
2019-07-08 14:04:21,680 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: [Mon Jul 08 14:04:20 +0100 2019] Scheduler has assigned a container for AM, waiting for AM container to be launched
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1562591060331
	 final status: UNDEFINED
	 tracking URL: http://pi1:8088/proxy/application_1562589758436_0002/
	 user: pi
2019-07-08 14:04:22,696 INFO yarn.Client: Application report for application_1562589758436_0002 (state: ACCEPTED)
2019-07-08 14:04:23,711 INFO yarn.Client: Application report for application_1562589758436_0002 (state: ACCEPTED)
2019-07-08 14:04:24,725 INFO yarn.Client: Application report for application_1562589758436_0002 (state: ACCEPTED)
...
2019-07-08 14:05:45,863 INFO yarn.Client: Application report for application_1562589758436_0002 (state: ACCEPTED)
2019-07-08 14:05:46,875 INFO yarn.Client: Application report for application_1562589758436_0002 (state: ACCEPTED)
2019-07-08 14:05:47,883 INFO yarn.Client: Application report for application_1562589758436_0002 (state: RUNNING)
2019-07-08 14:05:47,884 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.0.103
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time: 1562591060331
	 final status: UNDEFINED
	 tracking URL: http://pi1:8088/proxy/application_1562589758436_0002/
	 user: pi
2019-07-08 14:05:47,891 INFO cluster.YarnClientSchedulerBackend: Application application_1562589758436_0002 has started running.
2019-07-08 14:05:47,937 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46437.
2019-07-08 14:05:47,941 INFO netty.NettyBlockTransferService: Server created on pi1:46437
2019-07-08 14:05:47,955 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-07-08 14:05:48,178 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, pi1, 46437, None)
2019-07-08 14:05:48,214 INFO storage.BlockManagerMasterEndpoint: Registering block manager pi1:46437 with 90.3 MB RAM, BlockManagerId(driver, pi1, 46437, None)
2019-07-08 14:05:48,265 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, pi1, 46437, None)
2019-07-08 14:05:48,269 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, pi1, 46437, None)
2019-07-08 14:05:49,426 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> pi1, PROXY_URI_BASES -> http://pi1:8088/proxy/application_1562589758436_0002), /proxy/application_1562589758436_0002
2019-07-08 14:05:49,441 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill.
2019-07-08 14:05:49,816 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
2019-07-08 14:05:49,829 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@136bd1{/metrics/json,null,AVAILABLE,@Spark}
2019-07-08 14:05:49,935 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
2019-07-08 14:05:50,076 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
2019-07-08 14:05:52,074 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:38
2019-07-08 14:05:52,479 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 7 output partitions
2019-07-08 14:05:52,481 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2019-07-08 14:05:52,485 INFO scheduler.DAGScheduler: Parents of final stage: List()
2019-07-08 14:05:52,492 INFO scheduler.DAGScheduler: Missing parents: List()
2019-07-08 14:05:52,596 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2019-07-08 14:05:53,314 WARN util.SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
2019-07-08 14:05:53,404 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1936.0 B, free 90.3 MB)
2019-07-08 14:05:53,607 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1256.0 B, free 90.3 MB)
2019-07-08 14:05:53,625 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on pi1:46437 (size: 1256.0 B, free: 90.3 MB)
2019-07-08 14:05:53,639 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
2019-07-08 14:05:53,793 INFO scheduler.DAGScheduler: Submitting 7 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6))
2019-07-08 14:05:53,801 INFO cluster.YarnScheduler: Adding task set 0.0 with 7 tasks
2019-07-08 14:06:08,910 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-07-08 14:06:23,907 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-07-08 14:06:38,907 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-07-08 14:06:47,677 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.0.106:44936) with ID 1
2019-07-08 14:06:48,266 INFO storage.BlockManagerMasterEndpoint: Registering block manager pi6:39443 with 90.3 MB RAM, BlockManagerId(1, pi6, 39443, None)
2019-07-08 14:06:48,361 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, pi6, executor 1, partition 0, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:48,371 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, pi6, executor 1, partition 1, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:48,375 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, pi6, executor 1, partition 2, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:48,379 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, pi6, executor 1, partition 3, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:50,877 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on pi6:39443 (size: 1256.0 B, free: 90.3 MB)
2019-07-08 14:06:52,001 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, pi6, executor 1, partition 4, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:52,024 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, pi6, executor 1, partition 5, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:52,039 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, pi6, executor 1, partition 6, PROCESS_LOCAL, 7877 bytes)
2019-07-08 14:06:52,115 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 3733 ms on pi6 (executor 1) (1/7)
2019-07-08 14:06:52,143 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3891 ms on pi6 (executor 1) (2/7)
2019-07-08 14:06:52,144 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 3776 ms on pi6 (executor 1) (3/7)
2019-07-08 14:06:52,156 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 3780 ms on pi6 (executor 1) (4/7)
2019-07-08 14:06:52,217 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 222 ms on pi6 (executor 1) (5/7)
2019-07-08 14:06:52,249 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 215 ms on pi6 (executor 1) (6/7)
2019-07-08 14:06:52,262 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 247 ms on pi6 (executor 1) (7/7)
2019-07-08 14:06:52,270 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 
2019-07-08 14:06:52,288 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 59.521 s
2019-07-08 14:06:52,323 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 60.246659 s
Pi is roughly 3.1389587699411
2019-07-08 14:06:52,419 INFO server.AbstractConnector: Stopped Spark@89f072{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-07-08 14:06:52,432 INFO ui.SparkUI: Stopped Spark web UI at http://pi1:4040
2019-07-08 14:06:52,473 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
2019-07-08 14:06:52,602 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
2019-07-08 14:06:52,605 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
2019-07-08 14:06:52,640 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
2019-07-08 14:06:52,649 INFO cluster.YarnClientSchedulerBackend: Stopped
2019-07-08 14:06:52,692 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
2019-07-08 14:06:52,766 INFO memory.MemoryStore: MemoryStore cleared
2019-07-08 14:06:52,769 INFO storage.BlockManager: BlockManager stopped
2019-07-08 14:06:52,825 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
2019-07-08 14:06:52,851 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
2019-07-08 14:06:52,902 INFO spark.SparkContext: Successfully stopped SparkContext
2019-07-08 14:06:52,927 INFO util.ShutdownHookManager: Shutdown hook called
2019-07-08 14:06:52,935 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-1d1b5b79-679d-4ffe-b8aa-7e84e6be10f2
2019-07-08 14:06:52,957 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-ca0b9022-2ba9-45ff-8d63-50545ef98e55
```

If you sift through the output of the above command, you can see the the next result is:

```bash
took 60.246659 s
Pi is roughly 3.1389587699411
```

The very last thing I'd like to advise you to do is to add `stop-dfs.sh` and `stop-yarn.sh` to your `clusterreboot` and `clustershutdown` functions, if you've defined them. You should shut down the Hadoop cluster before powering off all of the machines, then reboot it when the machines boot back up:

```bash
function clusterreboot {
  stop-yarn.sh && stop-dfs.sh && \
  clustercmd sudo shutdown -r now
}
```

```bash
function clustershutdown {
  stop-yarn.sh && stop-dfs.sh && \
  clustercmd sudo shutdown now
}
```


[[ back to top ]](#top)

## <a name="conclusion">Conclusion</a>

So that's it! Hopefully this guide was helpful to you, whether you're setting up a Raspberry Pi, building your own Hadoop and Spark cluster, or you just want to learn about some Big Data technologies. Please let me know if you find any issues with the above guide; I'd be happy to amend it or make it clearer.

If you want more information about [commands you can run](https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html) on HDFS or [how to submit a job to Spark](https://spark.apache.org/docs/latest/submitting-applications.html), please click on those links. Thanks for making it to the end!

If you like content like this, be sure to [follow me on Twitter](https://twitter.com/_awwsmm) and [here on Dev.To](https://dev.to/awwsmm). I post about Java, Hadoop and Spark, R, and more.

[[ back to top ]](#top)