A Gentle Intro to Apache Spark for Developers
Published: 2019-08-23 16:28:30

---
title: A Gentle Intro to Apache Spark for Developers
published: true
description:spark,apache,data-science
tags:
---

I recently hosted an [online meetup on Apache Spark](https://www.crowdcast.io/e/8u3d6q3c) with [IBM Developer](https://medium.com/u/262975298e3a). Spark has been around for a few years, but the **_interest in still growing to my surprise_**. Apache Spark was developed at the University of California, Berkeley’s AMPLab. The Spark codebase was open sourced and donated to the Apache Software Foundation in 2010.

![](https://cdn-images-1.medium.com/max/1024/1*tP-dw4Oj_42BYbkdtYbjMA.png)<figcaption>Apache Spark</figcaption>

The background of the attendees was quite diverse

- **Developer (25%)**
- **Architect (12.5%)**
- **Data Scientist (41.7%)**
- **Other (12.8%)**

We looked at the **WHAT** and **WHY** of spark and then dove in the three data structures that you might encounter when working with Spark …

![](https://cdn-images-1.medium.com/max/970/1*KQYeTwCltFJBca0UZtD9-A.png)

We also looked at some Transformations, Actions, built-in functions and UDF (user defined functions). For example the following function creates a new column called **GENDER** based on the contents of the column **GenderCode.**

```
# ------------------------------
# Derive gender from salutation
# ------------------------------
def deriveGender(col):
    """ input: pyspark.sql.types.Column
        output: "male", "female" or "unknown"
    """    
    if col in ['Mr.', 'Master.']:
        return 'male'
    elif col in ['Mrs.', 'Miss.']:
        return 'female'
    else:
        return 'unknown';
    
deriveGenderUDF = func.udf(lambda c: deriveGender(c), types.StringType())
customer_df = customer_df.withColumn("GENDER", deriveGenderUDF(customer_df["GenderCode"]))
customer_df.cache()
```

**withColumn** creates a new column in the customer\_df dataframe with the values from **deriveGenderUDF** (our user defined function). The deriveGenderUDF is essentially the **deriveGender** function. If this does not make sense, watch the webinar as we go into a lot more detail.

![](https://cdn-images-1.medium.com/max/981/1*FTBmSEX9V89ZrG2ya77uYA.png)

Finally, we created a spark cluster environment on IBM Cloud, and used a Jupyter notebook to explore customer data with the following columns …

```
"CUST_ID", 
"CUSTNAME", 
"ADDRESS1", 
"ADDRESS2", 
"CITY", 
"POSTAL_CODE", 
"POSTAL_CODE_PLUS4", 
"STATE", 
"COUNTRY_CODE", 
"EMAIL_ADDRESS", 
"PHONE_NUMBER",
"AGE",
"GenderCode",
"GENERATION",
"NATIONALITY", 
"NATIONAL_ID", 
"DRIVER_LICENSE"
```

After cleaning the data using in-built and user defined methods, we used PixieDust to visualize the data. The cool thing about pixieDust is that you don’t need to set it up or configure it. You just pass it a Spark DataFrame or a Pandas DataFrame and you are good to go! You can find the complete [notebook here](https://github.com/IBM/analyze-customer-data-spark-pixiedust).

![](https://cdn-images-1.medium.com/max/960/1*VbfDDknhZAA_mqzSzTb0zw.png)

Thank you [IBM Developer](https://medium.com/u/262975298e3a) and [Max Katz](https://medium.com/u/bf01a11701fe) for the opportunity to present and special thanks to Lisa Jung for being a patient co-presenter!