Azure Blob Storage with Pyspark
Tags: python, spark, bigdata, azure
Published: 2019-09-22 14:41:36

---
title: Azure Blob Storage with Pyspark
published: true
description: Azure Blob Storage is a Microsoft solution for storing objects in the cloud. It is optimized for storing large amounts of data and can be easily accessed by your Python/spark application
tags: python, spark, bigdata, azure
canonical_url: https://luminousmen.com/post/azure-blob-storage-with-pyspark
cover_image: https://thepracticaldev.s3.amazonaws.com/i/qxybvh9561jafz79v2k4.jpg
---

[Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) is a service for storing large amounts of data stored in [any format](https://luminousmen.com/post/big-data-file-formats) or binary data. This is a good service for creating data warehouses or data lakes around it to store preprocessed or raw data for future analytics.

In this post, I'll explain how to access Azure Blob Storage using [Spark framework](https://luminousmen.com/post/spark-anatomy-of-spark-application) on Python.

---

Azure blob requires to install additional libraries for accessing data from it, because it uses [wasb/wasbs protocol](https://blogs.msdn.microsoft.com/cindygross/2015/02/04/understanding-wasb-and-hadoop-storage-in-azure/), not de facto standard hdfs protocol. Wasbs protocol is just an extension built on top of the HDFS APIs. In order to access resources from Azure blob you need to add built jar files, named [hadoop-azure.jar](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure) and [azure-storage.jar](https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage) to `spark-submit` when you submitting a job.

```bash
$ spark-submit --py-files src.zip \
			--master yarn \
			--deploy-mode=cluster \
			--jars hadoop-azure.jar,azure-storage.jar
			src/app.py
```

Also if you're using Docker or deploying the application to a cluster, there is a tip for you as well. You don't need to pass additional jars if they are already in the right place — where pyspark can find it(it need to be done on every cluster node of course). Use the commands below with caution — the versions/links may be wrong!

```bash
$ wget -nc -nv -O /usr/local/lib/python3.5/site-packages/pyspark/jars/azure-storage-2.2.0.jar http://central.maven.org/maven2/com/microsoft/azure/azure-storage/2.2.0/azure-storage-2.2.0.jar
$ wget -nc -nv -O /usr/local/lib/python3.5/site-packages/pyspark/jars/hadoop-azure-2.7.3.jar http://central.maven.org/maven2/org/apache/hadoop/hadoop-azure/2.7.3/hadoop-azure-2.7.3.jar
```

On the application level, first of all as always in spark applications, you need to grab a Spark Session. Spark Session is the entry point for the cluster resources — for reading data and execute SQL queries over data and getting the results. 

```python
session = SparkSession.builder.getOrCreate()
```

Then set up an account key to your blob container:

```python
session.conf.set(
	"fs.azure.account.key.<storage-account-name>.blob.core.windows.net",
	"<your-storage-account-access-key>"
)
```

or SAS token:

```python
session.conf.set(
	"fs.azure.sas.<container-name>.blob.core.windows.net",
	"<sas-token>"
)
```

Once an account access key or SAS token is set up you're ready to read/write to Azure blob:

```python
sdf = session.read.parquet(
	"wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<prefix>"
)
sdf.show()
```

------


**Thank you for reading!**

Any questions? Leave your comment below to start fantastic discussions!

Check out [my blog](https://luminousmen.com/) or come to say hi 👋 on [Twitter](https://twitter.com/luminousmen) or subscribe to [my telegram channel](https://t.me/iamluminousmen).
Plan your best!