Spark on AWS Glue: Performance Tuning 3 ( Impact of Partition Quantity)
Tags: aws, glue, spark, performance
Published: 2023-07-16 21:38:08

This is a continuation of my previous posts as follows.

1. [Spark on AWS Glue: Performance Tuning 1 (CSV vs Parquet)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-1-csv-vs-parquet-2l5)

2. [Spark on AWS Glue: Performance Tuning 2 (Glue DynamicFrame vs Spark DataFrame)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-2-glue-dynamicframe-vs-spark-dataframe-12gb)

3. [Spark on AWS Glue: Performance Tuning 3 (Impact of Partition Quantity)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-3-impact-of-partition-quantity-4i55)

4. [Spark on AWS Glue: Performance Tuning 4 (Spark Join)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-4-spark-join-39m3)

5. [Spark on AWS Glue: Performance Tuning 5 (Using Cache)](https://dev.to/tmyoda/spark-on-aws-glue-performance-tuning-5-using-cache-4ob5)

## Impact of Partition Quantity

we will compare the speeds of three different partition numbers: 1, default (unspecified), and 300. 
And also we will compare with and without shuffling.

### DataFrame Preparation
```python
df = spark.read.format("parquet").load("s3:/.../parquet-chunk-high/")
one_part_df = df.coalesce(1)
print(one_part_df.rdd.getNumPartitions())
one_part_df.count()

part_df = spark.read.format("parquet").load("s3:/.../parquet-chunk-high/")
print(part_df.rdd.getNumPartitions())
part_df.count()

df = spark.read.format("parquet").load("s3:/.../parquet-chunk-high/")
part_300_df = df.repartition(300)
print(part_300_df.rdd.getNumPartitions())
part_300_df.count()
```

```
1
94
300
```
By default (unspecified) 94 partitions were read.

### Without shuffling

```python
with timer('one part filter'):
    result = one_part_df.filter(one_part_df['request_processing_time'] < 0.0008).count()
    print(result)
    
with timer('part filter'):
    result = part_df.filter(part_df['request_processing_time'] < 0.0008).count()
    print(result)

with timer('part 300 filter'):
    result = part_300_df.filter(part_300_df['request_processing_time'] < 0.0008).count()
    print(result)
```
```
9
[one part filter] done in 45.5252 s
9
[part filter] done in 1.4579 s
9
[part 300 filter] done in 3.5410 s
```
94 partitions is the fastest.


### With shuffling
```python
with timer('one part shuffle'):
    result = one_part_df.groupBy('elb_name').agg(F.mean('request_processing_time').alias("mean_time")).orderBy('mean_time').count()
    print(result)
    
with timer('part shuffle'):
    result = part_df.groupBy('elb_name').agg(F.mean('request_processing_time').alias("mean_time")).orderBy('mean_time').count()
    print(result)

with timer('part 300 shuffle'):
    result = part_300_df.groupBy('elb_name').agg(F.mean('request_processing_time').alias("mean_time")).orderBy('mean_time').count()
    print(result)
    
```
```
9
[one part shuffle] done in 78.1068 s
9
[part shuffle] done in 2.6624 s
9
[part 300 shuffle] done in 12.2829 s
```

Again, 94 partitions is the fastest.

### Summary
- Basically, a larger number of partitions is better than a smaller number.
- "just right" number of partitions is the most efficient.