Leveraging the pipe method to write beautiful and concise data transformations in pandas
Tags: python, datascience, tutorial, codequality
Published: 2023-01-29 22:22:22

When it comes to **data science** and **analysis**, being able to **prepare and transform** our data is a **critical** component of any **successful** project

So let's learn how we can leverage the [`pandas`](https://pandas.pydata.org/) `pipe` method in Python to **abstract** complex data transformations into **easy-to-read, self documenting** operations!

## Table of Contents 
- [Overview of the .pipe() method](#overview-of-the-pipe-method)
- [A concrete example of the pipe method](#a-more-concrete-example-of-the-pipe-method")
- [The benefits of using the pipe operation](#the-benefits-of-using-the-pipe-operation)
- [Conclusion](#conclusion)
- [Additional resources](#additional-resources)

{% embed https://www.christophergreening.com %}

---

## Overview of the .pipe() method <a src="#overview-of-the-pipe-method"></a>

```python
import pandas as pd
```

The [`pipe`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pipe.html) method allows us to **chain** [`Series`](https://pandas.pydata.org/docs/reference/api/pandas.Series.html) or [`DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) data transformations together in a **semantically continuous** pipeline of **inputs** and **outputs**

It accomplishes this by leveraging Python's support for **higher-order functions** - the ability to pass a function as an argument to another function

Let's take a look at a simple example (*NOTE: assume the functions and `DataFrame` are pre-defined offscreen*):

```python
transformed_df = (
    df
    .pipe(_select_columns)
    .pipe(_multiply_columns_by_two)
    .pipe(_filter_segments)
)
```

The code snippet above shows each pipe method:
1. **Inputting** the output from the previous pipe
2. Performing a  **transformation** (i.e. selecting columns)
3. **Chaining the output** into the input of the next pipe

*"Wait I still don't understand what any of this means!!! Can we take a look at a more concrete example?!"*

No worries! Yeah - let's take a look at a more concrete example in the next section

---

## A concrete example of the pipe method
<a src="#a-more-concrete-example-of-the-pipe-method"></a>

Let's pretend we have a `DataFrame`, let's call it `town_df`, that contains **weekly time-series data** for how much electricity every single town in the United States consumes

```python
import pandas as pd
town_df = pd.read_csv("time_series_data_for_every_single_town_in_the_united_states.csv")
```

And let's say we want to perform these specific transformations in this specific order:
1. **select** relevant columns
2. **filter** date range
3. **approximate** missing values
4. **map** town to state
5. **aggregate** up to week and state
6. **upsample** week frequency to daily
7. **interpolate** daily values

*Wouldn't it be great if we could implement each of those steps as it's own self-contained function and then \*pipe\* those functions together in an explicitly obvious chain of transformations?...*

Well I'm glad you asked (:wink:)! Check this out:

```python
transformed_df = (
    df
    .pipe(_select_relevant_columns)
    .pipe(_filter_date_range)
    .pipe(_approximate_missing_values)
    .pipe(_map_town_to_state)
    .pipe(_aggregate_up_to_week_and_state)
    .pipe(_upsample_week_frequency_to_daily)
    .pipe(_interpolate_daily_values)
)
```

And that's it! 

A clear and concise chain of **immediately obvious** data transformations - let's talk about some of the benefits of writing our code like this

## The benefits of using the pipe method
<a src="#the-benefits-of-using-the-pipe-operation"></a>

You may have noticed that I did not *explicitly* reveal any of the implementation details behind any of the piped functions

And yet you *probably* didn't have a hard time understanding (at least from a top-level view) of what transformations were taking place behind the scenes!

I bet you could even show this to someone that has never written a single line of code in their life and even *they'd* be able to get the overall gist of what's happening to the dataset

While some simple transformations can be accomplished in a single line of code, more complex transformations might take dozens, hundreds, or even *thousands* of lines before we can move onto the "next" transformation

```python
transformed_df = (
    df
    .pipe(_some_oneliner_transformation)
    .pipe(_some_million_lines_of_code_transformation_but_guess_what_you_dont_have_to_know_how_its_implemented)
)
```

So being able to abstract the implementation details under a well-defined unit or block of code removes the cognitive overhead of having to read *every single line* to know what's going on - you can just focus on the big picture 

And when something (*inevitably*) does go wrong you're able to isolate, test, and debug your inputs and outputs because they're already logically isolated into well-defined units

---

## Conclusion <a src="#conclusion"></a>

If you want to take this a step further and **practice** with sample code and data, I've pulled together a **full working example** for you to explore on [GitHub](https://github.com/chris-greening/chris-greening-blog/blob/main/posts/Leveraging%20the%20pandas%20.pipe()%20method%20to%20write%20beautiful%20and%20concise%20data%20transformations%20in%20Python/code/pipe_method_example.py)!

**Thanks so much for reading** and if you liked my content, be sure to check out some of my other work or **connect** with me on social media or my [**personal website**](https://www.christophergreening.com/) :smile: 

{% embed https://www.christophergreening.com %}

Cheers!

{% embed https://dev.to/chrisgreening/connecting-to-a-relational-database-using-sqlalchemy-and-python-1619 %} 
{% embed https://dev.to/chrisgreening/joining-multiple-datasets-on-the-same-column-in-r-using-dplyr-and-purrr-3pa7 %}

---

## Additional resources <a src="#additional-resources"></a>

- [Using Pandas pipe function to improve code readability by B. Chen](https://towardsdatascience.com/using-pandas-pipe-function-to-improve-code-readability-96d66abfaf8)
- [Higher order functions in Python](https://www.geeksforgeeks.org/higher-order-functions-in-python/)
- [10 minutes to pandas](https://pandas.pydata.org/docs/user_guide/10min.html)
- [Pandas DataFrame pipe() Method](https://www.w3schools.com/python/pandas/ref_df_pipe.asp)