# Optimizing Pandas

## Summary


Performance of Pandas can be improved in terms of memory usage and speed of computation. Optimizations can be done in broadly two ways: (a) learning best practices and calling Pandas APIs the right way; (b) going under the hood and optimizing the core capabilities of Pandas. This article covers both these aspects.

As a general observation, Pandas was designed for data analysis and usability. On the other hand, NumPy was designed for performance. However, there are some techniques to make Pandas more performant without sacrificing its ease of use. There are also third-party libraries that build on or work with Pandas to achieve better performance.

## Discussion

### What's an essential approach towards performant Pandas code?

Perhaps the most important rule is to avoid using loops in Pandas code. Looping over a Series or a DataFrame processes data one item or row/column at a time. Instead, operations should be **vectorized**. This means an operation should be performed on the entire Series or DataFrame row/column. Developers should think of all operations as matrix computations that can be parallelized. 

The worst or slowest approach is to use `df.iloc[i][j]` within a `for` loop. A slightly better approach is to use `df.iterrows()` that returns a tuple containing a row index and a Series for the row. Even better is to remove loops completely and use `df.apply()` that takes as first argument a function that's applied to each row or column. This internally uses Cython iterators. 

By far, the best approach is to use vectorization. For example, if `addr` is a Series containing addresses, `addr.str.upper().str.replace('.', '')` is applied to all items "at once". Register size and the computer's instruction set determines how many items can be parallelized. 


### For data manipulation, what are some techniques for faster code?

Indices are commonly used in Pandas for lookup or merging two datasets. It's faster to **merge based the index**. For example, `listings_.merge(reviews_, left_index=True, right_index=True)` is faster than `listings.merge(reviews, on='listing_id')`, given that `reviews_ = reviews.set_index('listing_id')` and `listings_ = listings.set_index('listing_id')`. 

When chaining multiple operations, the order is important. For example, it's faster to **filter first and then merge**. Thus, `listings.merge(reviews[reviews[listing_id].isin(listings[listing_id])], on='listing_id')` is faster than `listings.merge(reviews, on='listing_id')`. 


### What are some best practices for handling datetime data type?

In general, adopt vectorization over explicit loops. If this is not possible, prefer `df.apply()`, `df.itertuples()` or `df.iterrows()` in that order. 

If datetime values are stored as object data type, use `pd.to_datetime()` method to convert to datetime type. Explicitly passing `format` argument to this method speeds up the conversion. 

In electricity billing, suppose different tariffs are applied based on time of the day. We can use `df.isin()` to selectively apply the tariff to data subsets. An even better way is to use `pd.cut()`. We can reduce execution time further by converting data to NumPy arrays. In this example, it's also convenient to use the datetime column as the index. 


### What are some techniques to improve Pandas performance?

There are a few known techniques to speed up Pandas:

    + **Cython**: Cython is a superset of Python. It's Python code with additional type information. This is translated into optimized C/C++ code and then compiled as Python extension modules. Passing NumPy ndarray (rather than creating and passing Pandas Series) into Cython functions gives further improvement.
    + **Numba**: This is a just-in-time (JIT) compiler. With a few annotations to Python code (`@numba.jit`, `@numba.vectorize`, etc), runtime performance can come close to C/C++ or Fortran. There's support for both CPU and GPU hardware. However, as of Numba v0.20, optimizations work on NumPy arrays and not Pandas objects.
    + **`eval()` and `query()`**: For datasets larger than 10,000 rows, `pandas.eval()`, `DataFrame.eval()` and `DataFrame.query()` evaluate expressions faster. Only some expressions can be optimized. For example, `is`, `lambda`, `yield`, comprehensions and generator expressions can't be optimized. Package *numexpr* needs to be installed.Pandas User Guide documents all the above techniques with useful code examples.

Particularly for large datasets, Pandas official documentation recommends *numexpr* that uses multiple cores, smart chunking and caching; and *bottleneck* that uses Cython routines to speed up some types of `nan` evaluations. 


### How does Pandas compare against NumPy in terms of performance?

NumPy is faster than Pandas because most of the calculations happen using precompiled optimized C code. Although Pandas makes use of NumPy, it does a lot of housekeeping: tracking data types and indices, and checking for errors. This makes Pandas slower than NumPy. Therefore, one way to speed up Pandas code is to convert critical computations into NumPy, for example by calling `to_numpy()` method. 

One study on selecting a data subset showed NumPy outperforming Pandas by 10x to 1000x, with the gains diminishing on very large datasets. Regardless of DataFrame size, Pandas paid an initial penalty 1ms. Similar results were seen when taking square root of numbers. 

Within NumPy, operations within an array are faster than operations that span multiple arrays. Thus, `a1 + a2` is slower than `np.sum(a_both, axis=1)` if we reorganize the data as `a_both[0, :] = a1` and `a_both[1, :] = a2`. 


### How do I deal with large datasets in Pandas?

Pandas does in-memory analytics, which makes it difficult to handle large datasets that don't fit in system memory. One approach is to **load less data**, such as loading only columns needed for analysis. For example, `pandas.read_csv()` has argument `usecols` to do this. 

Use **efficient datatypes**. Text data of low cardinality are those with lot of values but only a few unique values. Storing each value as a complete string is memory inefficient. Instead, they should be stored as categorical data. Data with all unique values will not benefit from such a conversion. Numerical values can be downcast to smallest types using `pandas.to_numeric()`. 

**Chunking** is a technique to split the dataset and process in chunks. This works best when chunks require little coordination. Some methods such as `pandas.read_csv()` has argument `chunksize` to do chunking. Group-by operations are typically harder to do in chunks. 

Where **complex expressions** are involved such as `(df.A < 0.5) & (df.B < 0.5)`), each sub-expression creates temporary memory. This can be avoided with `pd.eval()`, `df.eval()` and `df.query()`. This improves performance only on large datasets. 


### How does Pandas store a DataFrame under the hood?

Pandas groups columns of the same type into what is called a **block**. A DataFrame is actually stored as one or more blocks. Using metadata, these blocks are composed into a DataFrame by a **BlockManager**. Thus, only a block is contiguous in memory. 

It's possible to inspect the blocks using `df['A'].values.base` for column A. The output will also show values of other columns in that block. To see the internal storage of all blocks, call `df._data`. 

Referring to the figure, slicing with `df.loc[:'2015-07-03', ['quantity', 'points']]` doesn't involve data copy since both columns are of the same data type. But any cross-dtype slicing will typically involve copying. Another performance killer is appending a row to a DataFrame. This would result in reallocating memory and copying every block. Thus, it's better to concatenate two big datasets rather than append one to the other row by row. When a new column is added, block copy is deferred until an operation requires it. 


### Which third-party libraries help improve Pandas performance?

Indexing involves lots of lookups. **klib** is a C implementation that uses less memory and runs faster than Python's dictionary lookup. Since version 0.16.2, Pandas already uses klib. 

To run on multiple cores, use **multiprocessing**, **Modin**, **Ray**, **Swifter**, **Dask** or **Spark**. In one study, Spark did best on reading/writing large datasets and filling missing values. Pandas did best for group-by operation. An advantage of Swifter is that it decides what to use (vectorization, Pandas apply or Dask) depending on the dataset size. 

Swifter can also be used along with Modin. In fact, Modin partitions data in an optimal way but can work with other libraries that perform parallel execution. 

Dask can use multiple threads or processes on a single machine or a cluster of machines. Combining **Numba with Dask** is even better. 

PySpark is suitable for very large datasets but Pandas offers better APIs. A suitable approach is to use **Spark with Apache Arrow**. Arrow is an in-memory columnar format that helps to efficiently transfer data between Spark and Pandas. Based on Arrow, **PyPolars** is a highly performant DataFrame library. 

## Milestones

Jul  
2011

Wes McKinney, creator of Pandas, writes about **BlockManager** in a blog and explains how it works. It's the internal data structure in Pandas. A **block** is really a "homogeneously-typed N-dimensional NumPy ndarray object". Gluing together columns of the same type into a single block is called **consolidation**. In the early days of Pandas (0.1 and 0.2) a DataFrame was stored using Python `dict` until the BlockManager took over. 

Jun  
2015

With the release of Pandas 0.16.2, klib package is used to improve performance of indexing. This takes less memory and runs faster than Python's dictionary. This should not be confused with klib (first released in April 2020), a library that simplifies visualization of DataFrame. 

Sep  
2017

McKinney notes on his blog that Pandas requires typically **5-10 times RAM as the size of the dataset**. This is bad for analytics on large datasets. What's needed is a columnar format optimized for analytics with zero-copy access. This is where **Apache Arrow** fits in. Arrow 0.1.0 was released in October 2016. 

Nov  
2018

As a library for parallel computing in Python, **Dask 1.0.0** is released. Pandas DataFrame becomes Dask DataFrame whose processing can be scheduled on a multiple threads/processes of a single machine or distributed across machines. Dask takes care of the task scheduling. The release of Dask 0.2.0 can be traced to January 2015. 

Jan  
2020

**Pandas 1.0.0** is released. This provides a number of performance improvements. We mention a few: DataFrame arithmetic and comparison operations with scalars; indexing with a non-unique IntervalIndex; initializing a DataFrame using a range; `DataFrame.select_dtypes()` uses vectorization instead of a loop; comparing a Categorical with a scalar and the scalar is not found in the categories. 

Jul  
2020

**Swifter 1.0.0** is released. Swifter "efficiently applies any function to a pandas dataframe or series in the fastest available manner". If the function can be vectorized, it vectorizes it. If not, it selects the best of Pandas, Dask or Swifter apply method. Swifter can be traced to April 2018 when version 0.1 was released. 

Sep  
2020

Package **PyPolars 0.1.0** is released. It's implemented in Rust and uses Apache Arrow as its memory model. It supports lazy and eager evaluations. On datasets larger than 10,000 rows, it outperforms both pandas and pydatatable (another DataFrame library). 

Jan  
2021

**Pandas roadmap** as on this date notes a few items related to performance. Apache Arrow may be explored as an in-memory array library. At the least, Pandas would interoperate better with Arrow. BlockManager and 2-D blocks are complex. They could be replaced with a simpler collection of 1-D arrays. BlockManager uses both labels and positions for indexing. In future, it might use only the latter. Where Pandas accepts user-defined functions, developers should be able to provide Numba-jitted functions.