Views, Copies, and that annoying SettingWithCopyWarning
Tags: python, pandas
Published: 2021-02-04 23:54:12

---
title: Views, Copies, and that annoying SettingWithCopyWarning
published: true
date: 2021-02-04 23:49:44 UTC
tags: python,pandas
canonical_url: https://www.wrighters.io/views-copies-and-that-annoying-settingwithcopywarning/
---

If you’ve spent any time in pandas at all, you’ve seen `SettingWithCopyWarning`. If not, you will soon!

Just like any warning, it’s wise to not ignore it since you get it for a reason: it’s a sign that you’re probably doing something wrong. In my case, I usually get this warning when I’m knee deep in some analysis and don’t want to spend too much time figuring out how to fix it.

I’m going to cover a few typical examples of when this warning shows up, why it shows up, and how to quickly fix the underlying issue.

First, let’s make an example `DataFrame`. I’m using a handy Python package called [Faker](https://faker.readthedocs.io/en/stable/index.html) to create some test data. You may need to install it first, with `pip`.

```
%pip install Faker # notebook
pip install Faker # commmand line
```

As a quick aside, Faker is a great way to build test data for unit tests, test databases, or examples. It generates real-looking data that is not personally identifiable, since it’s all fake, but it’s based on rules that generate data combinations you’ll likely encounter in real life.

```
>>> import datetime
>>> import pandas as pd
>>> import numpy as np
>>> from faker import Faker
>>> fake = Faker()
>>> df = pd.DataFrame([
            [fake.first_name(),
             fake.last_name(),
             fake.date_of_birth(),
             fake.date_this_year(),
             fake.city(),
             fake.state_abbr(),
             fake.postalcode()]
                for _ in range(20)],
            columns = ['first_name', 'last_name', 'dob', 'lastupdate', 'city', 'state', 'zip'])
          
>>> df.head(3)
  first_name last_name dob        lastupdate city         state zip
0 Evan       Daniels   1943-05-27 2021-01-11 North Erin   AZ 27597
1 Christine  Herrera   2019-04-11 2021-01-29 Ellenview    AL 28989
2 Michelle   Warren    2015-05-29 2021-01-11 Mcknighttown VA 55551
```

## How do we set data again?

First, let’s just review the ways we can set data in a `DataFrame`, using use the `loc` or `iloc` indexers. These are for label based or integer offset based indexing respectively. (See [this article](https://www.wrighters.io/indexing-and-selecting-in-pandas-part-1/) for more detail on the two methods)

The first argument in the indexer is for the row, the second is for the column (or columns), and if we assign to this expression, we will update the underlying `DataFrame`.

Note that the index here is just a `RangeIndex`, so the labels are numbers. Because of that, even though I’m passing in int values to `loc`, this is looking up by label, not relative index.

```
>>> df.head(1)['zip']
0 27597
Name: zip, dtype: object
>>> df.loc[0, 'zip'] = '60601'
>>> df.head(1)['zip']
0 60601
Name: zip, dtype: object
>>> df.loc[0, ['city', 'state']] = ['Chicago', 'IL']
>>> df.head(1)
  first_name last_name dob lastupdate city state zip
0 Evan Daniels 1943-05-27 2021-01-11 Chicago IL 60601
>>> # Here's an example of an iloc update.
>>> df.iloc[0, 0] = 'Josh'
>>> df.head(1)
  first_name last_name dob        lastupdate city    state zip
0 Josh        Daniels  1943-05-27 2021-01-11 Chicago IL    60601
```

Now, you can also do updates with the array indexing operator, but this can look very confusing because remember that on a `DataFrame`, you are selecting columns first. I’d recommend not doing this for this reason alone, but as you’ll soon see, there are other issues that can arise.

```
>>> df["first_name"][0] = 'Joshy'
>>> df.head(1)
  first_name last_name dob        lastupdate city    state zip
0 Joshy      Daniels   1943-05-27 2021-01-11 Chicago IL    60601
```

## When do we see this warning?

OK, now that we have updated our `DataFrame` successfully, it’s time to see an example of where things can go wrong. For me, it’s very typical to select a subset of the original data to work with. For example, let’s say that we decide to only work with data where the person was born before 2000.

```
>>> dob_limit = datetime.date(2000, 1, 1)
>>> sub = df[df['dob'] < dob_limit]
>>> sub.shape
(16, 7)
>>> idx = sub.head(1).index[0] # save the location for update attempts below
>>> sub.head(1)
  first_name last_name dob        lastupdate city    state zip
0 Joshy      Daniels   1943-05-27 2021-01-11 Chicago IL    60601
```

Let’s try to update the `lastupdate` column.

```
>>> sub.loc[idx, 'lastupdate'] = datetime.date.today()
/Users/mcw/.pyenv/versions/3.8.6/envs/pandas/lib/python3.8/site-packages/pandas/core/indexing.py:670: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  iloc._setitem_with_indexer(indexer, value)
<ipython-input-14-5f1769c87aaf>:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  sub.loc[idx, 'lastupdate'] = datetime.date.today()
```

Boom! There it is, we are told we are trying to set values on a copy of a slice from a `DataFrame`. What ended up happening here? Well, `sub` _was_ updated, but `df` _wasn’t_, even though we had the warning.

```
>>> sub.loc[idx, 'lastupdate']
datetime.date(2021, 2, 4)
>>> df.loc[idx, 'lastupdate']
datetime.date(2021, 1, 11)
```

Pandas is warning you that you might have not done what you expected. When you created `sub`, you ended up with a copy of the data in `df`. When you updated the value, you’re warned that you only updated the copy, not the original.

## So how should you fix it?

There are two primary ways to address this, and which one you choose depends on what you are trying to accomplish in your code. The warning is telling you that you chose a path that could cause confusion or error down the road, and is pointing you toward using the best practices for updating data.

### Update the original

If your intention is to update your original data, you just need to update it directly. So instead of doing your update on `sub`, do it on `df` instead.

```
>>> df.loc[idx, 'lastupdate'] = datetime.date.today()
>>> df.loc[idx, 'lastupdate']
datetime.date(2021, 2, 4)
```

Now note that when you do this, since your view is a copy, it isn’t updated. If you want both `sub` and `df` to match, you need to either update both or recreate `sub` after the update. Because of this, it’s important for you to pause and think any time you update a `DataFrame`. Have you created views of this data that now need to be refreshed?

### Update the copy

If your goal is to update the copy of the data only, to eliminate the warning, tell pandas you want that view to always be a copy.

```
>>> sub2 = df[df['dob'] < dob_limit].copy()
>>> sub2.loc[idx, 'lastupdate'] = datetime.date.today()
>>> sub2.loc[idx, 'lastupdate']
datetime.date(2021, 2, 4)
```

### In between

One common situation that happens is an initial full sized `DataFrame` is narrowed down to a much smaller one by filtering the data. Maybe new columns are added as part of some calculations, and then as a final result, the original `DataFrame` should be updated. One way to do that is to use the index to help you out.

```
>>> sub3 = df[df['dob'] < dob_limit].copy() # we'll be updating this DataFrame
>>> sub3['manualupdate'] = datetime.date.today() - datetime.timedelta(days=10) # you can modify this DataFrame
>>> sub3 = sub3.head(3) # or even make it smaller
>>> sub3['manualupdate']
0 2021-01-25
1 2021-01-25
3 2021-01-25
Name: manualupdate, dtype: object
```

Now, we’ll use the fact that `sub3` shares an index with the original `df` to use it to update the data. We can update all matching row of column `lastupdate` for example.

```
>>> df.loc[sub3.index, 'lastupdate'] = sub3['manualupdate']
>>> df.loc[sub3.index]
  first_name last_name dob        lastupdate city         state zip
0 Joshy      Daniels   1943-05-27 2021-01-25 Chicago      IL 60601
3 Vernon     Hernandez 1989-04-10 2021-01-25 South Mark   NE 05048
4 Mary       Munoz     1933-03-16 2021-01-25 Ewingborough OK 31127
```

Now, you can see that those rows were updated from our smaller subset of data.

## Subsets of columns

You also may encounter this warning when working with subsets of columns in a `DataFrame`.

```
>>> df_d = df[['zip']]
>>> df_d.loc[idx, 'zip'] = "00313" # SettingWithCopyWarning
```

A great way to suppress the warning here is to do a full slice with `loc` in your initial selection. You can also use `copy`.

```
>>> df_d = df.loc[:, ['zip']]
>>> df_d.loc[idx, 'zip'] = "00313"
```

### For completeness, some more details

Now you can read about this warning in many [other](https://www.dataquest.io/blog/settingwithcopywarning/) [places](https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas), and if you’ve come here through a search engine maybe you’ve already found them either confusing or not directly applicable to your situation. I took a slightly different approach above to show the situation where I usually see this error. However, a more common reason new pandas users encounter this error is when trying to update their `DataFrame` using the array index operator (`[]`).

```
>>> df[df['dob'] < dob_limit]['lastupdate'] = datetime.date.today()
file.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[df['dob'] < dob_limit]['lastupdate'] = datetime.date.today()
```

The fix here is pretty straightforward, use `loc`. Let’s give that a try.

```
>>> df.loc[df['dob'] < dob_limit, 'lastupdate'] = datetime.date.today() - datetime.timedelta(days=1)
>>> df.loc[df['dob'] < dob_limit].head(1)
  first_name last_name dob lastupdate city state zip
0 Joshy Daniels 1943-05-27 2021-02-03 Chicago IL 60601
```

That works. The warning here was telling us that our first update is (potentially) operating on a copy of our original data. I don’t think this is quite as obvious as our opening case because pandas has some complicated reasons for choosing to sometimes return a copy and sometimes return a view into the original data, and this may not seem obvious when the update is on one line. When it can detect that this is happening, it raises this warning.

This is called chained assignment. The assignment above with the warning is really doing this:

```
df. __getitem__ (df. __getitem__ ('dob') < dob_limit). __setitem__ ('lastupdate', datetime.date.today())
```

When you use the array index operator, the ` __getitem__ ` and ` __setitem__ ` methods are invoked for getting and setting respectively. That first function call to ` __getitem__ ` is returning a copy of the data, then attempting to set data on it, triggering the warning.

If we use `loc`, though, it will be doing this, without returning a temporary view.

```
df.loc. __setitem__ ((df. __getitem__ ('dob') < dob_limit, 'lastupdate'), datetime.date.today())
```

So whenever you see this warning, just look at your code and check two things. Did you try to update the data using `[]`? If so, switch to `loc` (or `iloc`). If you’re doing that and it’s still complaining, it’s because your `DataFrame` was created from another `DataFrame`. Either make a full copy if you plant to update it, or update your original `DataFrame` instead.

The post [Views, Copies, and that annoying SettingWithCopyWarning](https://www.wrighters.io/views-copies-and-that-annoying-settingwithcopywarning/) appeared first on [wrighters.io](https://www.wrighters.io).