How to remove a column from a DataFrame, with some extra detail
Tags: pandas, python
Published: 2021-03-08 00:23:28

---
title: How to remove a column from a DataFrame, with some extra detail
published: true
date: 2021-03-08 00:14:57 UTC
tags: Pandas,Python
canonical_url: https://www.wrighters.io/how-to-remove-a-column-from-a-dataframe/
---

Removing one or more columns from a pandas `DataFrame` is a pretty common task, but it turns out there are a number of possible ways to perform this task. I found that [this StackOverflow question](https://stackoverflow.com/questions/13411544/delete-column-from-pandas-dataframe), along with solutions and discussion in it raised a number of interesting topics. It is worth digging in a little bit to the details.

First, what’s the “correct” way to remove a column from a `DataFrame`? The standard way to do this is to think in SQL and use `drop`.

```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.arange(25).reshape((5,5)),               
                  columns=list("abcde"))

display(df)

try:
    df.drop('b')
except KeyError as ke:
    print(ke)

   a  b  c  d  e
0  0  1  2  3  4
1  5  6  7  8  9
2 10 11 12 13 14
3 15 16 17 18 19
4 20 21 22 23 24
"['b'] not found in axis"
```

Wait, what? Why an error? That’s because the default axis that `drop` works with is the rows. As with many pandas methods, there’s more than one way to invoke the method (which some people find frustrating). 

You can drop rows using `axis=0` or `axis='rows'`, or using the `labels` argument.

```
df.drop(0) # drop a row, on axis 0 or 'rows'
df.drop(0, axis=0) # same
df.drop(0, axis='rows') # same
df.drop(labels=0) # same
df.drop(labels=[0]) # same
```

```
   a  b  c  d  e
1  5  6  7  8  9
2 10 11 12 13 14
3 15 16 17 18 19
4 20 21 22 23 24
```

### Again, how do we drop a column?

We want to drop a column, so what does that look like? You can specify the `axis` or use the `columns` parameter.

```
df.drop('b', axis=1) # drop a column
df.drop('b', axis='columns') # same
df.drop(columns='b') # same
df.drop(columns=['b']) # same
```

```
   a  c  d  e
0  0  2  3  4
1  5  7  8  9
2 10 12 13 14
3 15 17 18 19
4 20 22 23 24
```

There you go, that’s how you drop a column. Now you have to either assign to a new variable, or back to your old variable, or pass in `inplace=True` to make the change permanent.

```
df2 = df.drop('b', axis=1)

print(df2.columns)
print(df.columns)

Index(['a', 'c', 'd', 'e'], dtype='object')
Index(['a', 'b', 'c', 'd', 'e'], dtype='object')
```

It’s also worth noting that you can drop _both_ rows and columns at the same time using drop by using the `index` and `columns` arguments at once, and you can pass in multiple values.

```
df.drop(index=[0,2], columns=['b','c'])
```
```
   a  d  e
1  5  8  9
3 15 18 19
4 20 23 24
```

If you didn’t have the drop method, you can basically obtain the same results through indexing. There are many ways to accomplish this, but one equivalent solution is indexing using the `.loc` indexer and `isin`, along with inverting the selection.

```
df.loc[~df.index.isin([0,2]), ~df.columns.isin(['b', 'c'])]
```

```
   a  d  e
1  5  8  9
3 15 18 19
4 20 23 24
```

If none of that makes sense to you, I would suggest reading through my series on selecting and indexing in pandas, starting [here](https://www.wrighters.io/indexing-and-selecting-in-pandas-part-1/).

## Back to the question

Looking back at the original question though, we see there is another available technique for removing a column.

```
del df['a']
df
```
```
   b  c  d  e
0  1  2  3  4
1  6  7  8  9
2 11 12 13 14
3 16 17 18 19
4 21 22 23 24
```

Poof! It’s gone. This is like doing a drop with `inplace=True`. 

## What about attribute access?

We also know that we can use attribute access to _select_ columns of a `DataFrame`.

```
df.b
```

```
0  1
1  6
2 11
3 16
4 21
Name: b, dtype: int64
```

Can we delete the column this way?

```
del df.b
```

```
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-10-0dca358a6ef9> in <module>
----> 1 del df.b

AttributeError: b
```

We cannot. This is not an option for removing columns with the current pandas design. Is this technically impossible? How come `del df['b'] `works but `del df.b`  doesn’t?. Let’s dig into those details and see whether it would be possible to make the second work as well.

The first version works because in pandas, the `DataFrame` implements the ` __delitem__ ` method which gets invoked when you execute `del df['b']`. But what about `del df.b`, is there a way to handle that?

First, let’s make a simple class that shows how this works under the hood. Instead of being a real `DataFrame`, we’ll just use a `dict` as a container for our columns (which could really contain anything, we’re not doing any indexing here).

```
class StupidFrame:
    def __init__ (self, columns):
        self.columns = columns
        
    def __delitem__ (self, item):
        del self.columns[item]
        
    def __getitem__ (self, item):
        return self.columns[item]
    
    def __setitem__ (self, item, val):
        self.columns[item] = val
            
f = StupidFrame({'a': 1, 'b': 2, 'c': 3})
print("StupidFrame value for a:", f['a'])
print("StupidFrame columns: ", f.columns)
del f['b']
f.d = 4
print("StupidFrame columns: ", f.columns)
```

```
StupidFrame value for a: 1
StupidFrame columns: {'a': 1, 'b': 2, 'c': 3}
StupidFrame columns: {'a': 1, 'c': 3}
```

A couple of things to note here. First, we how that we can access the data in our `StupidFrame` with the index operators (`[]`), and use that for setting, getting, and deleting items. When we assigned `d` to our frame, it wasn’t added to our columns because it’s just a normal instance attribute. If we wanted to be able to handle the columns as attributes, we have to do a little bit more work.

So following the example from pandas (which supports attribute access of columns), we add the ` __getattr__ ` method, but we also will handle setting it with the ` __setattr__ ` method and pretend that any attribute assignment is a ‘column’. We have to update our instance dictionary (\_\_`dict__`) directly to avoid an infinite recursion.

```
class StupidFrameAttr:
    def __init__ (self, columns):
        self. __dict__ ['columns'] = columns
        
    def __delitem__ (self, item):
        del self. __dict__ ['columns'][item]
        
    def __getitem__ (self, item):
        return self. __dict__ ['columns'][item]
    
    def __setitem__ (self, item, val):
        self. __dict__ ['columns'][item] = val
        
    def __getattr__ (self, item):
        if item in self. __dict__ ['columns']:
            return self. __dict__ ['columns'][item]
        elif item == 'columns':
            return self. __dict__ [item]
        else:
            raise AttributeError
    
    def __setattr__ (self, item, val):
        if item != 'columns':
            self. __dict__ ['columns'][item] = val
        else:
            raise ValueError("Overwriting columns prohibited") 

            
f = StupidFrameAttr({'a': 1, 'b': 2, 'c': 3})
print("StupidFrameAttr value for a", f['a'])
print("StupidFrameAttr columns: ", f.columns)
del f['b']
print("StupidFrameAttr columns: ", f.columns)
print("StupidFrameAttr value for a", f.a)
f.d = 4
print("StupidFrameAttr columns: ", f.columns)
del f['d']
print("StupidFrameAttr columns: ", f.columns)
f.d = 5
print("StupidFrameAttr columns: ", f.columns)
del f.d
```
```
StupidFrameAttr value for a 1
StupidFrameAttr columns: {'a': 1, 'b': 2, 'c': 3}
StupidFrameAttr columns: {'a': 1, 'c': 3}
StupidFrameAttr value for a 1
StupidFrameAttr columns: {'a': 1, 'c': 3, 'd': 4}
StupidFrameAttr columns: {'a': 1, 'c': 3}
StupidFrameAttr columns: {'a': 1, 'c': 3, 'd': 5}
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-12-fd29f59ea01e> in <module>
     39 f.d = 5
     40 print("StupidFrameAttr columns: ", f.columns)
---> 41 del f.d

AttributeError: d
```

## How could we handle deletion?

Everything works but deletion using attribute access. We handle setting/getting columns using both the array index operator (`[]`) and attribute access. But what about detecting deletion? Is that possible?

One way to do this is using the ` __delattr__ ` method, which is described in the [data model](https://docs.python.org/3.8/reference/datamodel.html) documentation. If you define this method in your class, it will be invoked instead of updating an instance’s attribute dictionary directly. This gives us a chance to redirect this to our columns instance.

```
class StupidFrameDelAttr(StupidFrameAttr):
    def __delattr__ (self, item):
        # trivial implementation using the data model methods
        del self. __dict__ ['columns'][item]

f = StupidFrameDelAttr({'a': 1, 'b': 2, 'c': 3})
print("StupidFrameDelAttr value for a", f['a'])
print("StupidFrameDelAttr columns: ", f.columns)
del f['b']
print("StupidFrameDelAttr columns: ", f.columns)
print("StupidFrameDelAttr value for a", f.a)
f.d = 4
print("StupidFrameDelAttr columns: ", f.columns)
del f.d 
print("StupidFrameDelAttr columns: ", f.columns)

```

```
StupidFrameDelAttr value for a 1
StupidFrameDelAttr columns: {'a': 1, 'b': 2, 'c': 3}
StupidFrameDelAttr columns: {'a': 1, 'c': 3}
StupidFrameDelAttr value for a 1
StupidFrameDelAttr columns: {'a': 1, 'c': 3, 'd': 4}
StupidFrameDelAttr columns: {'a': 1, 'c': 3}
```

Now I’m not suggesting that attribute deletion for columns would be easy to add to pandas, but at least this shows how it could be possible. In the case of current pandas, deleting columns is best done using `drop`.

Also, it’s worth mentioning here that when you create a new column in pandas, you don’t assign it as an attribute. To better understand how to properly create a column, you can check out [this article](https://www.wrighters.io/basic-pandas-how-to-add-a-column-to-a-dataframe/).

If you already knew how to drop a column in pandas, hopefully you understand a little bit more about how this works.

The post [How to remove a column from a DataFrame, with some extra detail](https://www.wrighters.io/how-to-remove-a-column-from-a-dataframe/) appeared first on [wrighters.io](https://www.wrighters.io).