Python Pitfalls - Expecting The Unexpected
Tags: python, webdev, tutorial
Published: 2020-11-16 17:50:45


Regardless of which programming language you're coding in, you've probably encountered good chunk of weird and seemingly unexplainable issues that ended up being really stupid mistakes or quirks of that specific language. Python aims at being clean and simple language, yet it also has its portion of gotchas and quirks that can surprise both beginner and experienced software developers. So, to avoid unnecessary rage and frustration over some weird issue in your favourite programming language, here follows a list of common Python pitfalls, that you should try to avoid at all costs.

## Mutable Default Arguments Are a Bad Idea

Setting default arguments for a function is very common and useful for defining optional arguments or arguments that can usually use same, predefined value. Setting default argument to a mutable value such as `list` or `dict` can, however, cause unexpected behavior:

```python
def some_func(args=[]):
    args.append("data")
    ...
    
some_func()
print(some_func.__defaults__)
# (['data'],)
some_func()
print(some_func.__defaults__)
# (['data', 'data'],)
some_func()
print(some_func.__defaults__)
# (['data', 'data', 'data'],)

# Better solution:
def some_func(args=None):
    if args is None:
        args = []
    ...
```

The problem with using mutable value as default argument is that the default argument in not initialized every time the function is called. Instead, the recently used value will be passed in, which in case of mutable types is a problem. To avoid this problem, you should always use `None` or other immutable type instead, and perform check against the argument as shown above.

Even though this might seem like nuisance and a problem, it's an intended behavior and it can also be _exploited_ to make caching functions which can use the persistent mutable default argument as cache:

```python
def some_func(var, cache={}):
    if var in cache:
        return cache[var]
    # Do stuff...
    cache[var] = result
    return result
```

Similar behavior to the default arguments above, can also be seen with `dict.setdefault(key, value)`. In the below code we can see some surprising results:

```python
data = {}
key = 'some_key'
val = []  # Mutable

data.setdefault(key, val)

print('Before:', data)
# Before: {'some_key': []}
val.append('some_data')
print('After:', data)
# After: {'some_key': ['some_data']}
```

Even though we didn't touch the `data` dictionary above, it was modified by appending to default value `val`. That's because default value passed to `setdefault` is assigned directly into the dictionary when the key is missing instead of being copied from original. To avoid this issue, make sure you never reuse values when using `setdefault`.

## `NaN` (Non-) Reflexivity

Working with floats and non-integer numbers can often be difficult and annoying, but it gets especially weird when you get into _Not-a-Number_ and _Infinity_ territory. So, let's demonstrate this by making a few comparisons with these values:
    
```python
x = float("NaN")  # Define "Not a Number" (the string is case-insensitive), equivalent of math.nan
y = float("inf")  # Define "Infinity"
z = float("inf")  # Define more "Infinity"

x == 10  # False; Makes sense

y == z   # True; Kinda makes sense

x == x   # False... NaN != NaN
```

The above code shows the non-reflexivity of `NaN`. `NaN` in Python will never compare as equal even when compared with itself. So, in case you need to test for `NaN` or `inf`, then you should use `math.isnan()` and `math.isinf()`. Also be careful with any other arithmetic operation when working with code that might produce `NaN`, as it will propagate through all operations without raising an exception.
    
Python is usually clever and won't generally return `NaN` from math functions, e.g. `math.exp(1000.0)` will return `OverflowError: math range error` and both `math.sqrt(-1.0)` and `math.log(0.0)` will return `ValueError: math domain error`, but you might encounter it with _Numpy_ or _Pandas_ and if you do so, remember not to try comparing `NaNs` for equality.

## Late Binding Closures

There are quote a few gotchas, pitfalls and surprises surrounding scopes and closures in Python. The most common one - I'd say - is _late binding_ in closures. Let's start with example:

```python
funcs = []

for i in range(3):
    def some_func(n):
        return i * n
    funcs.append(some_func)

for f in funcs:
    print(f(2))

# 4
# 4
# 4
```

The code above shows definition of function inside a loop which is then added to a list. With each iteration the `i` variable increments and so does the `i` variable in the defined function, right? _Wrong._

_Late binding_ causes all the functions to assume value of 2 (from last iteration). This happens because of the closure in which all the functions are defined in - the global one. Because of this all of them refer to the same `i` which gets mutated in the loop.

There's more than one way to fix this, but the cleanest one in my opinion is to use `functools.partial`:

```python
from functools import partial

funcs = []

for i in range(3):
    def some_func(i, n):
        return i * n
    funcs.append(partial(some_func, i))

for f in funcs:
    print(f(2))

# 0
# 2
# 4
```

With `partial` we can create new callable object with predefined `i`, forcing the variable to be bound _immediately_ which fixes the issue. We can then supply the remaining original parameter `n` when we want to actually call the functions.

## Reassigning Global Variables 

Using a lot of global variables is generally discouraged and viewed as a bad practice. There are however, valid reason to use some global variables - for example to define various _flags_, which can be used to set log level of function.
 
But what if you decide to flip (reassign) this flag? Well, it can cause a massive headache:

```python
flag = False

def some_func():
    flag = True

def some_other_func():
    if flag:
        ...  # Do something when flag is set to True

some_func()
some_other_func()
```

Looking at the code above one might expect the value of global `flag` variable to change to `True` after execution of `some_func()`, but that's not the case. The `some_func` declares _new local_ variable `flag`, sets it to `True` and it then disappears after end of function body. The _global_ variable is never touched.

There's a simple fix to this, though. We need to first declare in the function body that we want to refer to the global variable instead of using local one. We do that with `global <var_name>` - in this case `global flag`:

```python
flag = False

def some_func():
    global flag
    flag = True

...
```

Another _"fun"_ issues with variables that you might run into - which is luckily much easier to debug and fix - is modification of _out-of-scope_ variable. Similarly to previous gotcha, it's caused by manipulating variable that was defined in outer scope:

```python
var = 1
def some_func():
    return var

def another_func():
    var += 1
    return var

print(some_func())
# 1
print(another_func())
# UnboundLocalError: local variable 'var' referenced before assignment
```

Here we try to increment variable `var` inside function scope, assuming that it will modify the global one. But again, that's wrong.

When you modify variable it becomes local to the scope, but you can't increment variable that wasn't declared before (in current scope), so `UnboundLocalError` is thrown.

This again can be fixed using `global <var_name>` in case of global variables. This so-called _scoping bug_ can also occur inside nested functions where you would use `nonlocal <var_name>` instead, to refer to variable in the nearest outer scope:

```python
def outer():
    flag = False     # Scope: 'outer'

    def inner():
        nonlocal flag
        flag = True  # Scope: 'inner'

    inner()
    print(flag)      # True; Did change
    return ...

outer()
```

## Proper Way to Define Tuples

One misconception that pretty much every Python developer has ingrained in their mind, is that tuples are defined by surrounding parenthesis. Unlike iterables like `dict` or `set`, Python `tuple` is defined by the comma separating its elements.

Mistakes originating from this misconception usually arise when we try to define tuple with just single element:

```python
x = ("value")  # Not a tuple.
type(x)
# <class 'str'>

x = ("value",)  # This is a tuple.
type(x)
# <class 'tuple'>

x = "value",  # No need for parenthesis.
type(x)
# <class 'tuple'>

x = ("some" "value")  # Don't forget the comma, otherwise strings will be implicitly concatenated.
print(x)
'somevalue'
```

In the snippet above, we can see that it's necessary to add `,` after the singular element to make Python recognize it as tuple. We can also completely omit parenthesis, which is pretty common practice with `return` statements that return multiple values.

Last example above shows one more similar pitfall. If you forget to separate elements with comma, Python will use implicit concatenation making it a single value of a type string. This kind of implicit concatenation can happen anywhere in the code not just when defining tuple, so always double check your strings and iterables if something fishy is happening with your program.

## Indexing Byte Values Instead of Byte Strings

When working with files and data in them we mostly just use ASCII or UTF-8 strings. From time to time however, you might have to read and write some binary data and you might be surprised with the results of indexing and iterating them: 
    
```python
text = "Some data"

for c in text:
    print(c)

# S
# o
# m
# e
# ...

text = b"Some data"  # Binary String!

for c in text:
    print(c)

# 83
# 111
# 109
# 101
# ...
```

When indexing into binary string, instead of receiving byte string, we get integer byte value, or in other words - ordinal value of the indexed character. To avoid this - especially when reading binary file - it's best to always use `text.decode('utf-8')` to get proper string. If you however want to keep the original data as binary string, then you can instead use `chr(c)` to convert individual characters to string representation.
    
## Indexing with Negated Variable

Slicing and dicing is one of the most handy features of Python including the ability to specify negative indexes, but if you are not careful with those, you might get unexpected results:

```python
x = "Some data"
print(x[-4:])
# data

print(x[-0:])  # Equivalent to x[:]
# Some data
```

If we slice a sequence with any negative value (variable) other than `-0` we will get the expected values, but if we happen to accidentally slice using `[-0:]` we will receive as a result a copy of whole sequence as it is equivalent to `[:]`.
    
## Why Is It Returning None!?

I left my _"favourite"_ gotcha the for last. It's easy to forget whether a function returns new value or modifies original _in-place_. Especially, when there are generally 2 types of methods - _list_ methods which modify the argument and return `None` and _string_ methods which modify the argument _in-place_.

```python
# List methods
some_list.sort()  # Sorts in-place, returns None
some_list = some_list.sort()  # Wrong! -> some_list == None

some_list.reverse()
some_list.extend(["another"])
some_list.clear()
some_list.append("value")

# String methods
some_string = "Some data"

some_string = some_string.split()
some_string = some_string.strip()
some_string = some_string.capitalize()
some_string = some_string.upper()
some_string = some_string.encode()
```

I'm guilty of making this mistake way too many times. It's easy to forget behavior of one of the many string or list methods and it can lead to hours of debugging. So, if you receive `None` where there should be whole string or list, then double check to make sure you're using all of the above shown methods correctly.

## Conclusion

It's inevitable that you will run into these or other similar gotchas and pitfalls that will cause a lot of rage and frustration. More often than not, the best way to solve any of these issues is to just step back for a moment. Go for walk. Go make a cup of coffee. Or at least take a deep breath. Most of the time all it takes to solve such an issue, is to leave it for bit and come back later.

If that doesn't help, maybe it's time for some rubber duck debugging or to bring in another pair of eye (colleague sitting next to you). Oftentimes, when you start explaining the problem to somebody else, you will immediately realise where the problem really is. 

When you eventually find the bug and manage to solve it, take a moment to think about what you could have done to find it faster. Next time you run into similar issue you might be able to resolve it a bit more quickly.