Dead Simple Python: Lambdas, Decorators, and Other Magic
Tags: python, beginners, functional
Published: 2019-08-07 19:19:36

---
title: Dead Simple Python: Lambdas, Decorators, and Other Magic
published: true
description: Python has a reputation for looking like magic, and that's probably due in part to the many forms a function can take.
tags: python, beginners, fp
cover_image: https://thepracticaldev.s3.amazonaws.com/i/c0yv1294zhfs06j2w47o.jpg
series: Dead Simple Python
---

Like the articles? Buy the book! [***Dead Simple Python*** by Jason C. McDonald is available from No Starch Press.](https://nostarch.com/dead-simple-python)

---

Python has a reputation for looking like magic, and that's probably due in part to the many forms a function can take: lambdas, decorators, closures, and more. A well-placed function call can do amazing things, without ever writing a single class!

You might say Functions Are Magic.

# Functions Revisited

We already touched on functions in [Data Typing and Immutability](https://dev.to/codemouse92/dead-simple-python-data-typing-and-immutability-41dm). If you haven't read that article yet, I'd recommend going back and taking a look now.

Let's look at a quick example of a function, just to make sure we're on the same page.

```python
def cheer(volume=None):
    if volume is None:
        print("yay.")
    elif volume == "Louder!":
        print("yay!")
    elif volume == "LOUDER!":
        print("*deep breath*")
        print("yay!")


cheer()  # prints "yay."
cheer("Louder!")  # prints "yay!"
cheer("LOUDER!")  # prints "*deep breath* ...yay!"
```

Nothing surprising here. The function `cheer()` accepts a single parameter `volume`. If we don't pass an argument for `volume`, it will default to the value `None` instead.

# What Is Functional Programming?

Those of us coming from object-oriented programming languages have learned to think of everything in terms of classes and objects. Data is organized into objects, alongside the functions responsible for accessing and modifying that data. Sometimes this works, but other times, classes just start feeling like too much boilerplate.

**Functional programming** is almost the opposite of this. We organize around *functions*, which we pass our data through. There are a few rules we have to follow:

* Functions should take only input, and produce only output.

* Functions should not have side effects; they should not modify anything external to themselves.

* Functions should (ideally) always produce the same output for the same inputs. There should be no state internal to the function that will break this pattern.

Now, before you go and rewrite all your Python code to be pure functional, ***STOP!*** Don't forget that one of the beautiful things about Python is that it's a *multi-paradigm language*. You don't have to choose one paradigm and stick to it; you can mix and match the best of each in your code.

In fact, we've already been doing this! Iterators and generators are both borrowed from functional programming, and they work quite well alongside objects. Feel free to incorporate lambdas, decorators, and closures too. It's all about choosing the best tool for the job.

In practice, we rarely can avoid side-effects all together. In applying the concept of functional programming to your Python code, you should focus more on being mindful and deliberate about side-effects, rather than just to avoid them altogether. Limit them to those situations where there is no better way to solve the problem. There's no hard-and-fast rule to rely on here; you will need to develop your own discernment.

# Recursion

When a function calls itself, that's called **recursion.** This can be helpful when we need to repeat the entire logic of a function, but a loop is unsuitable (or just feels too cluttered).

**NOTE:** My example below is simplified to highlight recusion itself. This wouldn't actually be a situation where recursion would be the best approach; recusion is better when you need to repeatedly call complicated language on different pieces of data, such as when you're traversing a tree structure.


```python
import random
random.seed()


class Villain:

    def __init__(self):
        self.defeated = False

    def confront(self):
        # Roll of the dice.
        if random.randint(0,10) == 10:
            self.defeated = True


def defeat(villain):
    villain.confront()
    if villain.defeated:
        print("Yay!")
        return True
    else:
        print("Keep trying...")
        return defeat(villain)


starlight = Villain()
victory = defeat(starlight)

if victory:
    print("YAY!")
``` 

The stuff related to `random` may look new. It's not really related to this topic, but in brief, we can generate random integers by first seeding the random number generator at the start of our program (`random.seed()`), and then calling `random.randint(min, max)`, where `min` and `max` define the inclusive range of possible values.

The important part of the logic here is the `defeat()` function. As long as the villain is not defeated, the function calls itself, passing the `villain` variable. This will happen until one of the function calls returns a value. In this case, the value is returned up the recursive call stack, eventually getting stored in `victory`.

No matter how long it takes`*`, we'll eventually defeat that villain.

## Beware Recursing Infinitely

Recursion can be a powerful tool, but it can also present a problem: *what if we have no way to stop?*

```python
def mirror_pool(lookers):
    reflections = []
    for looker in lookers:
        reflections.append(looker)
    lookers.append(reflections)

    print(f"We have {len(lookers) - 1} duplicates.")

    return mirror_pool(lookers)


duplicates = mirror_pool(["Pinkie Pie"])
```

Clearly, this is going to run forever! Some languages don't provide a clean way to handle this — the function will just recurse infinitely until something crashes.

Python stops this madness a little more gracefully. As soon as it reaches a set recursion depth (usually 997-1000 times), it stops the entire program and raises an error:

> RecursionError: maximum recursion depth exceeded while calling a Python object

Like all errors, we can catch this before things get out of hand:

```python
try:
    duplicates = mirror_pool(["Pinkie Pie"])
except RecursionError:
    print("Time to watch paint dry.")
```

Thankfully, because of how I wrote this code, I don't actually need to do anything special to clean up the 997 duplicates. The recursive function never returned, so `duplicates` remains undefined in this case.

We might want to control recursion another way, however, so we don't have to use a `try-except` to prevent disaster. Within our recursive function, we can keep track of how many times it's been called by adding a `calls` parameter, and aborting as soon as it gets too big.

```python
def mirror_pool(lookers, calls=0):
    calls += 1

    reflections = []
    for looker in lookers:
        reflections.append(looker)
    lookers.append(reflections)

    print(f"We have {len(lookers) - 1} duplicates.")

    if calls < 20:
        lookers = mirror_pool(lookers, calls)

    return lookers


duplicates = mirror_pool(["Pinkie Pie"])
print(f"Grand total: {len(duplicates)} Pinkie Pies!")
```

We still have to figure out how to get rid of 20 duplicates without losing the original, but at least the program didn't crash.

**NOTE:** You can override the maximum recursion level with `sys.setrecursionlimit(n)`, where `n` is the maximum you want.

# Nested Functions

From time to time, we may have a piece of logic which we want to reuse *within* a function, but we don't want to clutter up our code by making yet another function besides.

```python
def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element, target):
        print(f"Using Element of {element} on {target}.")

    for element in elements:
        use(element, target)


use_elements("Nightmare Moon")
```

Of course, the trouble with an example this simple is that the *usefulness* is not immediately apparent. Nested functions become helpful when we have a large chunk of logic that we want to abstract down to a function for reuability, but we don't want to define outside of our principal function. If the `use()` function were considerably more complicated, and perhaps called from more than just the loop, this design would be justified.

Still, the simplicity of the example shows the underlying concept. It also brings up another difficulty. You will notice that we're passing `target` to the inner function, `use()`, each time we call it, and that feels rather pointless. Couldn't we just use the `target` variable that's already in local scope?

In fact, we could.

```python
def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element):
        print(f"Using Element of {element} on {target}.")

    for element in elements:
        use(element)


use_elements("Nightmare Moon")
```

Yet as soon as we try to *modify* that variable, we run into trouble:

```python
def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element):
        print(f"Using Element of {element} on {target}.")
        target = "Luna"

    for element in elements:
        use(element)

    print(target)


use_elements("Nightmare Moon")
```

Running that code raises an error:

> UnboundLocalError: local variable 'target' referenced before assignment

Clearly, it is no longer seeing our local variable `target`. This is because assigning to a name, by default, *shadows* any existing names in enclosing scopes. So, the line `target == "Luna"` is trying to create a new variable limited to the scope of `use()`, and that hides (shadows) the variable `target` in the enclosing scope of `use_elements()`. Python sees this and assumes that, since we're defining `target` in the function `use()`, all references to that variable relate to that local name. That's not what we want!

The `nonlocal` keyword allows us to tell the inner function that we're working with the variable `target` from the *enclosing local scope*.

```python
def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element):
        nonlocal target
        print(f"Using Element of {element} on {target}.")
        target = "Luna"

    for element in elements:
        use(element)
    
    print(target)


use_elements("Nightmare Moon")
```

Now, when all is said and done, we see the value `Luna` printed out. Our work here is done!

**NOTE:** If you're wanting to allow a function to be able to modify a variable that was defined the *global* scope (outside of all functions), use the `global` keyword instead of `nonlocal`.

# Closures

Building on the idea of the nested function, and recalling that a function is treated no differently than any other object, we can create a function that actually builds and returns another function, called a **closure**.

```python
def harvester(pony):
    total_trees = 0

    def applebucking(trees):
        nonlocal pony, total_trees
        total_trees += trees
        print(f"{pony} harvested from {total_trees} trees so far.")

    return applebucking


apple_jack = harvester("Apple Jack")
big_mac = harvester("Big Macintosh")
apple_bloom = harvester("Apple Bloom")

north_orchard = 120
west_orchard = 80  # watch out for fruit bats
east_orchard = 135
south_orchard = 95
near_house = 20

apple_jack(west_orchard)
big_mac(east_orchard)
apple_bloom(near_house)
big_mac(north_orchard)
apple_jack(south_orchard)
```

In this example, `applebucking()` would be the closure, because it *closes over* the nonlocal variables `pony` and `total_trees`. Even after the outer function terminates, the closure retains references to these variables.

The closure is returned from the `harvester()` function, and can be stored in a variable like any other object. It is the fact it "closes over" a nonlocal variable that makes it a *closure* per se; otherwise, it'd just be a function.

In this example, I'm using the closure to effectively create objects with *state*. In other words, each harvester remembers how many trees he or she has harvested from. This particular usage isn't strictly compliant with functional programming, but it's quite useful if you don't want to create an entire class just to store one function's state!

`apple_jack`, `big_macintosh`, and `apple_bloom` are now three different functions, each with their own separate state; they each have a different name, and remember how many trees they have harvested from. What happens in one closure's state has no effect on the others.

When we run the code, we see this state in action:

```
Apple Jack harvested from 80 trees so far.
Big Macintosh harvested from 135 trees so far.
Apple Bloom harvested from 20 trees so far.
Big Macintosh harvested from 255 trees so far.
Apple Jack harvested from 175 trees so far.
```

Easy as apple pie.

## The Problem With Closures

Closures are essentially "implicit classes", because they put functionality and its persistent information (state) in the same object. There are, however, several unique disadvantages to closures:

* You cannot access the "member variables" as it were. In our example, I can never get to the `total_trees` variable on the `apple_jack` closure! I can only use that variable within the context of the closure's own code.

* The state of the closure is entirely opaque. Unless you know how the closure is written, you don't know what information it's keeping track of.

* Because of the previous two points, it is impossible to directly know when a closure has any state *at all*.

When using closures, you need to be prepared to handle these problems, and all the debugging difficulties they introduce. I recommend *only* using them when you need a single function to store a small amount of private state between calls, and only for such a limited period of time in the code that writing an entire class doesn't feel justified. (Also, don't forget about generators and coroutines, which may be better suited to many such scenarios.)

Basically, it's *there*. Closures can still be a useful part of your Python repitoire, so long as you use them with great care.

# Lambdas

A **lambda** is an anonymous function (no name) made up of a single expression.

That definition alone is the reason many programmers can't imagine why they'd ever need one. What's the point of writing a function that lacks a name, basically making reuse completely impractical? Sure, you can *assign* a lambda to a variable, but at that point, shouldn't you have just written a function?

To understand this, let's take a look at an example *without* lambdas first:

```python
class Element:

    def __init__(self, element, color, pony):
        self.element = element
        self.color = color
        self.pony = pony

    def __repr__(self):
        return f"Element of {self.element} ({self.color}) is attuned to {self.pony}"


elements = [
    Element("Honesty", "Orange", "Apple Jack"),
    Element("Kindness", "Pink", "Fluttershy"),
    Element("Laughter", "Blue", "Pinkie Pie"),
    Element("Generosity", "Violet", "Rarity"),
    Element("Loyalty", "Red", "Rainbow Dash"),
    Element("Magic", "Purple", "Twilight Sparkle")
]


def sort_by_color(element):
    return element.color


elements = sorted(elements, key=sort_by_color)
print(elements)

```

The main thing I want you to notice is the `sort_by_color()` function, which I had to write for the express purpose of sorting the Element objects in the list by their color. This is a bit of an annoyance, actually, since I won't be needing that function ever again.

Here's where lambdas come in. I can drop that entire function, and change the `elements = sorted(...)` line to:

```python
elements = sorted(elements, key=lambda e: e.color)
```

Using a lambda allows me to delineate my logic *exactly where I'm using it*, and nowhere else. (The `key=` part is just indicating that I'm passing the lambda to the `key` parameter on `sorted()`.)

A lambda has the structure `lamba <parameters>: <return expression>`. It can collect as many paramters as it likes, separated by commas, but it can only have *one* expression, the value of which is implicitly returned.

**GOTCHA ALERT:** Lambdas do not support type annotations (type hinting), unlike regular functions.

If I wanted to rewrite that lambda to sort by the name of the Element, instead of the color, I only need to change the expression part:

```python
elements = sorted(elements, key=lambda e: e.name)
```

It's as simple as that.

Again, lambdas are chiefly useful whenever you need to pass a function with a single expression to another function. Here's another example, this time with more parameters on the lambda.

To set up this example, let's start with a class for a Flyer, which stores the name and the maximum speed, and returns a random speed for the flyer.

```python
import random
random.seed()


class Flyer:

    def __init__(self, name, top_speed):
        self.name = name
        self.top_speed = top_speed

    def get_speed(self):
        return random.randint(self.top_speed//2, self.top_speed)
```

We want to be able to make any given Flyer object perform any flying trick, but putting all that logic into the class itself would be impractical...there are perhaps thousands of flying tricks and variants!

Lambdas are one way to define these tricks. We'll start by adding a function to this class that can accept a function as a parameter. We'll make the assumption that this function always takes a single argument: the speed at which the trick is performed.

```python
    def perform(self, trick):
        performed = trick(self.get_speed())
        print(f"{self.name} perfomed a {performed}")
```

To use this, we create a Flyer object, and then pass functions to its `perform()` method.

```python
rd = Flyer("Rainbow Dash", 780)
rd.perform(lambda s: f"barrel-roll at {s} mph.")
rd.perform(lambda s: f"flip at {s} mph.")
```

Because the lambda's logic is *in the function call*, it is a lot easier to see what's going on.

Recall that you *are* allowed to store lambdas in a variable. This actually can be helpful when you want the code to be this brief, but need some reuability. For example, assume we have another Flyer, and we want both of them to perform a barrel-roll.

```python
spitfire = Flyer("Spitfire", 650)
barrelroll = lambda s: f"barrel-roll at {s} mph."

spitfire.perform(barrelroll)
rd.perform(barrelroll)
```

Sure, we could have written `barrelroll` as a proper single-line function, but by doing it this way, we saved ourselves a little bit of boilerplate. And, since we won't be using the logic again after this section of code, there's no point in having a full-blown function hanging out.

Once again, **readability matters**. Lambdas are excellent for short, clear fragments of logic, but if you have anything more complicated, you should *definitely* write a proper function.

# Decorators

Imagine we want to modify the behavior of any function, without actually changing the function itself.

Let's start with a reasonably basic function:

```python
def partial_transfiguration(target, combine_with):
    result = f"{target}-{combine_with}"
    print(f"Transfiguring {target} into {result}.")
    return result


target = "frog"
target = partial_transfiguration(target, "orange")
print(f"Target is now a {target}.")
```

Running that gives us:

```
Transfiguring frog into frog-orange.
Target is now a frog-orange.
```

Simple enough. But what if we wanted to add some extra fanfare to this? As you know, we really shouldn't put that logic *in* our `partial_transfiguration` function.

This is where **decorators** come in. A decorator "wraps" additional logic around a function, such that we don't actually modify the original function itself. This makes for code that is far more maintainable.

Let's start by creating a decorator for the fanfare. The syntax here might look a little overwhelming at first, but rest assured I'll break this down.

```python
import functools


def party_cannon(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print("Waaaaaaaait for it...")
        r = func(*args, **kwargs)
        print("YAAY! *Party cannon explosion*")
        return r

    return wrapper
```

You may have already recognized that `wrapper()` is actually a *closure*, which is created and returned from our `party_cannon()` function. We pass the function we're "wrapping", `func`.

However, we really don't know anything about the function we're wrapping! It may or may not have arguments. The closure's parameter list `(*args, **kwargs)` can accept literally any number of arguments, from zero to (practically) infinity. We pass these arguments in the same way to `func()` when we call it.

Of course, if there's some sort of mismatch between the parameter list on `func()` and the arguments passed to it *through* the decorator, the usual and expected error will be raised (which is obviously a good thing.)

Within `wrapper()`, we are calling our function whenever and however we want to with `func()`. I chose to do so between printing my two messages.

I don't want to throw away the value that `func()` returns, so I assign that returned value to `r`, and I make sure to return it at the end of my wrapper with `return r`.

Mind you, there are really no hard-and-fast rules for the manner in which you call the function in the wrapper, or even *if* or *how many times* you call it. You can also handle the arguments and return values in whatever manner you see fit. The point is to make sure the wrapper doesn't actually *break* the function it wraps in some unexpected way.

The odd little line just before the wrapper, `@functools.wraps(func)`, is actually a decorator itself. Without it, the function being wrapped would essentially get confused as to its own identity, messing up our external access of such important function attributes as `__doc__` (the docstring) and `__name__`. This special decorator ensures that doesn't happen; the function being wrapped retains its own identity, which is accessible from outside of the function in all the usual ways. (To use that special decorator, we had to `import functools` first.)

Now that we have our `party_cannon` decorator written, we can use it to add that fanfare we wanted to the `partial_transfiguration()` function. Doing so is as simple as this:

```python
@party_cannon
def partial_transfiguration(target, combine_with):
    result = f"{target}-{combine_with}"
    print(f"Transfiguring {target} into {result}.")
    return result
```

That first line, `@party_cannon` is the only change we had to make! The `partial_transfiguration` function is now *decorated*.

**NOTE:** You can even *stack* multiple decorators on top of each other, one above the next. Just make sure each decorator comes *immediately* before the function or decorator it is wrapping.

Our usage from before hasn't changed at all:

```python
target = "frog"
target = partial_transfiguration(target, "orange")
print(f"Target is now a {target}.")
```

Yet the output has indeed changed:

```
Waaaaaaaait for it...
Transfiguring frog into frog-orange.
YAAY! *Party cannon explosion*
Target is now a frog-orange.
```

# Review

We've covered four aspects of functional "magic" in Python. Let's take a moment to recap.

* **Recursion** is when a function calls itself. Beware "infinite recursion"; Python won't let a recursion stack get more than approximate a thousand recursive calls deep.

* A **nested function** is a function defined in another function.

* A nested function can *read* the variables in its enclosing scope, but it cannot modify them unless you specify the variable as `nonlocal` first in the nested function.

* A **closure** is a nested function that *closes over* one or more nonlocal variables, and then is returned by the enclosing function.

* A **lambda** is an anonymous (unnamed) function made up of a single expression, the value of which is returned. Lambdas can be passed around and assigned to variables like any other object.

* **Decorators** "wrap around" another function to extend its behavior, without the function you're wrapping having to be directly modified.

You can read the documentation for more information about these topics. (You'll actually notice that nested functions and closures are rarely mentioned in the official documentation; they're *design patterns*, rather than formally defined language structures.)

+ [Python Reference: Compound statements — Function definitions](https://docs.python.org/3/reference/compound_stmts.html#function-definitions)

+ [Python Reference: Expressions — Lambda](https://docs.python.org/3/reference/expressions.html#lambda)

+ [PEP 318: Decorators for Functions and Methods](https://www.python.org/dev/peps/pep-0318/)

+ [Python Standard Library: functools](https://docs.python.org/3/library/functools.html)

---

*Thanks to @deniska, @asdf, @SnoopDeJi (Freenode IRC), and [@sandodargo](https://dev.to/sandordargo) (DEV) for suggested improvements.*