Python Decorators

In August 2009, I wrote a post titled Introduction to Python Decorators. It was an attempt to explain Python decorators in a way that I (and I hoped, others) could grok.

Recently I had occasion to re-read that post. It wasn’t a pleasant experience — it was pretty clear to me that the attempt had failed.

That failure — and two other things — have prompted me to try again.

  • Matt Harrison has published an excellent e-book Guide to: Learning Python Decorators.
  • I now have a theory about why most explanations of decorators (mine included) fail, and some ideas about how better to structure an introduction to decorators.

There is an old saying to the effect that “Every stick has two ends, one by which it may be picked up, and one by which it may not.” I believe that most explanations of decorators fail because they pick up the stick by the wrong end.

In this post I will show you what the wrong end of the stick looks like, and point out why I think it is wrong. And I will show you what I think the right end of the stick looks like.

 

The wrong way to explain decorators

Most explanations of Python decorators start with an example of a function to be decorated, like this:

def aFunction():
    print("inside aFunction")

and then add a decoration line, which starts with an @ sign:

@myDecorator
def aFunction():
    print("inside aFunction")

At this point, the author of the introduction often defines a decorator as the line of code that begins with the “@”. (In my older post, I called such lines “annotation” lines. I now prefer the term “decoration” line.)

For instance, in 2008 Bruce Eckel wrote on his Artima blog

A function decorator is applied to a function definition by placing it on the line before that function definition begins.

and in 2004, Phillip Eby wrote in an article in Dr. Dobb’s Journal

Decorators may appear before any function definition…. You can even stack multiple decorators on the same function definition, one per line.

Now there are two things wrong with this approach to explaining decorators. The first is that the explanation begins in the wrong place. It starts with an example of a function to be decorated and an decoration line, when it should begin with the decorator itself. The explanation should end, not start, with the decorated function and the decoration line. The decoration line is, after all, merely syntactic sugar — is not at all an essential element in the concept of a decorator.

The second is that the term “decorator” is used incorrectly (or ambiguously) to refer both to the decorator and to the decoration line. For example, in his Dr. Dobb’s Journal article, after using the term “decorator” to refer to the decoration line, Phillip Eby goes on to define a “decorator” as a callable object.

But before you can do that, you first need to have some decorators to stack. A decorator is a callable object (like a function) that accepts one argument—the function being decorated.

So… it would seem that a decorator is both a callable object (like a function) and a single line of code that can appear before the line of code that begins a function definition. This is sort of like saying that an “address” is both a building (or apartment) at a specific location and a set of lines (written in pencil or ink) on the front of a mailing envelope. The ambiguity may be almost invisible to someone familiar with decorators, but it is very confusing for a reader who is trying to learn about decorators from the ground up.

 

The right way to explain decorators

So how should we explain decorators?

Well, we start with the decorator, not the function to be decorated.

One
We start with the basic notion of a function — a function is something that generates a value based on the values of its arguments.

Two
We note that in Python, functions are first-class objects, so they can be passed around like other values (strings, integers, objects, etc.).

Three
We note that because functions are first-class objects in Python, we can write functions that both (a) accept function objects as argument values, and (b) return function objects as return values. For example, here is a function foobar that accepts a function object original_function as an argument and returns a function object new_function as a result.

def foobar(original_function):

    # make a new function
    def new_function():
        # some code

    return new_function

Four
We define “decorator”.

A decorator is a function (such as foobar in the above example) that takes a function object as an argument, and returns a function object as a return value.

So there we have it — the definition of a decorator. Anything else that we say about decorators is a refinement of, or an expansion of, or an addition to, this definition of a decorator.

Five
We show what the internals of a decorator look like. Specifically, we show different ways that a decorator can use the original_function in the creation of the new_function. Here is a simple example.

def verbose(original_function):

    # make a new function that prints a message when original_function starts and finishes
    def new_function(*args, **kwargs):
        print("Entering", original_function.__name__)
        original_function(*args, **kwargs)
        print("Exiting ", original_function.__name__)

    return new_function

Six
We show how to invoke a decorator — how we can pass into a decorator one function object (its input) and get back from it a different function object (its output). In the following example, we pass the widget_func function object to the verbose decorator, and we get back a new function object to which we assign the name talkative_widget_func.

def widget_func():
    # some code

talkative_widget_func = verbose(widget_func)

Seven
We point out that decorators are often used to add features to the original_function. Or more precisely, decorators are often used to create a new_function that does roughly what original_function does, but also does things in addition to what original_function does.

And we note that the output of a decorator is typically used to replace the original function that we passed in to the decorator as an argument. A typical use of decorators looks like this. (Note the change to line 4 from the previous example.)

def widget_func():
    # some code

widget_func = verbose(widget_func)

So for all practical purposes, in a typical use of a decorator we pass a function (widget_func) through a decorator (verbose) and get back an enhanced (or souped-up, or “decorated”) version of the function.

Eight
We introduce Python’s “decoration syntax” that uses the “@” to create decoration lines. This feature is basically syntactic sugar that makes it possible to re-write our last example this way:

@verbose
def widget_func():
    # some code

The result of this example is exactly the same as the previous example — after it executes, we have a widget_func that has all of the functionality of the original widget_func, plus the functionality that was added by the verbose decorator.

Note that in this way of explaining decorators, the “@” and decoration syntax is one of the last things that we introduce, not one of the first.

And we absolutely do not refer to line 1 as a “decorator”. We might refer to line 1 as, say, a “decorator invocation line” or a “decoration line” or simply a “decoration”… whatever. But line 1 is not a “decorator”.

Line 1 is a line of code. A decorator is a function — a different animal altogether.

 

Nine
Once we’ve nailed down these basics, there are a few advanced features to be covered.

  • We explain that a decorator need not be a function (it can be any sort of callable, e.g. a class).
  • We explain how decorators can be nested within other decorators.
  • We explain how decorators decoration lines can be “stacked”. A better way to put it would be: we explain how decorators can be “chained”.
  • We explain how additional arguments can be passed to decorators, and how decorators can use them.

Ten — A decorators cookbook

The material that we’ve covered up to this point is what any basic introduction to Python decorators would cover. But a Python programmer needs something more in order to be productive with decorators. He (or she) needs a catalog of recipes, patterns, examples, and commentary that describes / shows / explains when and how decorators can be used to accomplish specific tasks. (Ideally, such a catalog would also include examples and warnings about decorator gotchas and anti-patterns.) Such a catalog might be called “Python Decorator Cookbook” or perhaps “Python Decorator Patterns”.



So that’s it. I’ve described what I think is wrong (well, let’s say suboptimal) about most introductions to decorators. And I’ve sketched out what I think is a better way to structure an introduction to decorators.

Now I can explain why I like Matt Harrison’s e-book Guide to: Learning Python Decorators. Matt’s introduction is structured in the way that I think an introduction to decorators should be structured. It picks up the stick by the proper end.

The first two-thirds of the Guide hardly talk about decorators at all. Instead, Matt begins with a thorough discussion of how Python functions work. By the time the discussion gets to decorators, we have been given a strong understanding of the internal mechanics of functions. And since most decorators are functions (remember our definition of decorator), at that point it is relatively easy for Matt to explain the internal mechanics of decorators.

Which is just as it should be.


Revised 2012-11-26 — replaced the word “annotation” with “decoration”, following terminology ideas discussed in the comments.


					

Introduction to Python Decorators

This post was originally written in August 2009. Since then, I have come to believe that there is a much better way to explain Python decorators, which I describe in another post.


Writing introductions to decorators is a popular pastime in the Python community. Here, for example, are some useful links on the subject:

But when it comes to technical topics, everyone has his or her own style of learning and one size of explanation does not fit all.

So I thought I’d try my hand at writing an introduction to Python decorators. My goal is not to explain everything about decorators. Instead I want to try to explain just the basics, just enough to give you a workable mental model of what decorators are and how to use them. Just enough to get started on doing useful work with decorators.

As Aristotle said, “Let us begin at the beginning”, which is to say, we begin by looking at functions.

Functions

When the Python interpreter encounters this code:

def hello():
    print ("Hello, world!")

it:

  • compiles the code to create a function object
  • binds the name “hello” to that function object.

Then, to run the function object, you can code

hello()

which causes this to be printed:

Hello, world!

If you code:

print (hello)

you will get something like:

<function hello at 0x02D021E0>

which is the string representation of the hello function object.
 

Annotations

Many discussions of decorators use the word “decorator” rather loosely, to refer to different decorator-related concepts. This kind of ambiguity is disconcerting at best, and confusing at worst.

To help avoid this ambiguity I will use the term “annotation” in this discussion to refer to lines of code that begin with “@”.

Here is a snippet of Python code that begins with two annotations:

@helloGalaxy
@helloSolarSystem
def hello():
    print ("Hello, world!")

We can say that the definition of the hello function is “decorated” with these two annotations. Since there are multiple annotations, we say that the annotations are “stacked”.

When the interpreter sees these lines of code, here is what it does.

  • It pushes helloGalaxy onto the annotation stack.
  • It pushes helloSolarSystem onto the annotation stack.

then it does the standard processing for a function definition …

  • It compiles the code for hello into a function object (lets call it functionObject1)
  • It binds the name “hello” to functionObject1.

then…

  • It pops helloSolarSystem off of the annotation stack,
  • passes functionObject1 to helloSolarSystem
  • helloSolarSystem returns a new function object (lets call it functionObject2), and…
  • the interpreter binds the original name “hello” to functionObject2

then…

  • It pops helloGalaxy off of the annotation stack,
  • passes functionObject2 to helloGalaxy
  • helloGalaxy returns a new function object (lets call it functionObject3), and…
  • the interpreter binds the original name “hello” to functionObject3

As you can see, this process could be repeated for indefinitely many annotations.

I’ve been vague about what kind of thing that helloSolarSystem and helloGalaxy are. For now, think of them as a special kind of function — a kind of function that takes one function object as an argument, and returns another function object as a result. The annotations:

@helloGalaxy
@helloSolarSystem

are calls to these functions. So this snippet of Python code:

@helloGalaxy
@helloSolarSystem
def hello():
    print ("Hello, world!")

is functionally equivalent to this:

def hello():
    print ("Hello, world!")
hello = helloSolarSystem(hello)
hello = helloGalaxy(hello)

Decorators

Now we are ready to define “decorator”.

A decorator is a function that is called by an annotation.

Where where do decorators come from?

You write them, just the way that you write other function definitions.

So let’s write some decorators. Here is helloSolarSystem.

def helloSolarSystem(original_function):
    def new_function():
        original_function()  # the () after "original_function" causes original_function to be called
        print("Hello, solar system!")
    return new_function

And let’s write helloGalaxy.

def helloGalaxy(original_function):
    def new_function():
        original_function()  # the () after "original_function" causes original_function to be called
        print("Hello, galaxy!")
    return new_function

As you can see, both of these decorators add a bit of functionality to the function object — original_function — that they receive as input. They wrap a call to original_function in a new function, new_function, put some additional functionality in new_function, and then they return the new_function object. (They return the new_function object to the annotation, which binds the original name to the new function object.)

So now let’s run our whole program and see what we get. Here’s the program.

def helloSolarSystem(original_function):
    def new_function():
        original_function()  # the () after "original_function" causes original_function to be called
        print("Hello, solar system!")
    return new_function
	
def helloGalaxy(original_function):
    def new_function():
        original_function()  # the parentheses after "original_function" cause original_function to be called
        print("Hello, galaxy!")
    return new_function

@helloGalaxy
@helloSolarSystem
def hello():
    print ("Hello, world!")

# Here is where we actually *do* something!
hello()

And here is what we get:

Hello, world!
Hello, solar system!
Hello, galaxy!

Arguments to functions

Now lets look at decorating functions that take arguments. Let’s modify the hello function so it accepts an argument, like this:

def hello(targetName=None):
    if targetName:
        print("Hello, " +  targetName +"!")
    else:
        print("Hello, world!")

If we were to run an undecorated version of the hello function, we’d get a nice greeting, like this:

>>> hello("Earth")
Hello, Earth!

But if we run the decorated version of the hello function, we get this:

TypeError: new_function() takes no arguments (1 given)

What’s the problem?

Remember that we wrapped functionObject1 (created from hello) in functionObject2 (created from helloSolarSystem) and then in functionObject3 (created from Galaxy), and then bound the name “hello” to functionObject3. So when we use the “hello” function, we are calling functionObject3.

FunctionObject3 was created by the code for new_function in helloGalaxy, and it accepts no arguments. Which is why we get the error message:

TypeError: new_function() takes no arguments (1 given)

The solution is to add support for arguments to the function objects that our decorators create. We need to add code to new_function so that it will accept arguments, and we need to add code to original_function so that it will accept the arguments that its wrapper (new_function) makes available to it.

def helloSolarSystem(original_function):
    def new_function(*args, **kwargs):
        original_function(*args, **kwargs)
        print("Hello, solar system!")
    return new_function

def helloGalaxy(original_function):
    def new_function(*args, **kwargs):
        original_function(*args, **kwargs)
        print("Hello, galaxy!")
    return new_function

And now:

>>>hello("Earth")
Hello, Earth!
Hello, solar system!
Hello, galaxy!