Python Weekly #12 - Decorators 101

Decorators

Decorators 101 - how they work, and how to write them.

For someone new to Python, Decorators are one of the more baffling of the language features. In this post, I will explore why one might need decorators, and how to write them.
Simply put a decorator is a way to extend the functionality of a function or a method without changing the internal implementation of the function or crucially without changing the calling signature.

Examples of when to use decorators include :

• To add logging to multiple functions
• To add common code safety checks to multiple functions
• To transform function arguments or return values

Calling Signature

The calling signature is the list of arguments that you pass to the function, and the values that the function returns. So long as the calling signature is unchanged, other code can continue to call the function as normal, as if they decorator hadn't been applied.

The Basic Principles

In python everything is an 'object' - this includes numbers, strings, and crucially here, functions. Since functions are objects in Python they can be passed as arguments to other functions, and they can be returned by other functions. Objects are `bound` to names - so this code

>>> spam = 'Spam and Eggs.'

Creates a string object - and binds it to the name spam. Names given to functions are no different. A function is an object, which is bound to a name. This means that we can do some interesting things with functions :

• We can rename them, by assigning a different name
• We can pass them as arguments to other functions
• We can store them in lists, dictionaries or anywhere else
• We can call the function from anywhere we have the function object, so long as we honour the calling signature. It is irrelevant what name we currently use, or if the function has a name at all.  
• We can return functions from functions

It is these properties that allow us to write decorators.

Functions as arguments

>>> def greeting():
...    print 'Hello from John, Paul, George & Ringo'

>>> def logme( function ):
...    print 'INFO: Calling {} function'.format( function.__name__)
...    return function()

>>> logme( greeting )
INFO : Calling greeting function
Hello from John, Paul, George & Ringo

We are not quite at a decorator yet - but we have been able to pass a function (the greeting function) as an argument into the logme function, and then call it from the logme function. You will notice that the logme function does not use greeting by name - but it uses it's own function argument.

Although this works, it isn't very useful, as we have to remember to call the logme function, and pass the greeting function as argument, each time we want this function call to be logged, and if the greeting function had it's own arguments we would need a different version of 'logme', it would get messy to call greeting each time. Finally it is no longer obvious from the code logme( greeting ), that we are even calling the greeting function at all - all in all it makes the code more difficult to write and to read.

Before we get to a full decorator, we have one more python feature to explore - nested functions :

Nested Functions

>>> def outer( arg):
...    def inner( arg2):
...        print  'inner called with {}'.format(arg2)
...        return arg*arg2 + 1
...    print  'outer called with {}'.format(arg)
...    return inner( arg  + 1) 
...
>>> outer( 5 )
outer called with 5
inner called with 6
31
>>> inner( 6 )
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'inner' is not defined

There are a few things to note here :

• The inner function has access to the arguments passed to the outer function. If you try it you will find that inner can change the value of arg, but that change is not visible in the outer function, even after it calls inner.
• The inner function can only be accessed inside the outer function. As the error message shows inner can't be called from elsewhere.

Instead of having outer return the result of calling inner, what if outer just returned the inner function itself without calling it:

>>> def outer( arg):
...    def inner( arg2):
...        print  'inner called with {}'.format(arg2)
...        return arg * arg2 + 1
...    print  'outer called with {}'.format(arg)
...    return inner
...
>>> five = outer( 5 )
outer called with 5
>>> six = outer( 6 )
outer called with 6

Now the outer function doesn't return a value - it returns the inner function, and the inner function is never called (well not yet). Strictly speaking the outer function returns a function object. In our code above we call outer twice, and bind the results to the names five and six respectively. We know that the original inner object was expecting a single argument (arg2), and we know that five and six are bound to a function object so - we should be able to call them :

>>>five(2)
11
>>>five(3)
16
>>>six(2)
13
>>>six(3)
19

We can see that calling five and six generate different results - and working through the code, you can see that the five function behaves as we would expect if arg is 5, and five function behaves as we would expect if arg is 6. You will find this to always be true - when you define nested functions, and the inner function uses arguments or variables from the outer function, then the inner function will be locked to the values as they are when the inner function is defined. So lets glue this all together - and build a decorator

Our first decorator

>>> def logme( func ):
...     def wrapper( *args, **kwargs ):
...         print 'INFO : calling {}( {}, {} )'.format( function.__name__, args, kwargs)
...         return func( *args, **kwargs )
...     return wrapper
...
>>> def greeting( name ):
...     print 'Hello {}'.format( name )
...
>>> def goodbye( name ):
...     print 'Goodbye {}'.format( name )
...
>>> greeting( 'Tony' )
Hello Tony
>>> goodbye( 'Tony' )
Goodbye Tony
>>> greeting = logme( greeting)
>>> goodbye = logme( goodbye)
>>> greeting( 'Tony' )
INFO : calling greeting( (Tony,), {})
Hello Tony
>>> goodbye( 'Tony' )
INFO : calling goodbye( (Tony,), {})
Goodbye Tony

If you aren't sure - work through the logme function. Notice that it will return the wrapper function, which in turn uses the func argument which was passed to logme. If you haven't seen the *args and **kwargs notation - this is argument unpacking. You can see at the bottom of the code snippet that we redefine the names greeting and goodbye (remember function names are simply names, and they can be used and reused as we wish). We now have new functions which have added functionality, and we can add this functionality very easily to any function we wish. This ability is so useful, there is a very simple syntax which removes the need for use to redefine the function names :

>>> def logme( func ):
...     def wrapper( *args, **kwargs ):
...         print 'INFO : calling {}( {}, {} )'.format( function.__name__, args, kwargs)
...         return func( *args, **kwargs )
...     return wrapper
...
>>> @logme
>>> def greeting( name ):
...     print 'Hello {}'.format( name )
...
>>> @logme
>>> def goodbye( name ):
...     print 'Goodbye {}'.format( name )
...
>>> greeting( 'Tony' )
INFO : calling greeting( (Tony,), {})
Hello Tony
>>> goodbye( 'Tony' )
INFO : calling goodbye( (Tony,), {})
Goodbye Tony

The @logme line before each function is a special syntax which causes the function to be wrapped in the decorator (in this case the 'logme' decorator), and saves you having to redefine the function name each time. Just remember that

>>> @logme
>>> def greeting( name ):
...     pass 
...
... # The code does EXACTLY the same as the code below 
...
>>> def greeting( name ):
...     pass 
...
>>> greeting = logme(greeting)

You should now have a better understanding of how to write simple decorators, and how they work. The key rules for these simple decorators are :

1. The Outer function takes one argument - that is the function which will be decorated
2. The name of the outer function is the name of the decorator
3. The Outer function is called whenever the decorator is used - i.e. whenever the `@logme' is used in the above example.
4. The Outer function should only return the inner function. 
5. The inner function takes the same arguments as the function being decorated (or more usefully *args & **kwargs)
6. In general it is the inner function which implements the new functionality (either before or after calling the function)
7. The Inner function can return anything at all, but ideally it should return the same type as the function being decorated (or raise an exception) - remember the calling signature. 
8. The inner function is called every time the decorated function is called.

There will be other posts in this Decorator series - watch this space.