Reading: Python Cookbook, Chapter 9, metaprogramming
Video: Tutorial: Introduction to Decorators Geir Arne Hjelle, PyCon 2018.
A decorator is a function that takes another function as an argument, and returns a new function that modifies or decorates the input function.
First, we load some modules we will use below.
import time
from timeit import *
from functools import wraps
We will write a slow procedure (from Hjelle video). Uses f string formatting (new in Python 3.6).
def slow_square(number):
''' Take a nap and then print the square of a number.'''
print(f"Sleeping for {number} seconds")
time.sleep(number)
return number**2
slow_square(3)
Sleeping for 3 seconds
9
We will now import the "least recently used cache" decorator from the functools module.
from functools import lru_cache
lru_cache
<function functools.lru_cache(maxsize=128, typed=False)>
lru_cache is a Python function that takes another Python function as its argument. We can modify our slow_square function.
slow_square = lru_cache(slow_square)
slow_square(3)
Sleeping for 3 seconds
9
OK. It looks like nothing has changed. What's the big deal?
slow_square(3)
9
Lookie here! It did not take a nap! It remembered that it had already solved this problem and gave us the old solution. It had cached the answer. It had memoized it.
Let's try again with a different argument.
slow_square(4)
Sleeping for 4 seconds
16
slow_square(4)
16
Fool me once, shame on you. Fool me twice, shame on me! The decorated slow_square function is not so slow any more.
Now, we introduce the syntactic sugar for decoration: the @ syntax.
@lru_cache
def slow_square(number):
''' Take a nap and then print the square of a number.'''
print(f"Now sleeping for {number} seconds")
time.sleep(number)
return number**2
slow_square(3)
Now sleeping for 3 seconds
9
The new syntax has the same meaning as the old, but it is more succinct.
slow_square(3)
9
slow_square(2)
Now sleeping for 2 seconds
4
slow_square(2)
4
Let's try it with a function that has 2 arguments.
@lru_cache
def slow_add(a,b):
''' Take a nap and then add two numbers.'''
print(f"Sleeping for {a+b} seconds")
time.sleep(a+b)
return a+b
slow_add(2,2)
Sleeping for 4 seconds
4
slow_add(2,2)
4
Note that lru_cache can decorate functions with more than one variable.
dir(slow_add)
['__annotations__', '__call__', '__class__', '__copy__', '__deepcopy__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__wrapped__', 'cache_clear', 'cache_info', 'cache_parameters']
slow_add.__doc__
' Take a nap and then add two numbers.'
slow_add.__name__
'slow_add'
We observe that slow_add knows its doc string and name. This is because lru_cache does the right thing when it creates the decorated function. It also remembers the old (undecorated) value of the slow_add function.
slow_add.__wrapped__(2,2)
Sleeping for 4 seconds
4
slow_add.__wrapped__(2,2)
Sleeping for 4 seconds
4
slow_add.cache_info()
CacheInfo(hits=1, misses=1, maxsize=128, currsize=1)
We have seen how to use a decorator. Now will we see how to write one.
def bold(func):
def g(*args):
print ("<b>", end="")
func(*args)
print ("</b>", end="")
return g
def italic(func):
def g(*args):
print ("<i>", end="")
func(*args)
print ("</i>", end="")
return g
@bold
@italic
def test(s):
'this is test'
print (str(s), end="")
test("this is a string.")
<b><i>this is a string.</i></b>
test(12345)
<b><i>12345</i></b>
test('another test')
<b><i>another test</i></b>
The italic decorator surrounds the text string with open and close italic HTML tags:
<i>...</i>The bold decorator surrounds the text with open and close bold HTML tags:
<b>...</b>
The next example is actually useful. You can add debugging output to your functions without having to edit the functions themselves.
def debug(f):
def g(*args):
print ("Calling: {} with args: {}".format(f.__name__, args))
val = f(*args)
print ("Exiting: {} with value: {}".format(f.__name__, val))
return val
return g
@debug
def add2(a):
return a + 2
add2(9)
Calling: add2 with args: (9,) Exiting: add2 with value: 11
11
add2(-2)
Calling: add2 with args: (-2,) Exiting: add2 with value: 0
0
Another useful decoration. This one tells you how fast (or slow) a function executed. Note that we call the decorator wraps(funct) within the decorator. This does
the bookkeeping to fix the doc string, function name, and old (undecorated) function.
def timethis(func):
'''
Decorator that reports the execution time.
'''
@wraps(func)
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(func.__name__, end-start)
return result
return wrapper
time.time()
1666630749.5489774
time.time()
1666630749.9888592
@timethis
def countdown(n):
'''
Counts down
'''
while n > 0:
n -= 1
We did not use wraps in debug or the bold or italic decorators.
help(test)
Help on function g in module __main__: g(*args)
test.__doc__
help(timethis)
Help on function timethis in module __main__:
timethis(func)
Decorator that reports the execution time.
timethis.__doc__
'\n Decorator that reports the execution time.\n '
help(countdown)
Help on function countdown in module __main__:
countdown(n)
Counts down
countdown(1000)
countdown 0.0001289844512939453
countdown(1000000)
countdown 0.05829787254333496
countdown(10000000)
countdown 0.3950803279876709
We can unwrap countdown.
oldcountdown = countdown.__wrapped__
oldcountdown(10000)
help(oldcountdown)
Help on function countdown in module __main__:
countdown(n)
Counts down
dir(countdown)
['__annotations__', '__builtins__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__wrapped__']
countdown.__doc__
'\n Counts down\n '
test.__doc__
We can trace the excution of a function - particularly a recursive one.
def trace(f):
f.indent = 0
def g(x):
print('| ' * f.indent + '|--', f.__name__, x)
f.indent += 1
value = f(x)
print('| ' * f.indent + '|--', 'return', repr(value))
f.indent -= 1
return value
return g
@trace
def fib(n):
if n == 0:
return 1
elif n == 1:
return 1
else:
return fib(n - 1) + fib(n - 2)
fib(4)
|-- fib 4 | |-- fib 3 | | |-- fib 2 | | | |-- fib 1 | | | | |-- return 1 | | | |-- fib 0 | | | | |-- return 1 | | | |-- return 2 | | |-- fib 1 | | | |-- return 1 | | |-- return 3 | |-- fib 2 | | |-- fib 1 | | | |-- return 1 | | |-- fib 0 | | | |-- return 1 | | |-- return 2 | |-- return 5
5
Instead of using lru_cache we can write our own memoization decorator.
First, we will write a memoized fibonacci from scratch.
def memofib(n):
table = [False for x in range(n+1)]
return memofibaux(n,table)
def memofibaux(n, table):
if table[n]: return table[n]
if n <= 1: return 1
else:
table[n] = memofibaux(n-1, table) + memofibaux(n-2, table)
return table[n]
memofib(4)
5
memofibaux = trace(memofibaux)
memofib(4)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Input In [58], in <cell line: 1>() ----> 1 memofib(4) Input In [55], in memofib(n) 1 def memofib(n): 2 table = [False for x in range(n+1)] ----> 3 return memofibaux(n,table) TypeError: trace.<locals>.g() takes 1 positional argument but 2 were given
We have to modify trace to handle multiple arguments using *args (splat args).
def trace2(f):
f.indent = 0
def g(*x):
print('| ' * f.indent + '|--', f.__name__, x)
f.indent += 1
value = f(*x)
print('| ' * f.indent + '|--', 'return', repr(value))
f.indent -= 1
return value
return g
@trace2
def memofibaux(n, table):
if table[n]: return table[n]
if n <= 1: return 1
else:
table[n] = memofibaux(n-1, table) + memofibaux(n-2, table)
return table[n]
memofib(4)
|-- memofibaux (4, [False, False, False, False, False]) | |-- memofibaux (3, [False, False, False, False, False]) | | |-- memofibaux (2, [False, False, False, False, False]) | | | |-- memofibaux (1, [False, False, False, False, False]) | | | | |-- return 1 | | | |-- memofibaux (0, [False, False, False, False, False]) | | | | |-- return 1 | | | |-- return 2 | | |-- memofibaux (1, [False, False, 2, False, False]) | | | |-- return 1 | | |-- return 3 | |-- memofibaux (2, [False, False, 2, 3, False]) | | |-- return 2 | |-- return 5
5
Note that the table is passed as a parameter and that extra recursive calls are avoided.
memofib(10)
|-- memofibaux (10, [False, False, False, False, False, False, False, False, False, False, False]) | |-- memofibaux (9, [False, False, False, False, False, False, False, False, False, False, False]) | | |-- memofibaux (8, [False, False, False, False, False, False, False, False, False, False, False]) | | | |-- memofibaux (7, [False, False, False, False, False, False, False, False, False, False, False]) | | | | |-- memofibaux (6, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | |-- memofibaux (5, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | | |-- memofibaux (4, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | | | |-- memofibaux (3, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | | | | |-- memofibaux (2, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | | | | | |-- memofibaux (1, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | | | | | | |-- return 1 | | | | | | | | | |-- memofibaux (0, [False, False, False, False, False, False, False, False, False, False, False]) | | | | | | | | | | |-- return 1 | | | | | | | | | |-- return 2 | | | | | | | | |-- memofibaux (1, [False, False, 2, False, False, False, False, False, False, False, False]) | | | | | | | | | |-- return 1 | | | | | | | | |-- return 3 | | | | | | | |-- memofibaux (2, [False, False, 2, 3, False, False, False, False, False, False, False]) | | | | | | | | |-- return 2 | | | | | | | |-- return 5 | | | | | | |-- memofibaux (3, [False, False, 2, 3, 5, False, False, False, False, False, False]) | | | | | | | |-- return 3 | | | | | | |-- return 8 | | | | | |-- memofibaux (4, [False, False, 2, 3, 5, 8, False, False, False, False, False]) | | | | | | |-- return 5 | | | | | |-- return 13 | | | | |-- memofibaux (5, [False, False, 2, 3, 5, 8, 13, False, False, False, False]) | | | | | |-- return 8 | | | | |-- return 21 | | | |-- memofibaux (6, [False, False, 2, 3, 5, 8, 13, 21, False, False, False]) | | | | |-- return 13 | | | |-- return 34 | | |-- memofibaux (7, [False, False, 2, 3, 5, 8, 13, 21, 34, False, False]) | | | |-- return 21 | | |-- return 55 | |-- memofibaux (8, [False, False, 2, 3, 5, 8, 13, 21, 34, 55, False]) | | |-- return 34 | |-- return 89
89
Now we create a decorator, like lru_cache.
def memoized(f):
"""Decorator that caches a function's return value each time it is
called. If called later with the same arguments, the cached value
is returned, and not re-evaluated.
"""
cache = {}
@wraps(f)
def wrapped(*args):
try:
result = cache[args]
except KeyError:
result = cache[args] = f(*args)
return result
return wrapped
We will now use two decorators to trace and memoize another fib function.
@trace
@memoized
def mfib(n):
if n <= 0: return 1
if n == 1: return 1
else:
return mfib(n-1) + mfib(n-2)
mfib(10)
|-- mfib 10 | |-- mfib 9 | | |-- mfib 8 | | | |-- mfib 7 | | | | |-- mfib 6 | | | | | |-- mfib 5 | | | | | | |-- mfib 4 | | | | | | | |-- mfib 3 | | | | | | | | |-- mfib 2 | | | | | | | | | |-- mfib 1 | | | | | | | | | | |-- return 1 | | | | | | | | | |-- mfib 0 | | | | | | | | | | |-- return 1 | | | | | | | | | |-- return 2 | | | | | | | | |-- mfib 1 | | | | | | | | | |-- return 1 | | | | | | | | |-- return 3 | | | | | | | |-- mfib 2 | | | | | | | | |-- return 2 | | | | | | | |-- return 5 | | | | | | |-- mfib 3 | | | | | | | |-- return 3 | | | | | | |-- return 8 | | | | | |-- mfib 4 | | | | | | |-- return 5 | | | | | |-- return 13 | | | | |-- mfib 5 | | | | | |-- return 8 | | | | |-- return 21 | | | |-- mfib 6 | | | | |-- return 13 | | | |-- return 34 | | |-- mfib 7 | | | |-- return 21 | | |-- return 55 | |-- mfib 8 | | |-- return 34 | |-- return 89
89
If we call it again, it remembers the old answer.
mfib(10)
|-- mfib 10 | |-- return 89
89
mfib(12)
|-- mfib 12 | |-- mfib 11 | | |-- mfib 10 | | | |-- return 89 | | |-- mfib 9 | | | |-- return 55 | | |-- return 144 | |-- mfib 10 | | |-- return 89 | |-- return 233
233
Here is a better way with constant space. Do not need n-entry table
def fibbetter(n):
return fibbetteraux(n, 1, 1)
## Note: this is tail recursive. Reflected in trace output.
def fibbetteraux(n, x, y):
if n <= 0:
return x
else:
return fibbetteraux(n-1,y, x+y)
fibbetter(10)
89
We first untrace the functions.
## no trace this time.
def fib(n):
if n == 0:
return 1
elif n == 1:
return 1
else:
return fib(n - 1) + fib(n - 2)
def memofibaux(n, table):
if table[n]: return table[n]
if n <= 1: return 1
else:
table[n] = memofibaux(n-1, table) + memofibaux(n-2, table)
return table[n]
@memoized
def mfib(n):
if n <= 0: return 1
if n == 1: return 1
else:
return mfib(n-1) + mfib(n-2)
def doit():
f1 = timeit('fib(10)', setup = 'from __main__ import fib', number=1)
f2 = timeit('memofib(10)', setup = 'from __main__ import memofib', number=1)
f3 = timeit('mfib(10)', setup = 'from __main__ import mfib', number=1)
f4 = timeit('fibbetter(10)', setup = 'from __main__ import fibbetter', number=1)
return f1, f2, f3, f4
doit()
(6.396399112418294e-05, 2.163101453334093e-05, 3.597204340621829e-05, 2.214504638686776e-05)
def doit2(n):
f1 = timeit('fib(' + str(n) + ')', setup = 'from __main__ import fib', number=1)
f2 = timeit('memofib(' + str(n) + ')', setup = 'from __main__ import memofib', number=1)
f3 = timeit('mfib(' + str(n) + ')', setup = 'from __main__ import mfib', number=1)
f4 = timeit('fibbetter(' + str(n) + ')', setup = 'from __main__ import fibbetter', number=1)
return f1, f2, f3, f4
doit2(10)
(6.357498932629824e-05, 2.0317966118454933e-05, 3.1539821065962315e-06, 8.261995390057564e-06)
See Python Cookbook, chapter 9.2
from functools import wraps
def without_wraps(func):
def __wrapper(*args, **kwargs):
return func(*args, **kwargs)
return __wrapper
def with_wraps(func):
@wraps(func)
def __wrapper(*args, **kwargs):
return func(*args, **kwargs)
return __wrapper
We have created two decorators - without and with wraps. We will decorate two dummy functions.
@without_wraps
def my_func_a():
"""Here is my_func_a doc string text."""
pass
@with_wraps
def my_func_b():
"""Here is my_func_b doc string text."""
pass
We will now look at the properties of these functions.
dir(my_func_a)
['__annotations__', '__builtins__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
dir(my_func_b)
['__annotations__', '__builtins__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__wrapped__']
print (my_func_a.__doc__)
print (my_func_a.__name__)
print (my_func_a.__module__)
None __wrapper __main__
print (my_func_a.__wrapped__)
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Input In [80], in <cell line: 1>() ----> 1 print (my_func_a.__wrapped__) AttributeError: 'function' object has no attribute '__wrapped__'
We get an exception because we did not wrap the function, saving original definition.
print (my_func_b.__doc__)
print (my_func_b.__name__)
print (my_func_b.__module__)
print (my_func_b.__wrapped__)
Here is my_func_b doc string text. my_func_b __main__ <function my_func_b at 0x7f9200293c70>
Here is a decorator that executes a function twice and returns both results as a tuple.
def do_twice(func):
''' Decorator that executes a function twice and returns the results as a 2-tuple'''
@wraps(func)
def g(*args):
val1 = func(*args)
val2 = func(*args)
return (val1, val2)
return g
@do_twice
def add2(n):
''' add 2 to the input'''
return n+2
add2(5)
(7, 7)
This is really not very interesting. If a function always returns the same value for a given input, why execute multiple times? That was why we were memoizing functions!
Here is a better example.
import random
def roll_dice():
''' roll a fair die - a random value between 1 and 6 inclusive.'''
return random.randint(1,6)
roll_dice()
2
roll_dice()
1
@do_twice
def roll_dice():
''' roll a fair die - a random value between 1 and 6 inclusive.'''
return random.randint(1,6)
roll_dice()
(6, 3)
roll_dice()
(6, 5)
help(roll_dice)
Help on function roll_dice in module __main__:
roll_dice()
roll a fair die - a random value between 1 and 6 inclusive.
End of Decorator notebook.