Closures and Decorators (Advanced)

Please note that this is a long lesson because there are a lot of details that you need to understand closures and decorators.

In this lesson we will learn about:

  • Global variables
  • Nested functions
  • Nonlocal variables
  • Higher order functions
  • Closures
  • Decorated functions

1. Closures

1.1 Global Variables

I have deliberately not introduced global variables until now because in 99.9% of time if you are using a global variable you have designed your program badly.

I am introducing them so that we will understand nonlocal variables in a couple of sections.

Consider the following code,

def add_five(x):
    x = x + 5
    print(f"Value of x in add_five() is {x}")

x = 10
print(f"Value of x before add_five() is {x}")
add_five(x)
print(f"Value of x after add_five() is {x}")

will output:

Value of x before add_five() is 10
Value of x in add_five() is 15
Value of x after add_five() is 10

This is because the x in add_five() is local to the function add_five(). Modifying it does not change the x defined in the global scope (outside the function).

We can alter this by adding the global keyword in front of x in the function and not passing it as a parameter as follows:

def add_five():
    global x
    x = x + 5
    print(f"Value of x in add_five() is {x}")

x = 10
print(f"Value of x before add_five() is {x}")
add_five()
print(f"Value of x after add_five() is {x}")

The output now is:

Value of x before add_five() is 10
Value of x in add_five() is 15
Value of x after add_five() is 15

Now we are modifying the global variable x, it does not belong to the scope of the add_five() function.

1.2 Nested Functions

Python allows you to nest a function within a function. Here is an example.

def outer():
    
    print("Hi I am in the outer function")
    
    def inner():
        print("Hi I am in the inner function")
    
    # make a call to the inner function
    inner()
    print("Hi I am again in the outer function")

outer()

This will print out the following:

Hi I am in the outer function
Hi I am in the inner function
Hi I am in again in the outer function

1.3 Nonlocal variables

A nonlocal variable works very much like a global variable, but it is restricted to the scope of the outer function.

Consider the following example,

def outer():
    x = 10
    print(f"Hi I am in the outer function, the value of x is {x}")
    
    def inner():
        print(f"Hi I am in the inner function, the value of x is {x}")
    
    # make a call to the inner function
    inner()
    print(f"Hi I am again in the outer function, the value of x is {x}")

outer()

this will result in,

Hi I am in the outer function, the value of x is 10
Hi I am in the inner function, the value of x is 10
Hi I am in again in the outer function, the value of x is 10

as the inner() function has access to the x defined in the scope of the outer() function.

Now consider the following example,

def outer():
    x = 10
    print(f"Hi I am in the outer function, the value of x is {x}")
    
    def inner():
        x = 15
        print(f"Hi I am in the inner function, the value of x is {x}")
    
    # make a call to the inner function
    inner()
    print(f"Hi I am again in the outer function, the value of x is {x}")

outer()

this prints out the following:

Hi I am in the outer function, the value of x is 10
Hi I am in the inner function, the value of x is 15
Hi I am in again in the outer function, the value of x is 10

Notice how once the inner() function had ended, the value of x is 10, this is because the x that is defined at the beginning of outer() is not the same as the one that is used at the beginning of inner().

They are different, when we ran the code x = 15 it defined a new x in the scope of inner().

Now consider the following:

def outer():
    x = 10
    print(f"Hi I am in the outer function, the value of x is {x}")
    
    def inner():
        nonlocal x 
        x = 15
        print(f"Hi I am in the inner function, the value of x is {x}")
    
    # make a call to the inner function
    inner()
    print(f"Hi I am again in the outer function, the value of x is {x}")

outer()

Here we use the nonlocal keyword to let the inner() function know that we are working with the x that belongs to the outer() function.

The output is,

Hi I am in the outer function, the value of x is 10
Hi I am in the inner function, the value of x is 15
Hi I am in again in the outer function, the value of x is 15

and you can see that we actually changed the value of the x belonging to outer().

1.3.1 Being More Explicit

We can be more explicit by using the outer function name when declaring our x variable.

For example,

def outer():
    # declare the variable using the function name.
    outer.x = 10
    print(f"Hi I am in the outer function, the value of x is {outer.x}")
    
    def inner():
        x = 15
        print(f"Hi I am in the inner function, the value of x belonging to inner is {x}")
        print(f"Hi I am in the inner function, the value of x belonging to outer is {outer.x}")
    
    # make a call to the inner function
    inner()
    print(f"Hi I am again in the outer function, the value of x is {outer.x}")

outer()

will print out:

Hi I am in the outer function, the value of x is 10
Hi I am in the inner function, the value of x belonging to inner is 15
Hi I am in the inner function, the value of x belonging to outer is 10
Hi I am in again in the outer function, the value of x is 10

Now it is clear that the inner function is not overwriting the value of outer.x, but we can still access it within the inner function.

We could of course overwrite outer.x from within the inner function:

def outer():
    # declare the variable using the function name.
    outer.x = 10
    print(f"Hi I am in the outer function, the value of x is {outer.x}")
    
    def inner():
        outer.x = 15
        print(f"Hi I am in the inner function, the value of x is {outer.x}")
    
    # make a call to the inner function
    inner()
    print(f"Hi I am again in the outer function, the value of x is {outer.x}")

outer()

This prints out the following:

Hi I am in the outer function, the value of x is 10
Hi I am in the inner function, the value of x is 15
Hi I am in again in the outer function, the value of x is 15

1.4 Higher Order Functions

In Python, fucntions are first class objects. This means that you can do the same things with them as you do with other objects like:

  • Assign to a variable
  • Pass them into other functions
  • Return them from other functions
  • Store them in data strucutres like lists and dictionaries

For example the following assigns a function to a variable.

def add(x,y):
    return x + y

# assign the function add to the name my_add
my_add = add

print(my_add(5,3)) # prints 8
print(add(5,3)) # prints 8

This example returns a function from another function.

# this function returns a function
def another_function():
    def add(x,y):
        return x + y
    return add

# assign the return function to my_add
my_add = another_function()

# call the function using my_add
print(my_add(5,2))

Finally the following example passes a function into another function and then calls it.

def print_function(func, *args):
    # print the return values of the passed in function
    print(func(*args))

def add(x,y):
    return x + y

# pass the add function to print_function
print_function(add, 3, 4) # prints 7

1.5 Closures

What is a closure?

The following is the definition given on Geeks For Geeks:

A Closure is a function object that remembers values in enclosing scopes even if they are not present in memory.

  • It is a record that stores a function together with an environment: a mapping associating each free variable of the function (variables that are used locally but defined in an enclosing scope) with the value or reference to which the name was bound when the closure was created.

  • A closure—unlike a plain function—allows the function to access those captured variables through the closure’s copies of their values or references, even when the function is invoked outside their scope.

When do we have closures?

The criteria that must be met to create a closure in Python are summarised in the following points.

  • We must have a nested function (function inside a function).
  • The nested function must refer to a value defined in the enclosing function.
  • The enclosing function must return the nested function.

This last statement is important. We can return a function, just like a variable. In Python functions are first class objects and we pass them around just like variables, lists etc.

Programiz Python Closures

1.4.1 A Simple Example

Here is an example of a closure that returns a function that will multiply a number by a set number, e.g. 3 or 5.

def create_adder(n):
    def adder(x):
        return n + x
    return adder


# Creates a function that is adds 3 to the input
add3 = create_adder(3)

# Output: 12
print(add3(9))

# Creates a function that is a adds 5 to the input
add5 = create_adder(5)

# Output: 8
print(add5(3))

# Output: 10
print(add5(add3(2)))

We can see here that create_adder() is a function that returns another function using the enclosing functions n.

So add3 = create_adder(3) is actually creating and returning the following function.

def add3(x):
    return 3 + x

# Output: 12
print(add3(9))

2. Decorators

Decorators are basically a closure, whereby we pass in a function and add additional functionality around the function and then return another (decorated) function.

Functions are callable and implement the special __call__() method. Other objects such as methods (see object orientation and classes) implement this method as well.

Anything that is callable can be decorated.

2.1 Decorated Function

Here is a simple example which takes in a function and adds some additional print statements around the function.

def pretty_print(f):
    def inner():
        print("Let's print out this function in a pretty way")
        print("---------------------------------------------")
        # invoke the passed in function
        f()
        print("---------------------------------------------")
    return inner

def print_hello():
    print("Hello")

# pass pretty_print the print_hello() function
# this returns a new decorated function
pretty_hello = pretty_print(print_hello)

# invoke the decorated function
pretty_hello()

Prints out the following:

Let's print out this function in a pretty way
---------------------------------------------
Hello
---------------------------------------------

Here we are passing pretty_print() the print_hello() function and assigning it to the name pretty_hello.

We can then call this newly decorated function.

2.2 Using the @ Symbol for Decorating

It turns out that this is such a common pattern that Python has a special way of marking a function to be decorated by another function.

To do this we write @decorator_name above the function to be decorated. decorator_name is the name of the function that does the decorating.

For example we can amend the code above as follows:

def pretty_print(f):
    def inner():
        print("Let's print out this function in a pretty way")
        print("---------------------------------------------")
        f()
        print("---------------------------------------------")
    return inner


def print_hello():
    print("Hello")

# decorates the function using the @ symbol
@pretty_print
def pretty_hello():
    print_hello()

pretty_hello()

Prints out the following:

Let's print out this function in a pretty way
---------------------------------------------
Hello
---------------------------------------------

2.3 Decorating with Parameters

We can also decorate functions that take in parameters, good job because it would be very limiting if we couldn't.

Here is an excellent example modified from Programiz.

We decorate a standard division so that if we try to divide by 0 we get infinity (division by zero is actually an error, but some calculators display infinity because of something known as limits.)

import math

def smart_divide(func):
    # note that the parameters of inner match the parameters of func
    def inner(a, b):
        print("I am going to divide", a, "and", b)
        if b == 0:
            print("Whoops! cannot divide")
            return math.inf

        return func(a, b)
    return inner


@smart_divide
def divide(a, b):
    return a/b

if __name__ == "__main__":
  print(divide(3,2))
  print(divide(3,0))

The above prints out:

I am going to divide 3 and 2
1.5
I am going to divide 3 and 0
Whoops! cannot divide
inf

The following logs (via print statements) the result of a simple add() function.

def logger_add(func):
    # note that the parameters of inner match the parameters of func
    def inner(x, y):
        print(f"I am going to add {x} and {y}")
        result = func(x,y)
        print(f"The answer is {result}")
    return inner

@logger_add
def add(x, y):
    return x + y

add(3,4)
# prints
# I am going to add 3 and 4
# The answer is 7

You may notice that the parameters of inner() function in logger_add() match the parameters of the function passed in, in this case add().

We can get around this by using *args to take in a unspecified number of arguments.

Here is an example.

def logger_add(func):
    # note that the parameters of inner match the parameters of func
    def inner(*args):
        print(f"I am going to add {args}")
        result = func(*args)
        print(f"The answer is {result}")
    return inner

@logger_add
def add_two(x, y):
    return x + y

  
@logger_add
def add_three(x, y, z):
    return x + y + z

add_two(3,4)
# prints
# I am going to add (3, 4)
# The answer is 7

add_three(3,4,5)
# prints
# I am going to add (3, 4, 5)
# The answer is 12

You can also accepts any number of arguments and keyword arguments.

def universal_decorator(func):
    def inner(*args, **kwargs):
        print("I can decorate all functions")
        func(*args, **kwargs)
    return inner

@universal_decorator
def funky_func(x, y, z=1):
    print(x)
    print(y)
    print(z)

# call the decorated function
funky_func(3,4,z=10)

The above will print out:

I can decorate all functions
3
4
10

=== TASK ===

If you have read everything and understand it, then this should be reasonably straight forward.

Create a logger for a function called logger(). (I would use the examples below as your starting point.)

Your logger should decorate a function and do the following:

  • print some stuff
  • call the function
  • print some more stuff

It should work on any function with any number of arguments or keyword arguments.

Note you will need to look up how to get access to a functions name. You can read the following to help - Function Name

The following examples illustrate how the logger decorator should work. Currently the decorator does nothing but return the same function.

Example 1

def logger(func):
    # write your code here
    return func

@logger
def add_three(x, y, z):
    print(x + y + z)

if __name__ == "__main__":
    add_three(3,4,5)

The above code should print the following:

Logging function...
The functions name is add_three
Calling the function...
12
Ending logging...

Example 2

def logger(func):
    # write your code here
    return func

@logger
def hello():
    print("hello")

if __name__ == "__main__":
    hello()

The above code should print the following:

Logging function...
The functions name is hello
Calling the function...
hello
Ending logging...

Example 3

def logger(func):
    # write your code here
    return func

@logger
def test_keyword(a=1, b=2, c=3):
    print(a+b+c)

if __name__ == "__main__":
    test_keyword(a=2)

The above code should print the following:

Logging function...
The functions name is test_keyword
Calling the function...
7
Ending logging...

To get started copy and paste the following into a new Python file. Currently the logger function just returns the function undecorated.

def logger(func):
    # write your code here
    return func

@logger
def funky_func(x, y, z=1):
    print(x)
    print(y)
    print(z)

@logger
def add_three(x, y, z):
    return x + y + z

@logger
def hello():
    print("hello")

@logger
def test_keyword(a=1, b=2, c=3):
    print(a + b + c)

if __name__ == "__main__":
    funky_func(3, 4, z=10)
    print()
    funky_func(3, 4)
    print()
    add_three(3, 4, 5)
    print()
    hello()
    print()
    test_keyword()

It should output the following if implemented correctly.

Logging function...
The functions name is funky_func
Calling the function...
3
4
10
Ending logging...

Logging function...
The functions name is funky_func
Calling the function...
3
4
1
Ending logging...

Logging function...
The functions name is add_three
Calling the function...
12
Ending logging...

Logging function...
The functions name is hello
Calling the function...
hello
Ending logging...

Logging function...
The functions name is test_keyword
Calling the function...
6
Ending logging...