Functions and Classes in Python

Running with Pytest¶

Overview:

Teaching: 10 min
Exercises: 10 min

Questions

What is Pytest?
Why do I use it?

Objectives

Know that Pytest provides a framework to automate your testing
Know how to run Pytest
Understand how to interpret pytest output
Know that within the framework you can mark tests as known fails or exclude them for unwritten features

What is Pytest¶

Pytest has rapidly established itself as the standard python testing framework. Its power lies in its simplicity, which makes it super easy to write tests and to run them. Tests can be as simple as the ones that we've already seen, so it is perfect for small projects. However, hidden behind the simple exterior pytest provides a great deal of power and flexibility, meaning that it scales well to large and complex projects.

In this section we will learn how to automatically find and run tests using pytest, then query and interpret the output.

Running tests¶

Pytest comes with a command-line tool called, unsurprisingly, pytest. (With older versions of python this was called py.test.) When run, pytest searches for tests in all directories and files below the current directory, collects the tests together, then runs them. Pytest uses name matching to locate the tests. Valid names start or end with test, e.g.

# Files:
test_file.py       file_test.py

# Functions:
def test_func():   def func_test():

In what follows, we will use the convention of test_* when naming files and functions.

You can specify one or more paths and pytest will only look for tests in those paths, e.g.

pytest /path/to/my/awesome/module

You can also specify the name of a file (or files) containing test functions, e.g.

pytest /path/to/my/awesome/module/test/tests.py

When writing a python package it is good practice to set up a directory structure in order to keep things tidy. Throughout this course we will use the following:

mypkg/
    __init__.py
    whizz.py
    bang.py
    test/
        __init__.py
        test_whizz.py
        test_bang.py

Here the __init__.py files makes python aware that the directory should be treated as a package (a collection of python modules). Assuming we're in the top level directory (where our notebooks reside) this allows us to do the following:

import mypkg

Let's dive in and run some tests using mypkg that was introduced in the previous section.

!pytest mypkg

============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0
rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:
collected 17 items                                                             

mypkg/test/test_errors.py .s                                             [ 11%]
mypkg/test/test_mymath.py x.......s..F...                                [100%]

=================================== FAILURES ===================================
____________________________ test_greaterThan[3-7] _____________________________

x = 3, y = 7

    @pytest.mark.parametrize("x, y",
                            [(108, 56),
                             (-64, -333),
                             (3, 7),
                             (74, 15)])
    def test_greaterThan(x, y):
        """ Test the greaterThan function. """
    
        if x > y:
            assert greaterThan(x, y)
        else:
>           assert not greaterThan(x, y)
E           assert not True
E            +  where True = greaterThan(3, 7)

mypkg/test/test_mymath.py:47: AssertionError
========== 1 failed, 13 passed, 2 skipped, 1 xfailed in 2.74 seconds ===========

What just happened?

Well, pytest searched within the mypkg directory and collected a total of 17 tests. These tests were spread accross two files:

mypkg/test/test_errors.py
mypkg/test/test_mymath.py

The tests were then run and a single failure was reported for the function test_greaterThan. This test failed when asserting that 3 shouldn't be greater than 7.

E           assert not True
E            +  where True = greaterThan(3, 7)

What are all the cryptic symbols next to the name of each test file?

mypkg/test/test_mymath.py x.......s...F....                           [100%]

To get more detailed information about each test, we can run pytest using the verbose flag.

!pytest mypkg -v

============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:
collected 17 items                                                             

mypkg/test/test_errors.py::test_indexError PASSED                        [  5%]
mypkg/test/test_errors.py::test_BSoD SKIPPED                             [ 11%]
mypkg/test/test_mymath.py::test_add xfail                                [ 17%]
mypkg/test/test_mymath.py::test_sub[1-2--1] PASSED                       [ 23%]
mypkg/test/test_mymath.py::test_sub[7-3-4] PASSED                        [ 29%]
mypkg/test/test_mymath.py::test_sub[21-58--37] PASSED                    [ 35%]
mypkg/test/test_mymath.py::test_mul[3-1] PASSED                          [ 41%]
mypkg/test/test_mymath.py::test_mul[3-2] PASSED                          [ 47%]
mypkg/test/test_mymath.py::test_mul[4-1] PASSED                          [ 52%]
mypkg/test/test_mymath.py::test_mul[4-2] PASSED                          [ 58%]
mypkg/test/test_mymath.py::test_div SKIPPED                              [ 64%]
mypkg/test/test_mymath.py::test_greaterThan[108-56] PASSED               [ 70%]
mypkg/test/test_mymath.py::test_greaterThan[-64--333] PASSED             [ 76%]
mypkg/test/test_mymath.py::test_greaterThan[3-7] FAILED                  [ 82%]
mypkg/test/test_mymath.py::test_greaterThan[74-15] PASSED                [ 88%]
mypkg/test/test_mymath.py::test_isLucky PASSED                           [ 94%]
mypkg/test/test_mymath.py::test_bigSum PASSED                            [100%]

=================================== FAILURES ===================================
____________________________ test_greaterThan[3-7] _____________________________

x = 3, y = 7

    @pytest.mark.parametrize("x, y",
                            [(108, 56),
                             (-64, -333),
                             (3, 7),
                             (74, 15)])
    def test_greaterThan(x, y):
        """ Test the greaterThan function. """
    
        if x > y:
            assert greaterThan(x, y)
        else:
>           assert not greaterThan(x, y)
E           assert not True
E            +  where True = greaterThan(3, 7)

mypkg/test/test_mymath.py:47: AssertionError
========== 1 failed, 13 passed, 2 skipped, 1 xfailed in 2.77 seconds ===========

What does pytest output mean?¶

We now have more detailed information about each test. Matching up the output with the symbols we can see that:

. = PASSED
s = SKIPPED
F = FAILED
x = xfail

What does xfail mean, and why were some tests skipped? Also, note that some tests were run mutliple times, e.g. test_sub appears 3 times.

What do the numbers in square brackets mean? Looking at the report for the test that failed, we can see that the square brackets show the arguments to the test functions, e.g. x=3 and y=7.

To report more information on tests that were recorded as SKIPPED, or xfail, we can run pytest as follows.

!pytest mypkg -rsx

============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.0, pluggy-0.6.0
rootdir: /home/lester/Code/siremol.org/chryswoods.com/python_and_data/testing, inifile:
collected 17 items                                                             

mypkg/test/test_errors.py .s                                             [ 11%]
mypkg/test/test_mymath.py x.......s..F...                                [100%]
=========================== short test summary info ============================
SKIP [1] mypkg/test/test_errors.py:12: Only runs on Windows.
SKIP [1] mypkg/test/test_mymath.py:30: Not yet implemented.
XFAIL mypkg/test/test_mymath.py::test_add
  Broken code. Working on a fix.

=================================== FAILURES ===================================
____________________________ test_greaterThan[3-7] _____________________________

x = 3, y = 7

    @pytest.mark.parametrize("x, y",
                            [(108, 56),
                             (-64, -333),
                             (3, 7),
                             (74, 15)])
    def test_greaterThan(x, y):
        """ Test the greaterThan function. """
    
        if x > y:
            assert greaterThan(x, y)
        else:
>           assert not greaterThan(x, y)
E           assert not True
E            +  where True = greaterThan(3, 7)

mypkg/test/test_mymath.py:47: AssertionError
========== 1 failed, 13 passed, 2 skipped, 1 xfailed in 2.78 seconds ===========

We now have additional information about these tests. One test was skipped because it is only valid on the Windows operating system, another because it was testing functionality that hasn't been implemented yet.

By now, you might have guessed that xfail means expected to fail. You can see that the test_add was expected to fail with the reason "Broken code. Working on a fix." This was the function that we found to be broken in the previous section.

Let's switch to the terminal and try to work out why the test was failing. Since we know that test_greaterThan was the culprit, let's look at the greaterThan function in mypkg/mymodule.py to see what could be going wrong.

Well, that was silly. Having fixed the bug, let's confirm that our tests now pass. Let's also run pytest in quiet mode. Here we only get a minimal output showing the progress of the tests and reports for any failures (so hopefully none!).

!pytest mypkg -q

.sx.......s......                                                        [100%]
14 passed, 2 skipped, 1 xfailed in 2.72 seconds

Exercises¶

More details on how to invoke pytest can be found here.

1

Only run the test_mul unit test. This test is in the file mypkg/test/test_mymath.py

Solution

2

Which are the three slowest tests?

Solution

Key Points¶

Pytest provides a framework to automate running your tests
Pytest will search for and run all tests identifying passes and fails
You can mark tests to run in different conditions or not to run e.g. unimplemented functionality