Unit Testing with Python

Every year I try to learn a new programming language. This year I started learning Python.

During my day job, I work as a professional programmer. I use C++, C#, and Javascript. I am part of a development team that uses unit testing to verify if our code works how it should.

In this article, I investigate how to create unit tests with Python by discussing the following topics.

• Unit testing fundamentals
• Available Python testing frameworks
• Test Design principle
• Test Doubles
• Code Coverage

To be able to follow along with this article, I assume that you have a basic understanding of Python.

Unit Test Fundamentals

I created the unit test examples by using the FizzBuzz Coding Kata. A Coding Kata is an exercise for a programmer. During this exercise, the programmer tries to solve a specific problem. But the main goal is not to solve the problem but to practice programming.

There are different Code Katas you can use to practice; the one I used is the FizzBuzz Code Kata. FizzBuz is a simple Code Kata which is ideal for explaining and showing unit testing in Python.

The FizzBuzz Code Kata:

Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz.” For numbers which are multiples of both three and five print “FizzBuzz.“

Together with the examples of unit tests, I will define the unit test language so that we understand each other.

Unit Test

A unit test is an automated test written by a programmer that tests a small part of the program. Unit tests should run fasts. Tests that interact with the file system, a database, or the network are not Unit tests.

To create the first FizzBuzz unit test in Python, I define a class that inherits from unittest.TestCase. This unittest module is available in the standard installation of Python.

The first unit test validates that if the FizzBuzz filter receives the number one, it returns the string ‘1’. The second unit test does the same for the number two.

Test case

We call the method test_one_should_return_one() in the class FizzBuzzTest, a Test case. The test case is the actual test code that tests a specific part of a program.

The first Test Case validates that if the number 1 passes through the FizzBuzz filter, it returns the string ‘1’. The result is validated using the self.assertEqual method. The first argument of the method is the expected result, and the second argument is the actual result.

If you look at both test cases, you see that they both create an instance of the FizzBuzz class. The first one on line six and another on line eleven.

We can improve the code by refactoring the creation of the FizzBuzz instance out of both methods.

We use the setUp method to create the instance of the FizzBuzz class. The setUp from the TestCase base class is executed before every test case.

The other method tearDown is called after the executing of each unit test. You can use it to clean up or close resources.

Test Fixture

The methods setUp and tearDown are part of a Test Fixture. A Test Fixture is used to configure and construct the unit under test. Each Test case can use these general conditions. In our case, I use it to create the instance of the FizzBuzz class.

To run the unit tests, we need a test runner.

Test Runner

A Test Runner is a program that executes all the unit tests and reports the result. The standard test runner from Python can be run from the terminal using the following command.

python -m unittest test_fizzbuzz.py

Test Suite

The last term of our unit test vocabulary is Test Suite. A Test Suite is a collection of test cases or test suites. Often a Test Suit contains Test cases that should be run together.

With Test Suite, we finished our unit test vocabulary. You now know the meaning and difference between the following.

• Test Suite
• Test Case
• Test Fixture
• Test Runner
• Unit Test

Unit Test Design

Test cases should be well designed. The name of the test and how you structure your test is the most important.

Test Case Name

The name of the test is essential. It is like a headline that summarizes what the test is about. If a test fails, it is the first thing you see. So, the name should give a clear sign of what functionality is not working.

A list of test case names should read like a summary or a list of scenarios. This helps a reader to understand the behavior of the unit under test.

Structuring the Test Case method body

A well-designed test case consists of three parts. The first part, Arrange, set up the object to test. The second part, Act, exercises the unit under test. Finally, the third part, Assert, makes claims about what should happen.

Sometimes, I add the three parts as comments in my unit tests to make it more clear.

Single Assert Per Test Case

Although it is possible to have many asserts in a single Test Case. I always try to use a single assert.

The reason is that when an assert fails, the execution of the test case stops. Thus, you will never know if the next assert in the Test case succeeds or not.

Unit testing with pytest

In the previous section, we used the unittest module. The default installation of Python installs this module. The unittest module was first introduced in 2001. Based on JUnit, the popular Java unit testing framework by Kent Beck and Eric Gamma.

Another module pytest is currently the most popular Python unit testing framework. Compared to the unittest framework, it is more Pythonesque. Instead of deriving from a base class, you can define your test cases as functions.

As pytest is not in the default Python installation we install it using the Package Installer for Python, PIP. By executing the following command in the terminal, you can install pytest.

pip install pytest

I converted the first FizzBuzz test case to pytest below.

There are three differences. First, you don’t need to import any module. Secondly, you don’t need to implement a class and derive from a base class. And finally, you can use the standard Python assert method instead of custom ones.

Test Fixtures

As you remember, the unit test module used the setUp and tearDown to configure and construct the unit under test. Instead, pytest uses the attribute @pytest.fixture. In your test cases, you can use the name of the method you decorated with the attribute as an argument.

The pytest framework connects them during runtime and injects the fizzBuzz instance into your test cases.

If you want to mimic the behavior of the unit test tearDown() method, you can do this using the same method. Instead of return, you use the yield keyword. You can then put your clean up code after the yield[email protected]
def fizzBuzz():
   yield FizzBuzz()
   # put your clean up code here

pytest markers

Markers are attributes that you can use in your tests for various functions. For example, if you add the skip marker to your test case, the test runner will skip the test.

The pytest plugin ecosystem

There are many plugins for pytest that add additional functionality. At the time of writing, there are almost 900 plugins. For example, pytest-html and pytest-sugar.

pytest-html

pytest-html is a plugin for pytest that generates an HTML report for the test results. This can be really useful when you are running the unit tests on a build server.

pytest-sugar

pytest-sugar changes the default look and feel of pytest. It adds a progress bar and shows failing tests instantly.

Using Test Doubles (stubs, mocks, and fakes)

If you want to test a module or a class in isolation, but the module or class depends on a different class, you can use a Test Double to replace the dependency.

A Test Double is like a stunt double. It is a generic term for replacing a production object for testing purposes. There are various kinds of Test Doubles.

I use the following definitions from the xUnit Test Patterns by Gerard Meszaros.

• Mocks are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don’t expect and are checked during verification to ensure they got all the calls they were expecting.
• Fake objects have working implementations, but usually take a shortcut which makes them not suitable for production, for example, an in-memory database.
• Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.

Creating Stubs using unittest.Mock

To be able to demonstrate the use of stubs we will leave FizzBuzz and use a different example. In this example, we will implement an Alarm class that generates an alarm when the temperature in a Greenhouse reaches a certain threshold.

I want to test the Alarm functionality. But instead of using the actual Sensor class, I want to use a Test Double that I can manipulate to return a specific measurement to test the Alarm.

I use the Mock class from unittest.mock to create a Stub on line seven. Then on the next line, I instruct the stub to return a value of 35 when it calls take_measurement is called.

This allows me to test that the Alarm becomes active if the measured temperature is above 32 degrees Celcius.

So as you see, the sensor stubs look the same as the actual sensor class on the outside. Internally it is different. It does not measure the temperature.

Creating Mocks using unittest.Mock

The difference between a Mock and a Stub is that a Stub won’t let the test fail. A Mock, on the contrary, can fail the test if it is not called correctly. So, a Mock does more than a Stub.

Before we look a Mocks, it is important to understand that there are three different kinds of assert.

1. Assert a return value or an exception
2. Assert a change in state
3. Assert a method call. Did a specific method get called?

The FizzBuzz unit test used the first variant. Asserting a return value or an exception.def test_one_should_return_one(fizzBuzz):
   result = fizzBuzz.filter(1)
   assert result == ‘1’

The Alarm unit test used the second variant. Asserting a change in state.

assert alarm.active == True

For the third kind of assert, we need a Mock. We extend the Alarm unit test with an assertion that verifies that the take_measurementmethod was called.

In this unit test, we still use the Mock class from the unit test module. That is because the Mock class can act as a Mock or a Stub. By adding the verification on line ten, mock_sensor.take_measurement.assert_called() we change it from a Stub into a Mock. As the Mock can now fail the test.

Using Fakes

The last type of Test Double I want to talk about is a Fake. Fakes are classes that have a working implementation. The implementation is not the same as the production one. Usually, they have a simplified version of the production code.

You can create these Fakes yourself, but luckily for us, Python already has some Fakes out of the box.

Replace File with String.IO

The String.IO module is a class that acts as a file but reads and writes into string buffers. String.IO stores the data in memory. This means that in your test you can use String.IO instead of real files. This will make your unit test faster and easier to setup.

Example.

Replace Database with in-memory database

You can create an SQLite database in-memory by changing the connection string. The Python sqlite module also supports this behavior.

Using an in-memory database instead of on disk makes your unit test much faster.

import sqlite3 
conn = sqlite3.connect(':memory:')

Replace Webserver with lightweight webserver

Python includes SimpleHttpServer. You can use this module to create a lightweight HTTP server for your unit tests. For example, to test the interaction with an external API.

This will make your unit test much faster.

Creating a code coverage report

There are tools available that can create a code coverage report. This code coverage report show which code is executed (covered) by your unit test.

I use Coverage and pytest-cov to create a code coverage report. Coverage is a general package for measuring code coverage. The module pytest-cov is a plugin for pytest that connects to Coverage.

Both can be installed using pip.

pip install coverage

pip install pytest-cov

After you installed both, you can generate a coverage report with these two commands. Run them in a terminal or command.

coverage run -m pytest

coverage html

The first generates coverage data. The second command transforms the data into an HTML report. Coverage stores the report in a folder htmlcov on your file system.

If you open index.html in a browser it shows an overview of the coverage percentage per file.

If you select a file, it shows the screen below. Coverage added an indication to the source code that shows which line is covered by the unit tests.

Below we see that our unit test does not cover lines twelve and sixteen.

Branch coverage measurement

Coverage also supports branch coverage measurement. With branch coverage, if you have a line in your program that could jump to more than one next line, Coverage tracks if those destinations are visited.

You can create a coverage report with branch coverage by executing the following command.

pytest --cov-report html:htmlcov --cov-branch --cov=alarm

I instruct pytest to generate an HTML coverage report with branch coverage. It should store the result in htmlcov. Instead of generating coverage report for all files, I tell coverage only to use alarm.py.

Conclusion

This concludes my investigation into the unit testing possibilities with Python.

I showed you how to create and run unit tests in Python. We used two test frameworks and test runners, the unittest and pytest module.

We saw how to design your unit test. Come up with the correct name of a test case and the correct structure of the test case body.

I showed you the various types of test doubles, such as stubs, mocks, and fakes. And also explained the different type of asserts that belongs to each test double.

Finally, we looked at how to create a code coverage report. Coverage reports help to spot missing tests for new code or identify which tests to add to the existing code.

I stored the source code of the examples in this Github repository.

Thank you for reading.