Getting started with Python testing
“People also underestimate the time they spend debugging. They underestimate how much time they can spend chasing a long bug. With testing, I know straight away when I added a bug. That lets me fix the bug immediately, before it can crawl off and hide.” – Martin Fowler
Testing your code is a common practice that I feel requires more attention. During my degree there was a period where unit tests were emphasized as an appropriate thing to do, and other times where they were completely ignored. Similarly, during my high school years I competed in a number of programming competitions, often over a number of weeks where the solutions I was producing was being marked against some test but I was only manually testing my code before having it marked. While testing is brought up in high school curriculums as well as university, I believe it is generally only discussed as an idea and not actually implemented in class. The aim of this post is to give you a general introduction to how we might be able to test a function in Python 3.
Our Plan
There is a process used in the software industry called Test-Driven Development or TDD for short. While it’s not always followed, it does provide a nice way of developing software small and large. Simply stated, TDD requires the developer to write their tests first, and then write the actual product code. Today, I will demonstrate this practice in this blog post.
The Problem Space
I have decided to pick a non-trivial problem for this demonstration, writing a function that returns the nth Fibonacci Number. While this function is not particularly difficult to implement, I have chosen it so that people of all experience levels can follow along.
The Fibonacci sequence begins as follows:
1, 1, 2, 3, 5, 8, 13, 21, ...
where we start with the numbers 1, 1 and then get the next number in the sequence by adding the two previous numbers together. For example, the next number in our example after 21 would be 13 + 21 = 34.
Initial Requirements
When we approach a problem that requires tests, we need to come up with some requirements that we want to test are satisfied. Looking at the fibonacci sequence we have a few requirements, namely:
- When we pass the function some number
nwe expect it to return the number that is in positionnin the sequence, with the zeroth number being1. - Our function should raise a
ValueErrorif we pass in a number less than0because it makes no sense to have anything before the zeroth position in the sequence.
Our First Test
I use pytest when writing tests as I find it produces nice output and together with a number of plugins can lead to a very enjoyable testing experience. Using pip you can install pytest from the command line:
pip install pytest
Testing in Python doesn’t strictly require pytest but I find it makes testing a lot nicer.
Now opening a new file called test_fibonacci.py we can begin writing our tests. I like to start by importing TestCase from the unittest module included in the Python standard library:
from unittest import TestCase
This class will allow us to write subclasses to test different parts of our code, in this tutorial we will only write one. You can also write individual tests as functions as shown on the pytest homepage however by writing your tests as classes you can group your tests in a more pleasing fashion. To begin with, lets create a FibonacciTests class:
from unittest import TestCase
class FibonacciTests(TestCase):
pass
This snippet has created a new class that gives us all the utilities available to TestCase which we will cover as we go. Now to write some tests, unittest requires that we write our method names begining with test. The first test we will write is to check our first condition, that the fibonacci function will return the correct fibonacci number at the position. Our first test looks something like this:
from unittest import TestCase
class FibonacciTests(TestCase):
def test_returns_correct_fibonacci_number(self):
correct_sequence = [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]
for index in range(len(correct_sequence)):
response = fibonacci(index)
assert response == correct_sequence[index]
This test, test_returns_correct_fibonacci_number follows a common testing pattern:
- Setup your initial expected data
- Run the code that you are testing
- Check that the tested code did the right thing (in this case, returned the correct fibonacci number)
Disclaimer, the fibonacci sequence is an infinite sequence and so we can’t test all values but we can be reasonably sure of the correctness. The general advice is to try to test any edge cases, for us that’s positions less than 0, and also test the general case as best you can. While we could have tested hundreds or thousands more numbers, tests are often part of a test suite which might contain hundreds or thousands of tests which each require some time to run. This is a choice that the developer and potentially a QA team need to make as to how quickly you want tests to run versus how sure you need to be about the correctness of the code. Your tests may not ever be complete, even if you have full code coverage, and ultimately it is about reducing the risk that a user is going to come across a bug, that is we want our test suites to find bugs before we release any software to our actual users.
Back to the test and following the advice from above, we choose to test 12 different inputs to the function fibonacci (which we are yet to write). For each on of the positions we pass it to the fibonacci function and then test that the response from that call matches the expected result from our correct_sequence list using the assert keyword followed by the condition we want to check. In this case, we are asserting that the response from our fibonacci function matches the corresponding condition in our expected sequence.
Our Second Test
The second requirement we had for our fibonacci function is that it raises a ValueError if we pass it any position less than 0. Again, we can’t possibly test all numbers less than 0 but it is probably fine for us to just test -1 in this case since we have already tested 0 in test_returns_correct_fibonacci_number. Our code now looks like this:
from unittest import TestCase
import pytest
class FibonacciTests(TestCase):
def test_returns_correct_fibonacci_number(self):
correct_sequence = [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]
for index in range(len(correct_sequence)):
response = fibonacci(index)
assert response == correct_sequence[index]
def test_raise_value_error_on_negative_input(self):
with pytest.raises(ValueError):
fibonacci(-1)
To test that fibonacci raises a ValueError on -1 we can use the pytest.raises context manager as shown above. pytest.raises takes our exception type and then passes the test if the code in the scope of the with block raises that type of exception.
Running Our Tests
Now that we have our initial tests, it’s time to run them. To do this, simply run:
pytest
in your command line. By running this, we have stumbled across our first bug:
test_fibonacci.py FF [100%]
================================================================================================= FAILURES =================================================================================================
_________________________________________________________________________ FibonacciTests.test_raise_value_error_on_negative_input __________________________________________________________________________
self = <test_fibonacci.FibonacciTests testMethod=test_raise_value_error_on_negative_input>
def test_raise_value_error_on_negative_input(self):
with pytest.raises(ValueError):
> fibonacci(-1)
E NameError: name 'fibonacci' is not defined
test_fibonacci.py:15: NameError
___________________________________________________________________________ FibonacciTests.test_returns_correct_fibonacci_number ___________________________________________________________________________
self = <test_fibonacci.FibonacciTests testMethod=test_returns_correct_fibonacci_number>
def test_returns_correct_fibonacci_number(self):
correct_sequence = [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]
for index in range(len(correct_sequence)):
> response = fibonacci(index)
E NameError: name 'fibonacci' is not defined
test_fibonacci.py:10: NameError
========================================================================================= 2 failed in 0.07 seconds =========================================================================================
Looking at the pytest output, we can see the obvious issue, we have no fibonacci function yet. Lets fix that in a file called main.py:
def fibonacci(position):
if position == 1 or position == 2:
return 1
return fibonacci(position - 2) + fibonacci(position - 1)
Before we run our tests we will need to import this function at the top of our test_fibonacci.py file like so:
from main import fibonacci
Debugging fibonacci
Running our tests again, we get a RecursionError which means that our function called recusively too many times. Looking at our code, we notice we have accidentally forgotten to 0-index our positions. This can be fixed as follows:
def fibonacci(position):
if position == 0 or position == 1:
return 1
return fibonacci(position - 2) + fibonacci(position - 1)
Now when we run pytest we notice that we have actually passed the first test but failed test_raise_value_error_on_negative_input:
test_fibonacci.py F. [100%]
================================================================================================= FAILURES =================================================================================================
_________________________________________________________________________ FibonacciTests.test_raise_value_error_on_negative_input __________________________________________________________________________
position = -1883
def fibonacci(position):
> if position == 0 or position == 1:
E RecursionError: maximum recursion depth exceeded in comparison
main.py:10: RecursionError
==================================================================================== 1 failed, 1 passed in 1.14 seconds ====================================================================================
While we still have to fix our second test, congratulations on passing your first! It’s that pesky RecursionError again, how could that have happened? Well, looking at the last line of the function return fibonacci(position - 2) + fibonacci(position - 1) if we pass -1 as the position it will keep hitting this line and never exit, so there’s our issue. We can fix this by adding a check for negative numbers:
def fibonacci(position):
if position <= 0:
raise ValueError('position must be non-negative')
if position == 0 or position == 1:
return 1
return fibonacci(position - 2) + fibonacci(position - 1)
We re-run pytest and oops, we broke our first test:
test_fibonacci.py .F [100%]
================================================================================================= FAILURES =================================================================================================
___________________________________________________________________________ FibonacciTests.test_returns_correct_fibonacci_number ___________________________________________________________________________
self = <test_fibonacci.FibonacciTests testMethod=test_returns_correct_fibonacci_number>
def test_returns_correct_fibonacci_number(self):
correct_sequence = [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]
for index in range(len(correct_sequence)):
> response = fibonacci(index)
test_fibonacci.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
position = 0
def fibonacci(position):
if position <= 0:
> raise ValueError('position must be non-negative')
E ValueError: position must be non-negative
main.py:17: ValueError
==================================================================================== 1 failed, 1 passed in 0.06 seconds ====================================================================================
What a sneaky mistake, we accidently raised the ValueError when position is 0 as well. A quick fix should get everything passing:
def fibonacci(position):
if position < 0:
raise ValueError('position must be non-negative')
if position == 0 or position == 1:
return 1
return fibonacci(position - 2) + fibonacci(position - 1)
and TA-DA!!! We passed all our tests:
test_fibonacci.py .. [100%]
========================================================================================= 2 passed in 0.01 seconds =========================================================================================
Well Done!
Well that brings us to the end of testing our fibonacci function. I hope you have now covered:
- The fundamentals of Test-Driven Development
- How to use pytest to run simple tests
- How to use testing as a means to find bugs and describe requirements
The next thing you would need to do if this was production code, is assume that you will get bug reports about this code, perhaps we missed a requirement. When a new issue comes in, you would add another test case, run the tests to make sure that it fails, and then make any changes to the function to make that test pass. Once you again have reason to believe the code is correct, you can go ahead and re-release it.
You can find all the code related to this post here on GitHub.