Original notebook by Jarrod Millman, part of the Python-bootcamp.
Modifications Hans Fangohr, Sept 2013:
In ordinary computational practice by hand or by desk machines, it is the custom to check every step of the computation and, when an error is found, to localize it by a backward process starting from the first point where the error is noted.
- Norbert Wiener (1948)
The major cause of the software crisis is that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.
- Edsger W. Dijkstra (1972)
Program testing can be used to show the presence of bugs, but never to show their absence!
- Edsger W. Dijkstra (1969)
Programmers are always surrounded by complexity; we cannot avoid it. Our applications are complex because we are ambitious to use our computers in ever more sophisticated ways. Programming is complex because of the large number of conflicting objectives for each of our programming projects. If our basic tool, the language in which we design and code our programs, is also complicated, the language itself becomes part of the problem rather than part of its solution.
--- C.A.R. Hoare - The Emperor's Old Clothes - Turing Award Lecture (1980)
In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included.
- Edsger W. Dijkstra (1975)
assert, doctest, and unit testslogging, unittest, and nosewhile True print 'Hello world'
1/0
factorial
'1'+1
try:
file=open('test.txt')
except IOError:
print 'No such file'
def newfunction():
raise NotImplementedError
newfunction()
def foo(x):
return 1/x
def bar(y):
return foo(1-y)
bar(1)
%debug
def foo(x):
if x==0:
return float('Inf')
else:
return 1/x
bar(1)
bar(1.0)
def foo(x):
try:
return 1/x
except ZeroDivisionError:
return float('Inf')
bar(1)
bar(1.0)
i=raw_input("Please enter an integer: ")
if not isinstance(i,int):
print "Casting ", i, " to integer."
i=int(i)
if i%3 == 0:
print 1
elif i%3 == 1:
print 2
else:
assert i%3 == 2
print 3
Let's make a factorial function.
%%file myfactorial.py
def factorial2(n):
""" Details to come ...
"""
raise NotImplementedError
def test():
from math import factorial
for x in range(10):
print ".",
assert factorial2(x) == factorial(x), \
"My factorial function is incorrect for n = %i" % x
import myfactorial
myfactorial.test()
Looks like we will have to implement our function, if we want to make any progress...
%%file myfactorial.py
def factorial2(n):
""" Details to come ...
"""
if n == 0:
return 1
else:
return n*factorial2(n-1)
def test():
from math import factorial
for x in range(10):
assert factorial2(x) == factorial(x), \
"My factorial function is incorrect for n = %i" % x
reload(myfactorial)
myfactorial.test()
Seems to be okay so far. However, calling factorial2 with a negative number, say, will result in infinite loop. Thus:
What happens if we call factorial2 with a negative integer? Or something that's not an integer?
%%file myfactorial.py
def factorial2(n):
""" Find n!. Raise an AssertionError if n is negative or non-integral.
"""
assert n>=0. and type(n) is int, "Unrecognized input"
if n == 0:
return 1
else:
return n*factorial2(n-1)
def test():
from math import factorial
for x in range(10):
assert factorial2(x) == factorial(x), \
"My factorial function is incorrect for n = %i" % x
doctests -- executable examplesreload(myfactorial)
from myfactorial import factorial2
[factorial2(n) for n in range(5)]
%%file myfactorial.py
def factorial2(n):
""" Find n!. Raise an AssertionError if n is negative or non-integral.
>>> from myfactorial import factorial2
>>> [factorial2(n) for n in range(5)]
[1, 1, 2, 6, 24]
"""
assert n >= 0. and type(n) is int, "Unrecognized input"
if n == 0:
return 1
else:
return n * factorial2(n - 1)
def test():
from math import factorial
for x in range(10):
assert factorial2(x) == factorial(x), \
"My factorial function is incorrect for n = %i" % x
!python -m doctest -v myfactorial.py
unittest and nosenosetests, py.testtest in a module beginning with testSuch libraries have often testing routines, for example:
import scipy.integrate
scipy.integrate.test()
Mathematically
\(x = (\sqrt(x))^2\).
So what is happening here:
import math
assert 2 == math.sqrt(2)**2
math.sqrt(2)**2
What if we consider x and y almost equal? Can we modify our assertion?
import numpy as np
np.testing.assert_almost_equal(2, math.sqrt(2) ** 2)
x=1.000001
y=1.000002
np.testing.assert_almost_equal(x, y, decimal=5)
Going beyond doctest and Unittest, there are two frameworks widely spread for regression testing:
nose (http://nose.readthedocs.org/en/latest/)
pytest (http://pytest.org)
Here, we focus on pytest.
The example we use is the myfactorial.py file created earlier:
%load myfactorial.py
def factorial2(n):
""" Find n!. Raise an AssertionError if n is negative or non-integral.
>>> from myfactorial import factorial2
>>> [factorial2(n) for n in range(5)]
[1, 1, 2, 6, 24]
"""
assert n >= 0. and type(n) is int, "Unrecognized input"
if n == 0:
return 1
else:
return n * factorial2(n - 1)
def test():
from math import factorial
for x in range(10):
assert factorial2(x) == factorial(x), \
"My factorial function is incorrect for n = %i" % x
(Addition to original notebook, Hans Fangohr, 21 Sep 2013)
py.test is an executable that will search through a given file and find all functions that start with test, and execute those. Any failed assertions are reported as errors.
For example, py.test can run the test() function that has been defined already in myfactorial:
!py.test myfactorial.py
This output (the '.' after myfactorial.py) indicates success. We can get a more detailed output using the -v switch for extra verbosity:
!py.test -v myfactorial.py
Sometimes, we like having the tests for myfactorial.py gathered in a separate file, for example in test_myfactorial.py. We create such a file, and within the file we create a number of test functions, each with a name starting with test:
%%file test_myfactorial.py
from myfactorial import factorial2
def test_basics():
assert factorial2(0) == 1
assert factorial2(1) == 1
assert factorial2(3) == 6
def test_against_standard_lib():
import math
for i in range(20):
assert math.factorial(i) == factorial2(i)
def test_negative_number_raises_error():
import pytest
with pytest.raises(AssertionError): # this will pass if
factorial2(-1) # factorial2(-1) raises
# an AssertionError
with pytest.raises(AssertionError):
factorial2(-10)
We can now run the tests in this file using
!py.test -v test_myfactorial.py
The py.test command can also be given a directory, and it will search all files and files in subdirectories for files starting with test, and will attempt to run all the tests in those.
Or we can provide a list of test files to work through:
!py.test -v test_myfactorial.py myfactorial.py
!hg tip