{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Testing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Original notebook](https://github.com/profjsb/python-bootcamp/blob/master/Lectures/12_Testing/12_testing.ipynb) by Jarrod Millman, part of the Python-bootcamp.\n", "\n", "Modifications Hans Fangohr, Sept 2013:\n", "\n", "- Add py.test example\n", "- minor edits" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Motivation\n", "\n", "### Computing is error prone\n", "\n", "> In ordinary computational practice by hand or by desk machines, it is the\n", "> custom to check every step of the computation and, when an error is found,\n", "> to localize it by a backward process starting from the first point where the\n", "> error is noted.\n", ">\n", "> - Norbert Wiener (1948)\n", "\n", "### More computing, more problems\n", "\n", "> The major cause of the **software crisis** is that the machines have become\n", "> several orders of magnitude more powerful! To put it quite bluntly: as long\n", "> as there were no machines, programming was no problem at all; when we had a\n", "> few weak computers, programming became a mild problem, and now we have\n", "> gigantic computers, programming has become an equally gigantic problem.\n", ">\n", "> - Edsger W. Dijkstra (1972)\n", "\n", "## What testing is and is not...\n", "\n", "### Testing and debugging\n", "\n", "- debugging is what you do when you know a program is broken\n", "- testing is a determined, systematic attempt to break a program\n", "- writing tests is more interesting than debugging\n", "\n", "### Program correctness\n", "\n", "> Program testing can be used to show the presence of bugs, but never to show\n", "> their absence!\n", ">\n", "> - Edsger W. Dijkstra (1969)\n", "\n", "### In the imperfect world ...\n", "\n", "- avoid writing code if possible\n", "- write code as simple as possible\n", "- avoid cleverness\n", "- use code to generate code\n", "\n", "### Program languages play an important role\n", "\n", "> Programmers are always surrounded by complexity; we cannot avoid it. Our\n", "> applications are complex because we are ambitious to use our computers in\n", "> ever more sophisticated ways. Programming is complex because of the large\n", "> number of conflicting objectives for each of our programming projects. **If\n", "> our basic tool, the language in which we design and code our programs, is\n", "> also complicated, the language itself becomes part of the problem rather than\n", "> part of its solution.**\n", ">\n", "> --- C.A.R. Hoare - The Emperor's Old Clothes - Turing Award Lecture (1980)\n", "\n", "### Testing and reproducibility\n", "\n", "> In the good old days physicists repeated each other's experiments, just to\n", "> be sure. Today they stick to FORTRAN, so that they can share each other's\n", "> programs, bugs included.\n", ">\n", "> - Edsger W. Dijkstra (1975)\n", "\n", "### Pre- and post-condition tests\n", "\n", "- what must be true *before* a method is invoked\n", "- what must be true *after* a method is invoked\n", "- use assertions\n", "\n", "### Program defensively\n", "\n", "- out-of-range index\n", "- division by zero\n", "- error returns\n", "\n", "### Be systematic\n", "\n", "- incremental\n", "- simple things first\n", "- know what to expect\n", "- compare independent implementations\n", "\n", "### Automate it\n", "\n", "- **regression tests** ensure that changes don't break existing functionality\n", "- verify conservation\n", "- **unit tests** (white box testing)\n", "- measure test coverage\n", "\n", "### Interface and implementation\n", "\n", "- an **interface** is how something is used\n", "- an **implementation** is how it is written\n", "\n", "## Testing in Python\n", "\n", "### Landscape\n", "\n", "- errors, exceptions, and debugging\n", "- `assert`, `doctest`, and unit tests\n", "- `logging`, `unittest`, and `nose`\n", "\n", "### Errors & Exceptions\n", "\n", "#### Syntax Errors\n", "\n", "- Caught by Python parser, prior to execution\n", "- arrow marks the last parsed command / syntax, which gave an error" ] }, { "cell_type": "code", "collapsed": false, "input": [ "while True print 'Hello world'" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "SyntaxError", "evalue": "invalid syntax (, line 1)", "output_type": "pyerr", "traceback": [ "\u001b[0;36m File \u001b[0;32m\"\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m while True print 'Hello world'\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n" ] } ], "prompt_number": 2 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Exceptions\n", "\n", "- Caught during runtime" ] }, { "cell_type": "code", "collapsed": true, "input": [ "1/0" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "ZeroDivisionError", "evalue": "integer division or modulo by zero", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m: integer division or modulo by zero" ] } ], "prompt_number": 3 }, { "cell_type": "code", "collapsed": false, "input": [ "factorial" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "NameError", "evalue": "name 'factorial' is not defined", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfactorial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mNameError\u001b[0m: name 'factorial' is not defined" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "'1'+1" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "TypeError", "evalue": "cannot concatenate 'str' and 'int' objects", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;34m'1'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mTypeError\u001b[0m: cannot concatenate 'str' and 'int' objects" ] } ], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Exception handling" ] }, { "cell_type": "code", "collapsed": false, "input": [ "try:\n", " file=open('test.txt')\n", "except IOError:\n", " print 'No such file'" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 6 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Raising exceptions" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def newfunction():\n", " raise NotImplementedError\n", "\n", "newfunction()" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "NotImplementedError", "evalue": "", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mnewfunction\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNotImplementedError\u001b[0m: " ] } ], "prompt_number": 7 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Debugging" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def foo(x):\n", " return 1/x\n", "\n", "def bar(y):\n", " return foo(1-y)\n", "\n", "bar(1)" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "ZeroDivisionError", "evalue": "integer division or modulo by zero", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mbar\u001b[0;34m(y)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mfoo\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mZeroDivisionError\u001b[0m: integer division or modulo by zero" ] } ], "prompt_number": 10 }, { "cell_type": "code", "collapsed": false, "input": [ "%debug" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", "KeyboardInterrupt\n", "> \u001b[0;32m\u001b[0m(2)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 2 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "stream": "stdout", "text": [ "ipdb> x # is x really 0??\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "0\n" ] }, { "name": "stdout", "output_type": "stream", "stream": "stdout", "text": [ "ipdb> exit # okay, it is -> that's the problem\n" ] } ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fixing bugs " ] }, { "cell_type": "code", "collapsed": false, "input": [ "def foo(x):\n", " if x==0:\n", " return float('Inf')\n", " else:\n", " return 1/x\n", "\n", "bar(1)\n", "bar(1.0)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 11, "text": [ "inf" ] } ], "prompt_number": 11 }, { "cell_type": "code", "collapsed": false, "input": [ "def foo(x):\n", " try:\n", " return 1/x\n", " except ZeroDivisionError:\n", " return float('Inf')\n", "\n", "bar(1)\n", "bar(1.0)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 12, "text": [ "inf" ] } ], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test as you code\n", "\n", "### Type checking " ] }, { "cell_type": "code", "collapsed": false, "input": [ "i=raw_input(\"Please enter an integer: \")\n", "if not isinstance(i,int):\n", " print \"Casting \", i, \" to integer.\"\n", " i=int(i)" ], "language": "python", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "stream": "stdout", "text": [ "Please enter an integer: 5\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "Casting 5 to integer.\n" ] } ], "prompt_number": 14 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assert invariants" ] }, { "cell_type": "code", "collapsed": false, "input": [ "if i%3 == 0:\n", " print 1\n", "elif i%3 == 1:\n", " print 2\n", "else:\n", " assert i%3 == 2\n", " print 3" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "3\n" ] } ], "prompt_number": 15 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example\n", "\n", "Let's make a factorial function." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file myfactorial.py\n", "\n", "def factorial2(n):\n", " \"\"\" Details to come ...\n", " \"\"\"\n", "\n", " raise NotImplementedError\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " print \".\",\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Writing myfactorial.py\n" ] } ], "prompt_number": 16 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's test it ..." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import myfactorial\n", "myfactorial.test()" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "NotImplementedError", "evalue": "", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmyfactorial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmyfactorial\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/Users/fangohr/hgdocs/teaching/python/notebook/myfactorial.py\u001b[0m in \u001b[0;36mtest\u001b[0;34m()\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0;32mprint\u001b[0m \u001b[0;34m\".\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mfactorial2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mfactorial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 13\u001b[0m \u001b[0;34m\"My factorial function is incorrect for n = %i\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Users/fangohr/hgdocs/teaching/python/notebook/myfactorial.py\u001b[0m in \u001b[0;36mfactorial2\u001b[0;34m(n)\u001b[0m\n\u001b[1;32m 4\u001b[0m \"\"\"\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNotImplementedError\u001b[0m: " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks like we will have to implement our function, if we want to make any progress..." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file myfactorial.py\n", "\n", "def factorial2(n):\n", " \"\"\" Details to come ...\n", " \"\"\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n*factorial2(n-1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ " Overwriting myfactorial.py\n" ] } ], "prompt_number": 18 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's test it ..." ] }, { "cell_type": "code", "collapsed": false, "input": [ "reload(myfactorial)\n", "myfactorial.test()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 19 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Seems to be okay so far. However, calling ``factorial2`` with a negative number, say, will result in infinite loop. Thus:\n", "\n", "### What about preconditions\n", "\n", "What happens if we call `factorial2` with a negative integer? Or something that's not an integer?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", " \"\"\"\n", "\n", " assert n>=0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n*factorial2(n-1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting myfactorial.py\n" ] } ], "prompt_number": 20 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `doctests` -- executable examples" ] }, { "cell_type": "code", "collapsed": false, "input": [ "reload(myfactorial)\n", "from myfactorial import factorial2\n", "[factorial2(n) for n in range(5)]" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 21, "text": [ "[1, 1, 2, 6, 24]" ] } ], "prompt_number": 21 }, { "cell_type": "code", "collapsed": false, "input": [ "%%file myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Overwriting myfactorial.py\n" ] } ], "prompt_number": 22 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running doctests" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!python -m doctest -v myfactorial.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Trying:\r\n", " from myfactorial import factorial2\r\n", "Expecting nothing\r\n", "ok\r\n", "Trying:\r\n", " [factorial2(n) for n in range(5)]\r\n", "Expecting:\r\n", " [1, 1, 2, 6, 24]\r\n", "ok\r\n", "2 items had no tests:\r\n", " myfactorial\r\n", " myfactorial.test\r\n", "1 items passed all tests:\r\n", " 2 tests in myfactorial.factorial2\r\n", "2 tests in 3 items.\r\n", "2 passed and 0 failed.\r\n", "Test passed.\r\n" ] } ], "prompt_number": 23 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Real world testing and continuous integration\n", "\n", "### `unittest` and `nose`\n", "\n", "#### Test fixtures (Unittest)\n", "\n", "- create self-contained tests\n", "- setup: open file, connect to a DB, create datastructures\n", "- teardown: tidy up afterward\n", "\n", "#### Test runner (nose, pytest)\n", "\n", "- `nosetests`, `py.test`\n", "- test discovery: any callable beginning with `test` in a module\n", " beginning with `test`\n", "\n", "#### Testing scientific computing libraries\n", "\n", "Such libraries have often testing routines, for example:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import scipy.integrate\n", "scipy.integrate.test()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Running unit tests for scipy.integrate\n", "NumPy version 1.7.1" ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "K." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stderr", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "NumPy is installed in /Users/fangohr/anaconda/python.app/Contents/lib/python2.7/site-packages/numpy\n", "SciPy version 0.12.0\n", "SciPy is installed in /Users/fangohr/anaconda/python.app/Contents/lib/python2.7/site-packages/scipy\n", "Python version 2.7.5 |Anaconda 1.7.0 (x86_64)| (default, Jun 28 2013, 22:20:13) [GCC 4.0.1 (Apple Inc. build 5493)]\n", "nose version 1.3.0\n" ] }, { "output_type": "stream", "stream": "stderr", "text": [ "\n", "----------------------------------------------------------------------\n", "Ran 55 tests in 0.408s\n", "\n", "OK (KNOWNFAIL=1)\n" ] }, { "metadata": {}, "output_type": "pyout", "prompt_number": 24, "text": [ "" ] } ], "prompt_number": 24 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assertions revisited - numerical mathematics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mathematically\n", "\n", "$ x = (\\sqrt(x))^2$.\n", "\n", "So what is happening here:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import math\n", "assert 2 == math.sqrt(2)**2" ], "language": "python", "metadata": {}, "outputs": [ { "ename": "AssertionError", "evalue": "", "output_type": "pyerr", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0;36m2\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m: " ] } ], "prompt_number": 25 }, { "cell_type": "code", "collapsed": false, "input": [ "math.sqrt(2)**2" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 26, "text": [ "2.0000000000000004" ] } ], "prompt_number": 26 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### NumPy Testing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we consider x and y almost equal? Can we modify our assertion?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import numpy as np\n", "np.testing.assert_almost_equal(2, math.sqrt(2) ** 2)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 32 }, { "cell_type": "code", "collapsed": false, "input": [ "x=1.000001\n", "y=1.000002\n", "np.testing.assert_almost_equal(x, y, decimal=5)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 33 }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Testing with py.test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Going beyond ``doctest`` and Unittest, there are two frameworks widely spread for regression testing: \n", "\n", "* nose (http://nose.readthedocs.org/en/latest/)\n", "\n", "* pytest (http://pytest.org)\n", "\n", "Here, we focus on pytest.\n", "\n", "The example we use is the ``myfactorial.py`` file created earlier:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "%load myfactorial.py" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 34 }, { "cell_type": "code", "collapsed": false, "input": [ "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 35 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Providing test functions\n", "\n", "(Addition to original notebook, Hans Fangohr, 21 Sep 2013)\n", "\n", "py.test is an executable that will search through a given file and find all functions that start with ``test``, and execute those. Any failed assertions are reported as errors.\n", "\n", "For example, ``py.test`` can run the ``test()`` function that has been defined already in ``myfactorial``:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!py.test myfactorial.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 2.7.5 -- pytest-2.3.5\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 1 items\u001b[0m\u001b[1m\r", "collected 1 items \r\n", "\u001b[0m\r\n", "myfactorial.py .\r\n", "\r\n", "\u001b[1m=========================== 1 passed in 0.01 seconds ===========================\u001b[0m\r\n" ] } ], "prompt_number": 36 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This output (the '.' after ``myfactorial.py``) indicates success. We can get a more detailed output using the ``-v`` switch for extra verbosity:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!py.test -v myfactorial.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 2.7.5 -- pytest-2.3.5 -- /Users/fangohr/anaconda/bin/python\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 1 items\u001b[0m\u001b[1m\r", "collected 1 items \r\n", "\u001b[0m\r\n", "myfactorial.py:16: test \u001b[32mPASSED\u001b[0m\r\n", "\r\n", "\u001b[1m=========================== 1 passed in 0.01 seconds ===========================\u001b[0m\r\n" ] } ], "prompt_number": 37 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, we like having the tests for ``myfactorial.py`` gathered in a separate file, for example in ``test_myfactorial.py``. We create such a file, and within the file we create a number of test functions, each with a name starting with ``test``:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "%%file test_myfactorial.py\n", "\n", "from myfactorial import factorial2 \n", "\n", "def test_basics():\n", " assert factorial2(0) == 1\n", " assert factorial2(1) == 1\n", " assert factorial2(3) == 6\n", " \n", "def test_against_standard_lib():\n", " import math\n", " for i in range(20):\n", " assert math.factorial(i) == factorial2(i)\n", " \n", "def test_negative_number_raises_error():\n", " import pytest\n", "\n", " with pytest.raises(AssertionError): # this will pass if \n", " factorial2(-1) # factorial2(-1) raises \n", " # an AssertionError\n", " \n", " with pytest.raises(AssertionError):\n", " factorial2(-10)\n", " " ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Writing test_myfactorial.py\n" ] } ], "prompt_number": 38 }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now run the tests in this file using" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!py.test -v test_myfactorial.py " ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 2.7.5 -- pytest-2.3.5 -- /Users/fangohr/anaconda/bin/python\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 3 items\u001b[0m\u001b[1m\r", "collected 3 items \r\n", "\u001b[0m\r\n", "test_myfactorial.py:4: test_basics \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py:9: test_against_standard_lib \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py:14: test_negative_number_raises_error \u001b[32mPASSED\u001b[0m\r\n", "\r\n", "\u001b[1m=========================== 3 passed in 0.02 seconds ===========================\u001b[0m\r\n" ] } ], "prompt_number": 39 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ``py.test`` command can also be given a directory, and it will search all files and files in subdirectories for files starting with ``test``, and will attempt to run all the tests in those. \n", "\n", "Or we can provide a list of test files to work through:\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!py.test -v test_myfactorial.py myfactorial.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 2.7.5 -- pytest-2.3.5 -- /Users/fangohr/anaconda/bin/python\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 3 items\u001b[0m\u001b[1m\r", "collecting 4 items\u001b[0m\u001b[1m\r", "collected 4 items \r\n", "\u001b[0m\r\n", "test_myfactorial.py:4: test_basics \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py:9: test_against_standard_lib " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py:14: test_negative_number_raises_error \u001b[32mPASSED\u001b[0m\r\n", "myfactorial.py:16: test \u001b[32mPASSED\u001b[0m\r\n", "\r\n", "\u001b[1m=========================== 4 passed in 0.02 seconds ===========================\u001b[0m\r\n" ] } ], "prompt_number": 40 }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Final thoughts\n", "\n", "- [Continuous integration (SymPy)](https://github.com/sympy/sympy/pull/2399)\n", "- [Logging](http://docs.python.org/2/library/logging.html)\n", "- [Python debugger](http://docs.python.org/2/library/pdb.html)\n", "\n", "#### Learn more\n", "\n", "* [http://software-carpentry.org](http://software-carpentry.org)\n", "* [http://docs.python.org/library/exceptions.html](http://docs.python.org/library/exceptions.html)\n", "* [http://docs.python.org/library/doctest.html](http://docs.python.org/library/doctest.html)\n", "* [http://docs.python.org/library/unittest.html](http://docs.python.org/library/unittest.html)\n", "* [http://docs.scipy.org/doc/numpy/reference/routines.testing.html](http://docs.scipy.org/doc/numpy/reference/routines.testing.html)\n", "* [http://nedbatchelder.com/code/coverage](http://nedbatchelder.com/code/coverage)\n", "* [http://somethingaboutorange.com/mrl/projects/nose](http://somethingaboutorange.com/mrl/projects/nose)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!hg tip" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "changeset: 283:d3ff4ebdb13d\r\n", "tag: tip\r\n", "user: Hans Fangohr [MBA] \r\n", "date: Sat Sep 21 13:13:28 2013 +0100\r\n", "summary: Adding IPython Notebooks to support materials, updated notebooks.\r\n", "\r\n" ] } ], "prompt_number": 41 } ], "metadata": {} } ] }