from IPython.core.display import HTML
css_file = "./notebook_style.css"
HTML(open(css_file, 'r').read())
3. Continuous Integration & Code Coverage¶
import numpy
import matplotlib.pyplot as plt
%matplotlib inline
Continuous integration¶
Why?¶
So, you've written a set of tests for your code, you run them and everything passes - great! However, you then go back to work on developing your code and quickly forget about testing it. A few months later, after implementing several new features and refactoring several important functions, you remember to try testing your code again. You run the set of tests, only to find that they fail. At this point, you face trawling through months of changes to try and locate where the bugs were first introduced.
This could have been prevented if you had run your tests regularly, updating them often so that they reflect changes to your code. However, remembering to do this manually is less than ideal, plus it would be nice to have some kind of log of test results for each version of your code so you can avoid repeating past mistakes (as you can see things that broke the code in the past and avoid doing that again).
The solution to this is continuous integration. This is a way of automating your tests so that they are run regularly (e.g. every night, every time you push changes to a repository), generating reports of test results for you to peruse at leisure and notifying you (almost) instantly when tests fail. Assuming you have a comprehensive, effective set of tests, you can now go ahead and develop your code safe in the knowledge that if you break something, you should find out almost as soon as you have committed (and pushed) the change and therefore be able to fix it before the buggy code becomes too entrenched.
Travis CI¶
Travis CI is a remote continuous integration service that can be linked to your GitHub repository so that every time you push a change your tests get executed on a remote server. It will then generate a report detailing which tests passed and which failed and (if you wish) send you an email letting you know what happened.
For simple python projects, Travis CI is pretty straightforward to set up. First, set up your account by signing in to Travis CI with your GitHub account. Then go to your profile page and enable Travis CI for your chosen repository.
Next, you need to tell Travis what to do to test your project - how it should set up the remote server and which commands it should run to execute your tests. This is done by creating a .travis.yml file in your repository. For a python project, this file will look something like:
.travis.yml¶
yml
language: python # which language is our project written in
python: # which versions of python should the code be tested in
- "2.7"
- "3.5"
# command to install dependencies
install:
- pip install -r requirements.txt
# command to run tests
script: pytest # or py.test for Python versions 3.5 and below
Because travis will be testing our code on a remote server, we need to tell it if we have used any python libraries which are not included in the core python installation. For this example, we will assume that all the packages needed for our project are listed on PyPI so can be installed using pip. These packages are listed in the file requirements.txt. Once these packages have been installed, travis will then execute pytest. This particular .travis.yml file will test the code for python versions 2.7 and 3.5.
Finally, add this .travis.yml file to git, commit and push it to the remote repository. This will trigger a Travis CI build. You can check to see if the build passes or fails by checking the build status page.
Unfortunately, packages such as numpy are not included in the default python environment where travis does its testing. Therefore, if your project includes packages from the anaconda distribution, you'll need a slightly more complicated script such as the one below. This downloads and installs a barebones version of anaconda, miniconda, onto the remote server before it runs the tests.
.travis.yml¶
yml
language: python
python:
- "2.7"
- "3.5"
before_install:
- sudo apt-get update
# Setup anaconda
install:
# We do this conditionally because it saves us some downloading if the
# version is the same.
- if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
else
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
fi
- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- hash -r
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
# Useful for debugging any issues with conda
- conda info -a
- conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib ipython nose
- source activate test-environment
- pip install -r requirements.txt
# Run test
script:
- pytest
More advanced features for customising your travis build and a description of the travis build lifecycle can be found here.
Local CI¶
Remote continuous integration services are great for open source codes, however if your code is closed or uses non-standard hardware (e.g. GPUs), you may want instead to run your tests locally. Rather than manually running these tests every time you make changes to your code, you can use tools such as jenkins to manage this for you. Jenkins is an automation server which can be used to automate a variety of tasks including building and testing. Once set up, much like travis.ci, it will automatically run your tests every time you commit changes to your code and generate reports on whether your tests passed or failed.
Code coverage¶
So, you have written a test suite and are using continuous integration to run it regularly - awesome! However, how do you know that you are testing all parts of your code? If your tests only cover 20% of the code that executes, that is no guarantee that the other 80% is doing what it's supposed to, and therefore you still cannot trust your code's results. If you can't trust your results, how are you going to convince other people that the cool new result your code has produced is correct?
Fortunately, rather than trawling through your code, inspecting each line to see if it's being tested, there are code coverage tools that can automate this. These tools can be linked with your continuous integration service so that they are run whenever your tests execute, generating a .coveragerc report and allowing you to spot parts of your code that could do with some more testing.
There are code coverage libraries for most languages that will generate these coverage reports: coverage.py for python, gcov for C/C++, tcov for C/C++/fortran. Codecov is a nice tool that can then be used to analyse these coverage reports. It keeps track of code coverage for each version of the code, providing a graphical interface which highlights which exact lines are covered / not covered by testing, and generally provides a lot of graphs and features which make monitoring the coverage of your tests a lot easier. It is also particularly useful if your code is written in multiple languages, as it will combine the reports produced by each of the languages.
To run codecov on a python project, modify your .travis.yml file as follows:
With pytest¶
yml
language: python
python:
- "2.7"
- "3.5"
# command to install dependencies
install:
- pip install codecov
- pip install pytest-cov
# command to run tests
script:
- pytest --cov=./ # or py.test for Python versions 3.5 and below
# if tests were all successful, run codecov
after_sucess:
- codecov
With nose¶
yml
language: python
python:
- "2.7"
- "3.5"
# command to install dependencies
install:
- pip install codecov
- pip install coverage
# command to run tests
script:
- nosetests -- with-coverage
# if tests were all successful, run codecov
after_sucess:
- codecov
There may be a few sections of our code that do not make sense to test using automated testing and so we wish to exclude from coverage reports. For example, code that produces plots or graphical output would be difficult to write tests for and are perhaps better tested visually. We can exclude these sections from the coverage report using a .coveragerc file. For example, if all our plotting functions had names that began with plot_, our file would look like
.coveragerc¶
[report]
exclude_lines =
def plot_*