nbdev tutorial¶

A step by step guide

• image: images/nbdev_source.gif

nbdev is a system for exploratory programming. See the nbdev launch post for information about what that means. In practice, programming in this way can feel very different to the kind of programming many of you will be familiar with, since we've mainly be taught coding techniques that are (at least implicitly) tied to the underlying tools we have access to. I've found that programming in a "notebook first" way can make me 2-3x more productive than I was before (when we used vscode, Visual Studio, vim, PyCharm, and similar tools).

In this tutorial, I'll try to get you up and running with the basics of the nbdev system as quickly and easily as possible. You can also watch this video in which I take you through the tutorial, step by step (to view full screen, click the little square in the bottom right of the video; to view in a separate Youtube window, click the Youtube logo):

Set up Your Jupyter Server¶

Jupyter Environment¶

To complete this tutorial, you'll need a Jupyter Notebook Server configured on your machine. If you have not installed Jupyter before, you may find the Anaconda Individual Edition the simplest to install.

If you already have experience with Jupyter, please note that everything in this tutorial must be run using the same kernel.

Install nbdev¶

No matter how you installed Jupyter, you'll need to manually install nbdev. There is not a conda package for nbdev, so you'll need to use pip to install it. And you'll need to do that from a terminal window.

Jupyter notebook has a terminal window available, so we'll use that:

1. Start jupyter notebook
2. From the "New" dropdown on the right side, choose Terminal.
3. Enter "python -m pip install nbdev"

When the command completes, you're ready to go.

Set up Repo¶

Template¶

To create your new project repo, click here: nbdev template (you need to be logged in to GitHub for this link to work). Fill in the requested info and click Create repository from template.

NB: The name of your project will become the name of the Python package generated by nbdev. For that reason, it is a good idea to pick a short, all-lowercase name with no dashes between words (underscores are allowed).

Now, open your terminal, and clone the repo you just created.

Github pages¶

The nbdev system uses jekyll for documentation. Because GitHub Pages supports Jekyll, you can host your site for free on Github Pages without any additional setup, so this is the approach we recommend (but it's not required; any jekyll hosting will work fine).

To enable Github Pages in your project repo, click Settings in Github, then scroll down to Github Pages, and set "Source" to Master branch /docs folder. Once you've saved, if you scroll back down to that section, Github will have a link to your new website. Copy that URL, and then go back to your main repo page, click "edit" next to the description and paste the URL into the "website" section. While you're there, go ahead and put in your project description too.

Edit settings.ini¶

Next, edit the settings.ini file in your cloned repo. This file contains all the necessary information for when you'll be ready to package your library. The basic structure (that can be personalized provided you change the relevant information in settings.ini) is that the root of the repo will contain your notebooks, the docs folder will contain your auto-generated docs, and a folder with a name you select will contain your auto-generated modules.

You'll see these commented out lines in settings.ini. Uncomment them, and set each value as needed.

# lib_name = your_project_name
# user = your_github_username
# description = A description of your project
# keywords = some keywords
# author = Your Name
# author_email = email@example.com
# copyright = Your Name or Company Name

We'll see some other settings we can change later.

Install git hooks¶

Jupyter Notebooks can cause challenges with git conflicts, but life becomes much easier when you use nbdev. As a first step, run nbdev_install_git_hooks in the terminal from your project folder. This will set up git hooks which will remove metadata from your notebooks when you commit, greatly reducing the chance you have a conflict.

But if you do get a conflict later, simply run nbdev_fix_merge filename.ipynb. This will replace any conflicts in cell outputs with your version, and if there are conflicts in input cells, then both cells will be included in the merged file, along with standard conflict markers (e.g. =====). Then you can open the notebook in Jupyter and choose which version to keep.

Edit 00_core.ipynb¶

Now, run jupyter notebook, and click 00_core.ipynb (you don't have to start your notebook names with a number like we do here; but we find it helpful to show the order you've created your project in). You'll see something that looks a bit like this:

#default_exp core

module name here

API details.

Let's explain what these special cells mean.

Module name and summary¶

The markdown cell uses special syntax to define the title and summary of your module. Feel free to replace "module name here" with a title and "API details." with a summary for your module.

This cell can optionally be used to define jekyll metadata, see get_metadata for more details.

Add a function¶

Let's add a function to this notebook, e.g.:

#export
def say_hello(to):
    "Say hello to somebody"
    return f'Hello {to}!'

Notice how it includes #export at the top - this means it will be included in your module, and documentation. The documentation will look like this:

Add examples and tests¶

It's a good idea to give an example of your function in action. Just include regular code cells, and they'll appear (with output) in the docs, e.g.:

Examples can output plots, images, etc, and they'll all appear in your docs, e.g.:

You can also include tests:

You should also add markdown headings as you create your notebook; one benefit of this is that a table of contents will be created in the documentation automatically.

Build lib¶

Now you can create your python module. To do so, just run nbdev_build_lib from the terminal when anywhere in your project folder.

$ nbdev_build_lib
Converted 00_core.ipynb.
Converted index.ipynb.

Edit index.ipynb¶

Now you're ready to create your documentation home page and readme file; these are both generated automatically from index.ipynb. So click on that to open it now.

You'll see that there's already a line there to import your library - change it to use the name you selected in settings.ini. Then, add information about how to use your module, including some examples. Remember, these examples should be actual notebook code cells with real outputs.

Build docs¶

Now you can create your documentation. To do so, just run nbdev_build_docs from the terminal when anywhere in your project folder.

$ nbdev_build_docs
converting: /home/jhoward/git/nbdev/nbs/00_core.ipynb
converting: /home/jhoward/git/nbdev/nbs/index.ipynb

Commit to Github¶

You can now git commit and git push. Wait a minute or two for Github to process your commit, and then head over to the Github website to look at your results.

CI¶

Back in your project's Github main page, click where it says 1 commit (or 2 commits or whatever). Hopefully, you'll see a green checkmark next to your latest commit. That means that your documentation site built correctly, and your module's tests all passed! This is checked for you using continuous integration (CI) with GitHub actions. This does the following:

• Checks the notebooks are readable
• Checks the notebooks have been cleaned of needless metadata to avoid merge conflicts
• Checks there is no diff between the notebooks and the exported library
• Runs the tests in your notebooks

Edit the file .github/workflows/main.yml if you need to modify any of the CI steps.

If you have a red cross, that means something failed. Click on the cross, then click Details, and you'll be able to see what failed.

View docs and readme¶

Once everything is passing, have a look at your readme in Github. You'll see that your index.ipynb file has been converted to a readme automatically.

Next, go to your documentation site (e.g. by clicking on the link next to the description that you created earlier). You should see that your index notebook has also been used here.

Congratulations, the basics are now all in place! Let's continue and use some more advanced functionality.

Add a class¶

Create a class in 00_core.ipynb as follows:

#export
class HelloSayer:
    "Say hello to `to` using `say_hello`"
    def __init__(self, to): self.to = to

    def say(self):
        "Do the saying"
        return say_hello(self.to)

This will automatically appear in the docs like this:

Document with show_doc¶

However, methods aren't automatically documented. To add method docs, use show_doc:

show_doc(HelloSayer.say)

And add some examples and/or tests:

Add links with backticks¶

Notice above there is a link from our new class documentation to our function. That's because we used backticks in the docstring:

"Say hello to `to` using `say_hello`"

These are automatically converted to hyperlinks wherever possible. For instance, here are hyperlinks to HelloSayer and say_hello created using backticks.

Set up autoreload¶

Since you'll be often updating your modules from one notebook, and using them in another, it's helpful if your notebook automatically reads in the new modules as soon as the python file changes. To make this happen, just add these lines to the top of your notebook:

%load_ext autoreload
%autoreload 2

Add in-notebook export cell¶

It's helpful to be able to export all your modules directly from a notebook, rather than going to the terminal to do it. All nbdev commands are available directly from a notebook in Python. Add this line to any cell and run it to export your modules (I normally make this the last cell of my scripts).

notebook2script()

Run tests in parallel¶

Before you push to github or make a release, you might want to run all your tests. nbdev can run all your notebooks in parallel to check for errors. Just run nbdev_test_nbs in a terminal.

(base) jhoward@usf3:~/git/nbdev$ nbdev_test_nbs
testing: /home/jhoward/git/nbdev/nbs/00_core.ipynb
testing: /home/jhoward/git/nbdev/nbs/index.ipynb
All tests are passing!

View docs locally¶

If you want to look at your docs locally before you push to Github, you can do so by running a jekyll server. First, install Jekyll by following these steps. Then, install the modules needed for serving nbdev docs by cding to the docs directory, and typing bundle install. Finally, cd back to your repo root and type make docs_serve. This will launch a server on port 4000 (by default) which you can connect to with your browser to view your docs.

If Github pages fails to build your docs, running locally with Jekyll is the easiest way to find out what the problem is.

Set up prerequisites¶

If your module requires other modules as dependencies, you can add those prerequisites to your settings.ini in the requirements section. The requirements should be separated by a space and if the module requires at least or at most a specific version of the requirement this may be specified here, too.

For example if your module requires the fastcore module of at least version 1.0.5, the torchvision module of at most version 0.7 and any version of matplotlib, then the prerequisites would look like this:

requirements = fastcore>=1.0.5 torchvision<0.7 matplotlib

Set up console scripts¶

Behind the scenes, nbdev uses that standard package setuptools for handling installation of modules. One very useful feature of setuptools is that it can automatically create cross-platform console scripts. nbdev surfaces this functionality; to use it, use the same format as setuptools, with whitespace between each script definition (if you have more than one).

console_scripts = nbdev_build_lib=nbdev.cli:nbdev_build_lib

Test with editable install¶

To test and use your modules in other projects, and use your console scripts (if you have any), the easiest approach is to use an editable install. To do this, cd to the root of your repo in the terminal, and type:

pip install -e .

(Note that the trailing period is important.) Your module changes will be automatically picked up without reinstalling. If you add any additional console scripts, you will need to run this command again.

Upload to pypi¶

If you want people to be able to install your project by just typing pip install your-project then you need to upload it to pypi. The good news is, we've already created a fully pypi compliant installer for your project! So all you need to do is register at pypi (click "Register" on pypi) if you haven't previously done so, and then create a file called ~/.pypirc with your login details. It should have these contents:

[pypi]
username = your_pypi_username
password = your_pypi_password

To upload your project to pypi, just type make release in your project root directory. Once it's complete, a link to your project on pypi will be printed.

NB: make release will automatically increment the version number in settings.py before pushing a new release to pypi. If you don't want to do this, run make pypi instead.

Install collapsible headings and toc2¶

There are two jupyter notebook extensions that I highly recommend when working with projects like this. They are:

• Collapsible headings: This lets you fold and unfold each section in your notebook, based on its markdown headings. You can also hit left to go to the start of a section, and right to go to the end
• TOC2: This adds a table of contents to your notebooks, which you can navigate either with the Navigate menu item it adds to your notebooks, or the TOC sidebar it adds. These can be modified and/or hidden using its settings.

Look at nbdev "source" for more ideas¶

Don't forget that nbdev itself is written in nbdev! It's a good place to look to see how fast.ai uses it in practice, and get a few tips. You'll find the nbdev notebooks here in the nbs folder on Github.