Jupyter and IPython Configuration Tips

Enabling the Host's Certificates

When you want to access REST API endpoints secured by TLS then you need to have the related certificates visible to the requests library, which is normally used inder the hood to estalish the TLS connections.

Call these commands in your user account to set relevant environment variables:

ipython profile create
cat >~/.ipython/profile_default/startup/00-ca-certificates.py <<'EOF'
import os as _os
_os.environ.setdefault('REQUESTS_CA_BUNDLE', '/etc/ssl/certs/ca-certificates.crt')
del _os
EOF

⚠️ If you have notebooks running, restart their kernel via the web UI to load those changes.

How to Provide Your Own Personal CSS

You can provide a custom.css file as part of the personal configuration in ~/.ipython:

mkdir -p ~/.ipython/profile_default/static/custom
test -f $_/custom.css || echo >$_ "div.input_area { background: #ddf; }"

The above is a complete example for input cells with a faint blu-ish background.

⚠️ Restart your kernel after applying changes.

Embedded Dependency Installation

Overview

To install the requirements of a notebook from within a code cell, you can use the %pip magic available with IPython 0.7.3 and up (in earlier versions, use !pip instead). This also makes sure that installing uncommon packages used in your notebook is documented, improving reproducability.

You need permission to write into pip's install prefix though, which depending on the exact runtime environment you won't have – especially with JupyterHub installations. Passing the --user option will install to a writable path, but then you might want to have your notebook packages separate from ones you use in the shell.

Therefore, installed packages go into ~/.ipython, but not directly into that directory, which would become a mess otherwise. Instead, we install into the usual lib/pythonX.X/site-packages hierarchy, and then make that visible to Python. An additional important advantage is that kernels using different Python versions co-exist peacefully.

%env magic can be used to point pip at a local repository server like devpi or Artifactory.

Install & Import Mechanics

The following code is pure boilerplate to provide the require function. In production environments, you probably want to have this in your underlying configuration, or as a custom %require magic.

In [1]:
%env PIP_INDEX_URL=https://pypi.org/pypi
%env PIP_PREFIX=~/.ipython

def require(spec, alias=None, pkgname=None):
    """ Helper for importing custom packages.
    
        Use ``alias`` to change the ‘imported-as’ name,
        and ``pkgname`` in cases where the package's name
        is different from the project name on PyPI.
        Both default to the name part from ``spec``.
    """
    import importlib, os, re, sys

    site_packages = os.path.expanduser(
        '~/.ipython/lib/python{v.major}.{v.minor}/site-packages'
        .format(v=sys.version_info))
    if site_packages not in sys.path:
        sys.path.insert(0, site_packages)

    name = re.split('[ ;,<>=]', spec)[0].replace('-', '_')
    pkgname = pkgname or name
    try:
        module = importlib.import_module(pkgname)
        print("⚠ Using already installed '{}' package."
              .format(pkgname))
    except ImportError:
        pip_opts = ' '.join([
            '-q',
            '--disable-pip-version-check',
            '--no-warn-script-location',
        ])
        %pip install {pip_opts} "{spec}"
        module = importlib.import_module(pkgname)

    globals()[alias or name] = module
    return module
env: PIP_INDEX_URL=https://pypi.org/pypi
env: PIP_PREFIX=~/.ipython

This code shows the extended Python search path with the user-specific site-packages directory upfront.

In [2]:
def prettify(path):
    import os
    return path.replace(os.path.expanduser('~/'), '~/')

require('sys')
[prettify(x) for x in sys.path]
⚠ Using already installed 'sys' package.
Out[2]:
['~/.ipython/lib/python3.6/site-packages',
 '/opt/venvs/jupyterhub/lib/python36.zip',
 '/opt/venvs/jupyterhub/lib/python3.6',
 '/opt/venvs/jupyterhub/lib/python3.6/lib-dynload',
 '/usr/lib/python3.6',
 '',
 '/opt/venvs/jupyterhub/lib/python3.6/site-packages',
 '/opt/venvs/jupyterhub/lib/python3.6/site-packages/IPython/extensions',
 '~/.ipython']

Defining & Using Requirements

Given the above code is executed once in a kernel, you can now replace a normal import with a require call, and the specified package is installed if not available yet.

In [3]:
require('distro>=1.4', 'do', pkgname='distro')
prettify(repr(do))
Note: you may need to restart the kernel to use updated packages.
Out[3]:
"<module 'distro' from '~/.ipython/lib/python3.6/site-packages/distro.py'>"

Finally, just use the package as usual.

In [4]:
do.info()
Out[4]:
{'id': 'ubuntu',
 'version': '18.04',
 'version_parts': {'major': '18', 'minor': '04', 'build_number': ''},
 'like': 'debian',
 'codename': 'bionic'}