The IPython notebook is part of the IPython project. The IPython project is one of the packacges making up the python scientific stack called SciPi. SciPy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering. In particular, these are some of the core packages:
IPython provides a rich architecture for interactive computing with:
interactive shells
(terminal and Qt-based).browser-based notebook
with support for code, text, mathematical expressions, inline plots and other rich media.interactive data visualization
and use of GUI toolkits.The main reasons I have been using it includes:
superior
shellmagic
functions makes life easier (magics gets called with a %, use %-tab to see them all)replacement shell
for Windows Shell or TerminalCode Completion
GNU Readline
based editing and command historyThe four most helpful commands, as well as their brief description, is shown to you in a banner, every time you start IPython:
command |
description |
---|---|
? | Introduction and overview of IPython's features. |
%quickref | Quick reference. |
help | Python's own help system. |
object? | Details about 'object', use 'object??' for extra details. |
The following code cells make sure that plotting is enabled and also loads a customised matplotlib confirguration file that spices up the inline plots. The custom matplotlib file has been taken from the Bayesian Methods for Hackers Project
# makes sure inline plotting is enabled
%pylab --no-import-all inline
Populating the interactive namespace from numpy and matplotlib
#loads a customer marplotlib configuration file
def CustomPlot():
import json
s = json.load( open("static/matplotlibrc.json") )
matplotlib.rcParams.update(s)
figsize(18, 6)
The code cell below is an example of how you should not be chaning the layout and css of the notebook. From IPython V1.00 it is possible to include custom css by creating IPython profiles. Since this file needs to be distributable I have opted for the hack below as used by the Bayesian Methods for Hackers Team
from IPython.core.display import HTML
def css_styling():
styles = open("static/custom.css", "r").read()
return HTML(styles)
css_styling()
The IPython Notebook is a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document.
SHIFT+ENTER will run the contents of a cell and move to the next one
CTRL+ENTER run the cell in place and don't move to the next cell. (best for presenting)
CTRL-m h show keyboard shorcuts
# press shift-enter to run code
print "Hallo Pycon"
Hallo Pycon
CTRL-S will save the notebook
The %quickref
commmand can be used to obtain a bit more information
#IPython -- An enhanced Interactive Python - Quick Reference Card
%quickref # now press shift-ender
The cell below defines a function with a bit of a long name. By using the ?
command the docstring can we viewed. ??
will open up the source code. The autocomplete function is also demostrated, and for fun the function is called and the output displayed
# lets degine a function with a long name.
def long_silly_dummy_name(a, b):
"""
This is the docstring for dummy.
It takes two arguments a and b
It returns the sum of a and b
No error checking is done!
"""
return a+b
# lets get the docstring or some help
long_silly_dummy_name?
long_silly_dummy_name??
#press tab to autocplete
long_si
# press shift-enter to run
long_silly_dummy_name(5,6)
11
You need to activate the Cell Toolbar in the Toolbar above. Here you can set if this cell should be compiled as a slide or not. The options are given below:
You can set the contents type of a cell in the toolbar above. When Markdown is selected you can enter markdown in a cell and it's contents will be rendered as HTML. The markdown syntax can by found here
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
IPython has a set of predefined ‘magic functions’ that you can call with a command line style syntax. There are two kinds of magics, line-oriented and cell-oriented. Line magics are prefixed with the % character and work much like OS command-line calls: they get as an argument the rest of the line, where arguments are passed without parentheses or quotes. Cell magics are prefixed with a double %%, and they are functions that get as an argument not only the rest of the line, but also the lines below it in a separate argument.
The timeit magic can be used to evaluate the average time your loop or piece of code is taking to complete it's run.
%%timeit
x = 0 # setup
for i in range(100000): #lets use range here
x = x + i**2
100 loops, best of 3: 12.2 ms per loop
%%timeit
x = 0 # setup
for i in xrange(100000): #replace range with slightly improved xrange
x += i**2
100 loops, best of 3: 10.7 ms per loop
Have a look at the top right hand side of the notebook and run the code cell above again. This shows that the kernel is busy running the current cell.
In the snippet below it the raw_input()
function is used to read some user input to a variable raw
and printed to stdout.
from IPython.display import HTML
raw = raw_input("enter your input here >>> ")
print "Hallo, ",raw
enter your input here >>> World! Hallo, World!
from IPython.display import FileLink, FileLinks
FileLinks('.', notebook_display_formatter=True)
./ .DS_Store builddocs.ipy calling_r_example.ipynb calling_ruby_example.ipynb pycon13_ipython.ipynb README.md ./.ipynb_checkpoints/ calling_r_example-checkpoint.ipynb calling_ruby_example-checkpoint.ipynb pycon13_ipython-checkpoint.ipynb ./data/ CapeTown_2009_Temperatures.csv READEME.md ./output/ .DS_Store pycon13_ipython.html pycon13_ipython.slides.html pycon13_ipython_complete.pdf pycon13_ipython_pdf.pdf ./static/ .DS_Store custom.css frontpage.docx frontpage.pdf ip.png ip2.png matplotlibrc.json python-vs-java.jpg scistack.png
I now use ipython as my default shell scripting language. lets put the contents of the current directory into a list. by using the !
before a command indicates that you want to run a system command.
filelist = !ls #read the current directory into variable
for x,i in enumerate(filelist):
print '#',x, '--->', i
# 0 ---> README.md # 1 ---> builddocs.ipy # 2 ---> calling_r_example.ipynb # 3 ---> calling_ruby_example.ipynb # 4 ---> data # 5 ---> output # 6 ---> pycon13_ipython.ipynb # 7 ---> static
Image released under CC BY-NC-ND 2.5 IN) by Rhul Singh
from IPython.display import Image
Image('static/python-vs-java.jpg')
I am making the video small as it does not embed into the final output pdf.
from IPython.display import YouTubeVideo
YouTubeVideo('iwVvqwLDsJo', width=200, height=200)
matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. matplotlib can be used in python scripts, the python and ipython shell, web application servers, and six graphical user interface toolkits.
from matplotlib.pylab import xkcd
#xkcd()
CustomPlot()
from numpy import *
#generate some data
n = array([0,1,2,3,4,5])
xx = np.linspace(-0.75, 1., 100)
x = linspace(0, 5, 10)
y = x ** 2
fig, axes = plt.subplots(1, 4, figsize=(12,3))
axes[0].scatter(xx, xx + 0.25*randn(len(xx)))
axes[0].set_title('scatter')
axes[1].step(n, n**2, lw=2)
axes[1].set_title('step')
axes[2].bar(n, n**2, align="center", width=0.5, alpha=0.5)
axes[2].set_title('bar')
axes[3].fill_between(x, x**2, x**3, color="green", alpha=0.5);
axes[3].set_title('fill')
for i in range(4):
axes[i].set_xlabel('x')
axes[0].set_ylabel('y')
show()
CustomPlot()
font_size = 20
figsize(11.5, 6)
fig, ax = plt.subplots()
ax.plot(xx, xx**2, xx, xx**3)
ax.set_title(r"Combined Plot $y=x^2$ vs. $y=x^3$", fontsize = font_size)
ax.set_xlabel(r'$x$', fontsize = font_size)
ax.set_ylabel(r'$y$', fontsize = font_size)
fig.tight_layout()
# inset
inset_ax = fig.add_axes([0.29, 0.45, 0.35, 0.35]) # X, Y, width, height
inset_ax.plot(xx, xx**2, xx, xx**3)
inset_ax.set_title(r'zoom $x=0$',fontsize=font_size)
# set axis range
inset_ax.set_xlim(-.2, .2)
inset_ax.set_ylim(-.005, .01)
# set axis tick locations
inset_ax.set_yticks([0, 0.005, 0.01])
inset_ax.set_xticks([-0.1,0,.1]);
show()
CustomPlot()
figsize(11.5, 6)
font_size = 20
fig, ax = plt.subplots()
ax.plot(xx, xx**2, xx, xx**3)
ax.set_xlabel(r'$x$', fontsize = font_size)
ax.set_ylabel(r'$y$', fontsize = font_size)
ax.set_title(r"Adding Text $y=x^2$ vs. $y=x^3$", fontsize = font_size)
ax.text(0.15, 0.2, r"$y=x^2$", fontsize=font_size, color="blue")
ax.text(0.65, 0.1, r"$y=x^3$", fontsize=font_size, color="green");
matplolib v1.3 now includes a setting to make plots resemple xkcd styles.
from matplotlib import pyplot as plt
import numpy as np
plt.xkcd()
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
plt.xticks([])
plt.yticks([])
ax.set_ylim([-30, 10])
data = np.ones(100)
data[70:] -= np.arange(30)
plt.annotate(
'THE DAY I REALIZED\nI COULD COOK BACON\nWHENEVER I WANTED',
xy=(70, 1), arrowprops=dict(arrowstyle='->'), xytext=(15, -10))
plt.plot(data)
plt.xlabel('time')
plt.ylabel('my overall health')
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.bar([-0.125, 1.0-0.125], [0, 100], 0.25)
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.set_xticks([0, 1])
ax.set_xlim([-0.5, 1.5])
ax.set_ylim([0, 110])
ax.set_xticklabels(['CONFIRMED BY\nEXPERIMENT', 'REFUTED BY\nEXPERIMENT'])
plt.yticks([])
plt.title("CLAIMS OF SUPERNATURAL POWERS")
plt.show()
SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python and does not require any external libraries.
from sympy import *
init_printing(use_latex=True)
x = Symbol('x')
y = Symbol('y')
series(exp(x), x, 1, 5)
eq = ((x+y)**2 * (x+1))
eq
expand(eq)
a = 1/x + (x*sin(x) - 1)/x
a
simplify(a)
pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive
The two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering.
For R users, DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top of NumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.
from pandas import DataFrame, read_csv
Cape_Weather = DataFrame( read_csv('data/CapeTown_2009_Temperatures.csv' ))
Cape_Weather.head()
high | low | radiation | |
---|---|---|---|
0 | 25 | 16 | 29.0 |
1 | 23 | 15 | 25.7 |
2 | 25 | 15 | 21.5 |
3 | 26 | 16 | 15.2 |
4 | 26 | 17 | 10.8 |
CustomPlot()
figsize(11.5, 6)
font_size = 20
title('Cape Town temparature(2009)',fontsize = font_size)
xlabel('Day number',fontsize = font_size)
ylabel(r'Temperature [$^\circ C$] ',fontsize = font_size)
Cape_Weather.high.plot()
Cape_Weather.low.plot()
show()
CustomPlot()
figsize(11.5, 6)
font_size = 20
title( 'Mean solar radiation(horisontal plane)', fontsize=font_size)
xlabel('Month Number', fontsize = font_size)
ylabel(r'$MJ / day / m^2$',fontsize = font_size)
Cape_Weather.radiation.plot()
show()
# lets look at a proxy for heating degree and cooling degree days
level = 25
print Cape_Weather[ Cape_Weather['high'] > level ].count()
print Cape_Weather[ Cape_Weather['high'] <= level ].count()
high 59 low 59 radiation 5 dtype: int64 high 306 low 306 radiation 7 dtype: int64
# Basic descriptive statistics
print Cape_Weather['high'].describe()
count 365.000000 mean 21.545205 std 4.764943 min 12.000000 25% 18.000000 50% 21.000000 75% 25.000000 max 36.000000 dtype: float64
CustomPlot()
figsize(11.5, 6)
font_size = 20
title('Cape Town temperature distribution', fontsize=font_size)
ylabel('Day count',fontsize = font_size)
xlabel(r'Temperature [$^\circ C $] ',fontsize = font_size)
Cape_Weather['high'].hist(bins=6)
show()
from IPython.display import Math
Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx')
from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}""")
%pdb on
Automatic pdb calling has been turned ON
foo = 1
bar = 'a'
print foo+bar
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-41-08464a413a31> in <module>() 1 foo = 1 2 bar = 'a' ----> 3 print foo+bar TypeError: unsupported operand type(s) for +: 'int' and 'str'
> <ipython-input-41-08464a413a31>(3)<module>() 1 foo = 1 2 bar = 'a' ----> 3 print foo+bar ipdb> q
%load http://pastebin.com/raw.php?i=mGiV1FwY
CustomPlot()
font_size = 20
figsize(11.5, 6)
t = arange(0.0, 2.0, 0.01)
s = sin(2*pi*t)
plot(t, s)
xlabel(r'time $(s)$', fontsize=font_size)
ylabel('voltage $(mV)$', fontsize=font_size)
title('Voltage', fontsize=font_size)
grid(True)
from IPython.display import HTML
input_form = """
<div style="background-color:gainsboro; border:solid black; width:630px; padding:20px;">
Variable Name: <input type="text" id="var_name" value="var"><br>
Variable Value: <input type="text" id="var_value" value="val"><br>
<button onclick="set_value()">Set Value</button>
</div>
"""
javascript = """
<script type="text/Javascript">
function set_value(){
var var_name = document.getElementById('var_name').value;
var var_value = document.getElementById('var_value').value;
var command = var_name + " = '" + var_value + "'";
console.log("Executing Command: " + command);
var kernel = IPython.notebook.kernel;
kernel.execute(command);
}
</script>
"""
HTML(input_form + javascript)
qwerty
'foo'
It is possible to save spesific lines of code to a GitHub gist. This is achieved with the pastebin
magic as demonstrated below.
%pastebin "cell one" 0-10
u'https://gist.github.com/6651917'
Using the %connect_info
magic you can obtain the connection info to connect to this workbook from another ipython console or qtconsole using :
ipython qtconsole --existing
%connect_info
{ "stdin_port": 55291, "ip": "127.0.0.1", "control_port": 55292, "hb_port": 55293, "signature_scheme": "hmac-sha256", "key": "dcc990e7-2eeb-4c41-8099-a25cf2308be4", "shell_port": 55289, "transport": "tcp", "iopub_port": 55290 } Paste the above JSON into a file, and connect with: $> ipython <app> --existing <file> or, if you are local, you can connect with just: $> ipython <app> --existing kernel-5db61520-8ecd-492b-9afb-5c8e65985a19.json or even just: $> ipython <app> --existing if this is the most recent IPython session you have started.
Newly added in the 1.0 release of IPython is the nbconvert tool, which allows you to convert an .ipynb
notebook document file into various static formats.
Currently, nbconvert is provided as a command line tool, run as a script using IPython. A direct export capability from within the IPython Notebook web app is planned.
The command-line syntax to run the nbconvert script is: MORE OPTIONS
ipython nbconvert --to FORMAT notebook.ipynb
This page is converted and published to the following formats using this tool:
HTML
PDF (the PDF is created using wkhtml2pdf
that takes the html file as an input)
LATEX
Reveal.js
slideshow
You can even call the build script from the notebook. The script will convert this page to an html and slide file. It will also compile to PDF and stitch a front page to it. Some of the last text in the building process wont appear as this notebook is being updated as it is being compile. Maybe not the best idea but saved a lot of time...
!ipython builddocs.ipy;
print "Done"
The links below can be used to verify the output from the convertion process. This saved me a lot of time as I could just click below and have a look at the files without exiting the notebook.
FileLinks('output/')
The following notebooks showcase multiple aspects of IPython, from its basic use to more advanced scenarios. They introduce you to the use of the Notebook and also cover aspects of IPython that are available in other clients, such as the cell magics for multi-language integration or our extended display protocol.
For beginners, we recommend that you start with the 5-part series that introduces the system, and later read others as the topics interest you.
Once you are familiar with the notebook system, we encourage you to visit our gallery where you will find many more examples that cover areas from basic Python programming to advanced topics in scientific computing.
Since this talk focussed on the life cycle of the analysis to publication many of the code examples were taken from their respective websites. If I have not given credit at any point please let me know and I will make sure that the work is updated
Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, omputing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org3
Bayesian Methods for Hackers, the use of the custom css and also the custom matplotlib skin
The build script generates a slideshow version of this notebook and saves it in the output directory. You can also use normal HTML in a cell and using the iframe
tag the slideshow was embedded to a cell below. Since this document has not been build yet...we are editing it now, the slideshow below is linked to the previous saved version of this notebook. So if we did not make to many changes it should be pretty close to being the same thing.