Using Parallel Magics¶

IPython has a few magics for working with your engines.

This assumes you have started an IPython cluster, either with the notebook interface, or the ipcluster/controller/engine commands.

In [1]:

from IPython import parallel
rc = parallel.Client()
dv = rc[:]
rc.ids

Out[1]:

[0, 1, 2, 3]

Creating a Client registers the parallel magics %px, %%px, %pxresult, pxconfig, and %autopx.
These magics are initially associated with a DirectView always associated with all currently registered engines.

Now we can execute code remotely with %px:

In [2]:

%px a=5

In [3]:

%px print a

[stdout:0] 5
[stdout:1] 5
[stdout:2] 5
[stdout:3] 5

In [4]:

%px a

Out[0:3]: 5

Out[1:3]: 5

Out[2:3]: 5

Out[3:3]: 5

In [5]:

with dv.sync_imports():
    import sys

importing sys on engine(s)

In [6]:

%px print >> sys.stderr, "ERROR"

[stderr:0] ERROR
[stderr:1] ERROR
[stderr:2] ERROR
[stderr:3] ERROR

You don't have to wait for results. The %pxconfig magic lets you change the default blocking/targets for the %px magics:

In [9]:

%pxconfig --noblock

In [10]:

%px import time
%px time.sleep(5)
%px time.time()

Out[10]:

<AsyncResult: execute>

But you will notice that this didn't output the result of the last command. For this, we have %pxresult, which displays the output of the latest request:

In [11]:

%pxresult

Out[0:10]: 1363142098.048282

Out[1:10]: 1363142098.055627

Out[2:10]: 1363142098.048737

Out[3:10]: 1363142098.057886

Remember, an IPython engine is IPython, so you can do magics remotely as well!

In [21]:

%px %pylab inline

[stdout:0] 
Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.
[stdout:1] 
Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.
[stdout:2] 
Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.
[stdout:3] 
Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

%%px can also be used as a cell magic, for submitting whole blocks. This one acceps --block and --noblock flags to specify the blocking behavior, though the default is unchanged.

In [22]:

dv.scatter('id', dv.targets, flatten=True)
dv['stride'] = len(dv)

In [23]:

%%px --block
print "    id=%s" % id
print "stride=%s" % stride

[stdout:0] 
    id=0
stride=4
[stdout:1] 
    id=1
stride=4
[stdout:2] 
    id=2
stride=4
[stdout:3] 
    id=3
stride=4

In [26]:

%%px --noblock
x = linspace(0,pi,1000)
for n in range(id, 12, stride):
    print n
    plt.plot(x,sin(n*x))
plt.title("Plot %i" % id);

Out[26]:

<AsyncResult: execute>

In [25]:

%pxresult

[stdout:0] 
0
4
8
[stdout:1] 
1
5
9
[stdout:2] 
2
6
10
[stdout:3] 
3
7
11

[output:0]

[output:1]

[output:2]

[output:3]

Out[0:20]: <matplotlib.text.Text at 0x10ad0e710>

Out[1:20]: <matplotlib.text.Text at 0x10e046710>

Out[2:20]: <matplotlib.text.Text at 0x10f814710>

Out[3:20]: <matplotlib.text.Text at 0x10b2c2710>

Parallel Exceptions¶

When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.

In [31]:

%%px
from numpy.random import random
A = random((100,100,'invalid shape'))

[0:execute]: 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)<ipython-input-26-2f86496ae392> in <module>()
      1 from numpy.random import random
----> 2 A = random((100,100,'invalid shape'))
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.RandomState.random_sample (numpy/random/mtrand/mtrand.c:5833)()
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.cont0_array (numpy/random/mtrand/mtrand.c:1370)()
ValueError: negative dimensions are not allowed

[1:execute]: 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)<ipython-input-26-2f86496ae392> in <module>()
      1 from numpy.random import random
----> 2 A = random((100,100,'invalid shape'))
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.RandomState.random_sample (numpy/random/mtrand/mtrand.c:5833)()
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.cont0_array (numpy/random/mtrand/mtrand.c:1370)()
ValueError: negative dimensions are not allowed

[2:execute]: 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)<ipython-input-26-2f86496ae392> in <module>()
      1 from numpy.random import random
----> 2 A = random((100,100,'invalid shape'))
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.RandomState.random_sample (numpy/random/mtrand/mtrand.c:5833)()
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.cont0_array (numpy/random/mtrand/mtrand.c:1370)()
ValueError: negative dimensions are not allowed

[3:execute]: 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)<ipython-input-26-2f86496ae392> in <module>()
      1 from numpy.random import random
----> 2 A = random((100,100,'invalid shape'))
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.RandomState.random_sample (numpy/random/mtrand/mtrand.c:5833)()
/Users/minrk/dev/py/numpy/numpy/random/mtrand.so in mtrand.cont0_array (numpy/random/mtrand/mtrand.c:1370)()
ValueError: negative dimensions are not allowed

Remote Cell Magics¶

Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.

In [40]:

%%px
%%bash
echo "remote parallel interactive bash!"
hostname
date
echo $$

[stdout:0] 
remote parallel interactive bash!
horus
Tue Mar 12 19:46:41 PDT 2013
58149
[stdout:1] 
remote parallel interactive bash!
horus
Tue Mar 12 19:46:41 PDT 2013
58150
[stdout:2] 
remote parallel interactive bash!
horus
Tue Mar 12 19:46:41 PDT 2013
58151
[stdout:3] 
remote parallel interactive bash!
horus
Tue Mar 12 19:46:41 PDT 2013
58152

In [41]:

%%px
%%ruby
puts 'hello from ruby'

[stdout:0] hello from ruby
[stdout:1] hello from ruby
[stdout:2] hello from ruby
[stdout:3] hello from ruby

In [39]:

dv.scatter('rank', dv.targets, flatten=True)

Out[39]:

<AsyncResult: scatter>

In [46]:

%%px
%%timeit
from numpy.random import random
from numpy.linalg import norm
N = 100 * (rank + 1)
A = random((N,N))
norm(A, 2)    

[stdout:0] 100 loops, best of 3: 1.97 ms per loop
[stdout:1] 100 loops, best of 3: 6.63 ms per loop
[stdout:2] 10 loops, best of 3: 33.2 ms per loop
[stdout:3] 10 loops, best of 3: 57.4 ms per loop

In [ ]: