Interacting with dask workers via IPython¶

Experimentation/illustration for https://github.com/dask/distributed/pull/370

In [1]:

from distributed import Executor

e = Executor('127.0.0.1:8786')
e

Out[1]:

<Executor: scheduler="127.0.0.1:8786" processes=2 cores=8>

In [2]:

futures = e.map(lambda x: 1/(x-5), range(10))
futures

Out[2]:

[<Future: status: pending, key: <lambda>-de7611e4ed920ad8c9a767c463dcbb65>,
 <Future: status: pending, key: <lambda>-1958825fcb9738b652a1a38c77efa7d6>,
 <Future: status: pending, key: <lambda>-4b6c95b99a5b37af7160c6c07bd8963e>,
 <Future: status: pending, key: <lambda>-1747fdb7eb7cd577fa5cc527964f96e6>,
 <Future: status: pending, key: <lambda>-1267f5ed15de96509325a074c362ca5e>,
 <Future: status: pending, key: <lambda>-786555fd0653a80aaaddeb882f4075b0>,
 <Future: status: pending, key: <lambda>-5c1535a32dbd218b6259a0f6a66ee5bf>,
 <Future: status: pending, key: <lambda>-d61a5c11b7520ad75ade4d408488e31d>,
 <Future: status: pending, key: <lambda>-2c6aa9b4d2e58aad942d33ce1c72f3bc>,
 <Future: status: pending, key: <lambda>-2e8510267dc20d551dfde05ee5a7d3c4>]

Executor.start_ipython() starts IPython on workers, returning the connection information needed to connect Jupyter clients to them.

In [3]:

info = e.start_ipython()
info

Out[3]:

{'10.34.24.251:62619': {'control_port': 64470,
  'hb_port': 64473,
  'iopub_port': 64471,
  'ip': '10.34.24.251',
  'key': b'9f4b7af2-9e02-4822-b139-fd4da6168316',
  'shell_port': 64467,
  'signature_scheme': 'hmac-sha256',
  'stdin_port': 64469,
  'transport': 'tcp'},
 '10.34.24.251:63122': {'control_port': 64476,
  'hb_port': 64479,
  'iopub_port': 64477,
  'ip': '10.34.24.251',
  'key': b'1df5f45f-d303-4fda-ae05-5e958147b998',
  'shell_port': 64474,
  'signature_scheme': 'hmac-sha256',
  'stdin_port': 64475,
  'transport': 'tcp'}}

If you pass qtconsole=True, then dask will spawn a local Juptyer QtConsole, connected to the remote worker(s).

The worker object is injected into the worker namespace as worker.

In [4]:

workers = list(e.ncores())
info = e.start_ipython(workers[0], qtconsole=True)

In [8]:

from IPython.display import Image
Image('./qtc.png')

Out[8]:

If we pass magic_names as a list of strings of the same length, we will get a line and cell magic that redirects execution to the given worker.

In [5]:

info = e.start_ipython(workers[:2], magic_names=['w1', 'w2'])

In [6]:

%w1 os.getpid()
%w2 os.getpid()
os.getpid()

Out[6]:

Since the remote kernel is a full IPython instance, we can even do plotting with matplotlib and see the figures:

In [7]:

%%w1
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(list(worker.data.values()))

Out[7]:

[<matplotlib.lines.Line2D at 0x1155c9a20>]