Using `importnb` with Luigi¶

luigi is a very reliable task scheduling tool in Python. luigi has provides a command line interface.

It may be common for a data science to interactively develop an algorithm in the notebook. When the importnb-install has been applied, notebooks can be imported as modules.

This notebook shows how luigi can be used with a notebook module for a very simple example.

In [5]:

import luigi

class MyTask(luigi.Task):
    x = luigi.IntParameter()
    y = luigi.IntParameter(default=45)

    def run(self):
        print(self.x + self.y)

Run this notebook as module using luigi's command line interface.

In [8]:

if __name__ == '__main__':
    !ipython -m luigi -- --module __Luigi__ MyTask --x 100 --local-scheduler

DEBUG: Checking if MyTask(x=100, y=45) is complete
/Users/tonyfast/anaconda/envs/p6/lib/python3.6/site-packages/luigi/worker.py:346: UserWarning: Task MyTask(x=100, y=45) without outputs has no custom complete() method
  is_complete = task.complete()
INFO: Informed scheduler that task   MyTask_100_45_98a6d68a8b   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 19479] Worker Worker(salt=172344101, workers=1, host=Admins-MacBook-Pro.local, username=tonyfast, pid=19479) running   MyTask(x=100, y=45)
]0;IPython: deathbeds.github.io/deathbeds145
INFO: [pid 19479] Worker Worker(salt=172344101, workers=1, host=Admins-MacBook-Pro.local, username=tonyfast, pid=19479) done      MyTask(x=100, y=45)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   MyTask_100_45_98a6d68a8b   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
INFO: Worker Worker(salt=172344101, workers=1, host=Admins-MacBook-Pro.local, username=tonyfast, pid=19479) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 ran successfully:
    - 1 MyTask(x=100, y=45)

This progress looks :) because there were no failed tasks or missing external dependencies

===== Luigi Execution Summary =====

The :) can be so satisfying on a bad day at the keyboard.

In [9]:

    if __name__ == '__main__': import disqus

Using importnb with Luigi¶

Using `importnb` with Luigi¶