#|hide
#|default_exp script
A fast way to turn your python function into a script.
Part of fast.ai's toolkit for delightful developer experiences.
Sometimes, you want to create a quick script, either for yourself, or for others. But in Python, that involves a whole lot of boilerplate and ceremony, especially if you want to support command line arguments, provide help, and other niceties. You can use argparse for this purpose, which comes with Python, but it's complex and verbose.
fastcore.script
makes life easier. There are much fancier modules to help you write scripts (we recommend Python Fire, and Click is also popular), but fastcore.script is very fast and very simple. In fact, it's <50 lines of code! Basically, it's just a little wrapper around argparse
that uses modern Python features and some thoughtful defaults to get rid of the boilerplate.
For full details, see the docs for core
.
Here's a complete example (available in examples/test_fastcore.py
):
from fastcore.script import *
@call_parse
def main(msg:str, # The message
upper:bool): # Convert to uppercase?
"Print `msg`, optionally converting to uppercase"
print(msg.upper() if upper else msg)
If you copy that info a file and run it, you'll see:
$ examples/test_fastcore.py --help
usage: test_fastcore.py [-h] [--upper] msg
Print `msg`, optionally converting to uppercase
positional arguments:
msg The message
optional arguments:
-h, --help show this help message and exit
--upper Convert to uppercase? (default: False)
As you see, we didn't need any if __name__ == "__main__"
, we didn't have to parse arguments, we just wrote a function, added a decorator to it, and added some annotations to our function's parameters. As a bonus, we can also use this function directly from a REPL such as Jupyter Notebook - it's not just for command line scripts!
You should provide a default (after the =
) for any optional parameters. If you don't provide a default for a parameter, then it will be a positional parameter.
If you want to use the full power of argparse
, you can do so by using Param
annotations instead of type annotations and docments, like so:
from fastcore.script import *
@call_parse
def main(msg:Param("The message", str),
upper:Param("Convert to uppercase?", store_true)):
"Print `msg`, optionally converting to uppercase"
print(msg.upper() if upper else msg)
If you use this approach, then each parameter in your function should have an annotation Param(...)
(as in the example above). You can pass the following when calling Param
: help
,type
,opt
,action
,nargs
,const
,choices
,required
. Except for opt
, all of these are just passed directly to argparse
, so you have all the power of that module at your disposal. Generally you'll want to pass at least help
(since this is provided as the help string for that parameter) and type
(to ensure that you get the type of data you expect). opt
is a bool that defines whether a param is optional or required (positional) - but you'll generally not need to set this manually, because fastcore.script will set it for you automatically based on default values.
There's a really nice feature of pip/setuptools that lets you create commandline scripts directly from functions, makes them available in the PATH
, and even makes your scripts cross-platform (e.g. in Windows it creates an exe). fastcore.script supports this feature too. The trick to making a function available as a script is to add a console_scripts
section to your setup file, of the form: script_name=module:function_name
. E.g. in this case we use: test_fastcore.script=fastcore.script.test_cli:main
. With this, you can then just type test_fastcore.script
at any time, from any directory, and your script will be called (once it's installed using one of the methods below).
You don't actually have to write a setup.py
yourself. Instead, just use nbdev. Then modify settings.ini
as appropriate for your module/script. To install your script directly, you can type pip install -e .
. Your script, when installed this way (it's called an editable install), will automatically be up to date even if you edit it - there's no need to reinstall it after editing. With nbdev you can even make your module and script available for installation directly from pip and conda by running make release
.
#|export
import inspect,argparse,shutil
from functools import wraps,partial
from fastcore.imports import *
from fastcore.utils import *
from fastcore.docments import docments
#|hide
from fastcore.test import *
#|export
def store_true():
"Placeholder to pass to `Param` for `store_true` action"
pass
#|export
def store_false():
"Placeholder to pass to `Param` for `store_false` action"
pass
#|export
def bool_arg(v):
"Use as `type` for `Param` to get `bool` behavior"
return str2bool(v)
#|export
def clean_type_str(x:str):
x = str(x)
x = re.sub("(enum |class|function|__main__\.|\ at.*)", '', x)
x = re.sub("(<|>|'|\ )", '', x) # spl characters
return x
class Test: pass
test_eq(clean_type_str(argparse.ArgumentParser), 'argparse.ArgumentParser')
test_eq(clean_type_str(Test), 'Test')
test_eq(clean_type_str(int), 'int')
test_eq(clean_type_str(float), 'float')
test_eq(clean_type_str(store_false), 'store_false')
#|export
class Param:
"A parameter in a function used in `anno_parser` or `call_parse`"
def __init__(self, help="", type=None, opt=True, action=None, nargs=None, const=None,
choices=None, required=None, default=None):
if type in (store_true,bool): type,action,default=None,'store_true' ,False
if type==store_false: type,action,default=None,'store_false',True
if type and isinstance(type,typing.Type) and issubclass(type,enum.Enum) and not choices: choices=list(type)
help = help or ""
store_attr()
def set_default(self, d):
if self.default is None:
if d==inspect.Parameter.empty: self.opt = False
else: self.default = d
if self.default is not None:
self.help += f" (default: {self.default})"
@property
def pre(self): return '--' if self.opt else ''
@property
def kwargs(self): return {k:v for k,v in self.__dict__.items()
if v is not None and k!='opt' and k[0]!='_'}
def __repr__(self):
if not self.help and self.type is None: return ""
if not self.help and self.type is not None: return f"{clean_type_str(self.type)}"
if self.help and self.type is None: return f"<{self.help}>"
if self.help and self.type is not None: return f"{clean_type_str(self.type)} <{self.help}>"
test_eq(repr(Param("Help goes here")), '<Help goes here>')
test_eq(repr(Param("Help", int)), 'int <Help>')
test_eq(repr(Param(help=None, type=int)), 'int')
test_eq(repr(Param(help=None, type=None)), '')
Each parameter in your function should have an annotation Param(...)
. You can pass the following when calling Param
: help
,type
,opt
,action
,nargs
,const
,choices
,required
(i.e. it takes the same parameters as argparse.ArgumentParser.add_argument
, plus opt
). Except for opt
, all of these are just passed directly to argparse
, so you have all the power of that module at your disposal. Generally you'll want to pass at least help
(since this is provided as the help string for that parameter) and type
(to ensure that you get the type of data you expect).
opt
is a bool that defines whether a param is optional or required (positional) - but you'll generally not need to set this manually, because fastcore.script will set it for you automatically based on default values. You should provide a default (after the =
) for any optional parameters. If you don't provide a default for a parameter, then it will be a positional parameter.
Param's __repr__
also allows for more informative function annotation when looking up the function's doc using shift+tab. You see the type annotation (if there is one) and the accompanying help documentation with it.
def f(required:Param("Required param", int),
a:Param("param 1", bool_arg),
b:Param("param 2", str)="test"):
"my docs"
...
help(f)
Help on function f in module __main__: f(required: int <Required param>, a: bool_arg <param 1>, b: str <param 2> = 'test') my docs
p = Param(help="help", type=int)
p.set_default(1)
test_eq(p.kwargs, {'help': 'help (default: 1)', 'type': int, 'default': 1})
#|export
class _HelpFormatter(argparse.HelpFormatter):
def __init__(self, prog, indent_increment=2):
cols = shutil.get_terminal_size((120,30))[0]
super().__init__(prog, max_help_position=cols//2, width=cols, indent_increment=indent_increment)
def _expand_help(self, action): return self._get_help_string(action)
#|export
def anno_parser(func, # Function to get arguments from
prog:str=None): # The name of the program
"Look at params (annotated with `Param`) in func and return an `ArgumentParser`"
p = argparse.ArgumentParser(description=func.__doc__, prog=prog, formatter_class=_HelpFormatter)
for k,v in docments(func, full=True, returns=False, eval_str=True).items():
param = v.anno
if not isinstance(param,Param): param = Param(v.docment, v.anno)
param.set_default(v.default)
p.add_argument(f"{param.pre}{k}", **param.kwargs)
p.add_argument(f"--pdb", help=argparse.SUPPRESS, action='store_true')
p.add_argument(f"--xtra", help=argparse.SUPPRESS, type=str)
return p
This converts a function with parameter annotations of type Param
into an argparse.ArgumentParser
object. Function arguments with a default provided are optional, and other arguments are positional.
_en = str_enum('_en', 'aa','bb','cc')
def f(required:Param("Required param", int),
a:Param("param 1", bool_arg),
b:Param("param 2", str)="test",
c:Param("param 3", _en)=_en.aa):
"my docs"
...
p = anno_parser(f, 'progname')
p.print_help()
usage: progname [-h] [--b B] [--c {aa,bb,cc}] required a my docs positional arguments: required Required param a param 1 optional arguments: -h, --help show this help message and exit --b B param 2 (default: test) --c {aa,bb,cc} param 3 (default: aa)
It also works with type annotations and docments:
def g(required:int, # Required param
a:bool_arg, # param 1
b="test", # param 2
c:_en=_en.aa): # param 3
"my docs"
...
p = anno_parser(g, 'progname')
p.print_help()
usage: progname [-h] [--b B] [--c {aa,bb,cc}] required a my docs positional arguments: required Required param a param 1 optional arguments: -h, --help show this help message and exit --b B param 2 (default: test) --c {aa,bb,cc} param 3 (default: aa)
#|export
def args_from_prog(func, prog):
"Extract args from `prog`"
if prog is None or '#' not in prog: return {}
if '##' in prog: _,prog = prog.split('##', 1)
progsp = prog.split("#")
args = {progsp[i]:progsp[i+1] for i in range(0, len(progsp), 2)}
annos = type_hints(func)
for k,v in args.items():
t = annos.get(k, Param()).type
if t: args[k] = t(v)
return args
Sometimes it's convenient to extract arguments from the actual name of the called program. args_from_prog
will do this, assuming that names and values of the params are separated by a #
. Optionally there can also be a prefix separated by ##
(double underscore).
exp = {'a': False, 'b': 'baa'}
test_eq(args_from_prog(f, 'foo##a#0#b#baa'), exp)
test_eq(args_from_prog(f, 'a#0#b#baa'), exp)
#|export
SCRIPT_INFO = SimpleNamespace(func=None)
#|export
def call_parse(func=None, nested=False):
"Decorator to create a simple CLI from `func` using `anno_parser`"
if func is None: return partial(call_parse, nested=nested)
@wraps(func)
def _f(*args, **kwargs):
mod = inspect.getmodule(inspect.currentframe().f_back)
if not mod: return func(*args, **kwargs)
if not SCRIPT_INFO.func and mod.__name__=="__main__": SCRIPT_INFO.func = func.__name__
if len(sys.argv)>1 and sys.argv[1]=='': sys.argv.pop(1)
p = anno_parser(func)
if nested: args, sys.argv[1:] = p.parse_known_args()
else: args = p.parse_args()
args = args.__dict__
xtra = otherwise(args.pop('xtra', ''), eq(1), p.prog)
tfunc = trace(func) if args.pop('pdb', False) else func
return tfunc(**merge(args, args_from_prog(func, xtra)))
mod = inspect.getmodule(inspect.currentframe().f_back)
if getattr(mod, '__name__', '') =="__main__":
setattr(mod, func.__name__, _f)
SCRIPT_INFO.func = func.__name__
return _f()
else: return _f
@call_parse
def test_add(
a:int=0, # param a
b:int=0 # param 1
):
"Add up `a` and `b`"
return a + b
call_parse
decorated functions work as regular functions and also as command-line interface functions.
test_eq(test_add(1,2), 3)
This is the main way to use fastcore.script
; decorate your function with call_parse
, add Param
annotations (as shown above) or type annotations and docments, and it can then be used as a script.
Use the nested
keyword argument to create nested parsers, where earlier parsers consume only their known args from sys.argv
before later parsers are used. This is useful to create one command line application that executes another. For example:
myrunner --keyword 1 script.py -- <script.py args>
A separating --
after the first application's args is recommended though not always required, otherwise args may be parsed in unexpected ways. For example:
myrunner script.py -h
would display myrunner
's help and not script.py
's.
#|hide
import nbdev; nbdev.nbdev_export()