Snakemake Overview¶

A Snakemake workflow is defined in terms of rules that are written in a file named Snakefile (similar to a Makefile with GNU Make). Rules consist of a name, input file(s), output file(s), and a shell command to generate the output from the input. Dependencies between rules are handled implicitly, by matching filenames of input files against output files.

A first workflow¶

To illustrate the use of Snakemake, we will test if a book follows the Zipf law: an empirical law which states that given a large sample of words, the frequency of any word is inversely proportional to its rank in the frequency table.

The first rule (named count_words) will take a book stored in text file as input and generate a list of words sorted by the number of occurrences in the book and a second rule (named fit_zipf) will try to fit the data from previous step to check if it follows the Zipf law. The final step (rule named plot_zipf) will generate a graph from the generated data.

These 3 rules are in the file named Snakefile that will be used to provide the workflow description to Snakemake.

We can check the validy of our workflow by performing a dry run. This can be done by passing the -n option when invoking Snakemake

We use the --forceall option to make sure that the whole workflow is executed regardless of the eventual presence of previously generated files.

Another thing we can do is visualize the directed acyclic graph (DAG) our workflow of our workflow with the --dag option and pipe the output to the Graphviz dot command to get a PNG from the DAG.

Now, let's run our workflow:

If the output of a rule is already present on the filesystem then Snakemake will not run the job that generates it. In general, a job (rule) is executed when

• output file is the target and does not exist
• output file needed by another executed job and does not exist
• input file newer than output file
• input file will be updated by other job
• execution is enforced (--forceall)

For example, if we remove the isles.png file that is generated by the plot_zipf rule but keep the output of the other rules, then only the plot_zipf rule will be executed.

We can run part of the workflow by passing the target (output) we are interested in as the last argument when invoking Snakemake. Snakemake will only run the jobs that produces the dependencies of the job that produce the output you pass as a command-line argument.

By using a wildcard, we only need to specify explicitly the name of the book in the plot_zipf rule and let Snakemake do the wildcards substitution to determine which rules should be run.

In our case, Snakemake will determine that in order to produce outputs/isles.png, outputs/isles.dat and outputs/isles.fit.dat are required and that these files may be generated by substituting the {book} wildcard with isles in the fit_zipf and count_words rules.

Extending the workflow¶

The advantage of using wildcards is that the workflow can quickly be extended to include more books. For that we will use the fact that Snakefiles are Python code: we can include any Python code in our workflow. For example, we can define a list with the book titles:

BOOKS = ['isles', 'abyss', 'last', 'sierra']

and then create a new list listing the plots we want to generate and use that list in a new rule that we will name all.

PLOTS = ['outputs/{book}.png'.format(book=book) for book in BOOKS]

rule all:     
    input: PLOTS

An alternative is to use the Snakemake expand function which produces the same result as Python brackets syntax:

PLOTS = expand('outputs/{book}.png', book=BOOKS)

Our workflow thus becomes

Running a Python script¶

Snakemake is not limited to running shell command. We can invoke a script using the script directive. For example, we can add script that will agreggate the data of all the books:

DATS = expand(f'{OUTPUT_DIR}/{{book}}.dat', book=BOOKS)

rule aggregate:
    input: DATS
    output: 'outputs/aggregated.dat'
    script: 'scripts/aggregate.py'

with the aggregate.py script

data = {}
for input in snakemake.input:
    read_input(input, data)

sorted_data = reversed(sorted(data.items(), key = itemgetter(1)))

with open(snakemake.output[0],'w') as outfile:
    for key, value in sorted_data:
        outfile.write('{:s} {:d}\n'.format(key, value))

note that inside the script we have access to a snakemake object and make use of it in order to iterate over all the inputs (snakemake.input) and determine where we need to write the output (snakemake.input).

Snakemake in not limited to Python script. It can also be used with R, Julia and Rust. For all these languages, Snakemake provides an object that can be manipulated to gather information about the workflow.

Using a configuration file¶

We can use a configuration file In order to set the parameters of our data analysis. For that we will create a YAML file named config.yml with the following content:

In the Snakefile, we can make use of this configuration file using the configfile directive:

configfile: 'config.yml'

Snakemake will read this configuration file and make the values available in a dictionary named config. We can then use this dictionary to extract the configuration for our workflow

OUTPUT_DIR = config['output_dir']
BOOKS_DIR = config['books_dir']

Then, we use this variable to define variable that will be used in the input and output directives of our workflow:

BOOK_TXT = f'{BOOKS_DIR}/{{book}}.txt'
BOOK_DAT = f'{OUTPUT_DIR}/{{book}}.dat'
BOOK_FIT = f'{OUTPUT_DIR}/{{book}}.fit.dat'

Note that we use double curly braces ({{book}}) so that we obtain the wildcard {book} in the output of the formatted string. If the value of the BOOKS_DIR is books, then the value of BOOK_TXT in example above will be 'books/{book}.txt'.

We can also ensure that it is no longer necessary to explicitly specify the title of books by extracting the name of the books stored as txt files in the books directory. To achieve that we can use the Snakemake glob_wildcards function.

BOOKS = glob_wildcards(f'{BOOKS_DIR}/{{book}}.txt').book

The glob_wildcards function matches the given pattern against the files present in the filesystem and thereby infers the values for all wildcards in the pattern. This function returns a named tuple. In our case it will list the name of all the files in the books directory.

With what as been discussed above, our workflow is now:

More information about Snakemake¶

• Snakemake website
• Snakemake documentation