Set-up for situation presented.
(Used session launched from here; go there and click on 'launch binder' to get started. That was because when I tried with an older Python version from sessions launched from binder's requirements.txt example repo, pip reported for Snakemake 8.18.1, I needed 3.11(?). [Although it seemed to work to install with pip on Anaconda cloud with Python 3.10.])
%pip install snakemake==8.18.1
Collecting snakemake==8.18.1
Downloading snakemake-8.18.1-py3-none-any.whl.metadata (2.5 kB)
Collecting appdirs (from snakemake==8.18.1)
Downloading appdirs-1.4.4-py2.py3-none-any.whl.metadata (9.0 kB)
Collecting immutables (from snakemake==8.18.1)
Downloading immutables-0.20-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.6 kB)
Collecting configargparse (from snakemake==8.18.1)
Downloading ConfigArgParse-1.7-py3-none-any.whl.metadata (23 kB)
Collecting connection-pool>=0.0.3 (from snakemake==8.18.1)
Downloading connection_pool-0.0.3.tar.gz (3.8 kB)
Preparing metadata (setup.py) ... done
Collecting datrie (from snakemake==8.18.1)
Downloading datrie-0.8.2.tar.gz (63 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.3/63.3 kB 4.2 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting docutils (from snakemake==8.18.1)
Downloading docutils-0.21.2-py3-none-any.whl.metadata (2.8 kB)
Collecting gitpython (from snakemake==8.18.1)
Downloading GitPython-3.1.43-py3-none-any.whl.metadata (13 kB)
Collecting humanfriendly (from snakemake==8.18.1)
Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
Requirement already satisfied: jinja2<4.0,>=3.0 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (3.1.4)
Requirement already satisfied: jsonschema in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (4.22.0)
Requirement already satisfied: nbformat in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (5.10.4)
Requirement already satisfied: packaging in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (24.1)
Requirement already satisfied: psutil in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (5.9.8)
Collecting pulp<2.9,>=2.3.1 (from snakemake==8.18.1)
Downloading PuLP-2.8.0-py3-none-any.whl.metadata (5.4 kB)
Requirement already satisfied: pyyaml in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (6.0.1)
Requirement already satisfied: requests<3.0,>=2.8.1 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from snakemake==8.18.1) (2.32.3)
Collecting reretry (from snakemake==8.18.1)
Downloading reretry-0.11.8-py2.py3-none-any.whl.metadata (5.5 kB)
Collecting smart-open<8.0,>=4.0 (from snakemake==8.18.1)
Downloading smart_open-7.0.4-py3-none-any.whl.metadata (23 kB)
Collecting snakemake-interface-executor-plugins<10.0,>=9.2.0 (from snakemake==8.18.1)
Downloading snakemake_interface_executor_plugins-9.2.0-py3-none-any.whl.metadata (7.7 kB)
Collecting snakemake-interface-common<2.0,>=1.17.0 (from snakemake==8.18.1)
Downloading snakemake_interface_common-1.17.3-py3-none-any.whl.metadata (760 bytes)
Collecting snakemake-interface-storage-plugins<4.0,>=3.2.3 (from snakemake==8.18.1)
Downloading snakemake_interface_storage_plugins-3.3.0-py3-none-any.whl.metadata (9.8 kB)
Collecting snakemake-interface-report-plugins<2.0.0,>=1.0.0 (from snakemake==8.18.1)
Downloading snakemake_interface_report_plugins-1.0.0-py3-none-any.whl.metadata (3.4 kB)
Collecting stopit (from snakemake==8.18.1)
Downloading stopit-1.1.2.tar.gz (18 kB)
Preparing metadata (setup.py) ... done
Collecting tabulate (from snakemake==8.18.1)
Downloading tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Collecting throttler (from snakemake==8.18.1)
Downloading throttler-1.2.2-py3-none-any.whl.metadata (7.4 kB)
Collecting toposort<2.0,>=1.10 (from snakemake==8.18.1)
Downloading toposort-1.10-py3-none-any.whl.metadata (4.1 kB)
Collecting wrapt (from snakemake==8.18.1)
Downloading wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting yte<2.0,>=1.5.1 (from snakemake==8.18.1)
Downloading yte-1.5.4-py3-none-any.whl.metadata (3.4 kB)
Collecting dpath<3.0.0,>=2.1.6 (from snakemake==8.18.1)
Downloading dpath-2.2.0-py3-none-any.whl.metadata (15 kB)
Collecting conda-inject<2.0,>=1.3.1 (from snakemake==8.18.1)
Downloading conda_inject-1.3.2-py3-none-any.whl.metadata (855 bytes)
Requirement already satisfied: MarkupSafe>=2.0 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from jinja2<4.0,>=3.0->snakemake==8.18.1) (2.1.5)
Requirement already satisfied: charset-normalizer<4,>=2 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from requests<3.0,>=2.8.1->snakemake==8.18.1) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from requests<3.0,>=2.8.1->snakemake==8.18.1) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from requests<3.0,>=2.8.1->snakemake==8.18.1) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from requests<3.0,>=2.8.1->snakemake==8.18.1) (2024.6.2)
Collecting argparse-dataclass<3.0.0,>=2.0.0 (from snakemake-interface-common<2.0,>=1.17.0->snakemake==8.18.1)
Downloading argparse_dataclass-2.0.0-py3-none-any.whl.metadata (7.2 kB)
Collecting plac<2.0.0,>=1.3.4 (from yte<2.0,>=1.5.1->snakemake==8.18.1)
Downloading plac-1.4.3-py2.py3-none-any.whl.metadata (5.9 kB)
Collecting gitdb<5,>=4.0.1 (from gitpython->snakemake==8.18.1)
Downloading gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB)
Requirement already satisfied: attrs>=22.2.0 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from jsonschema->snakemake==8.18.1) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from jsonschema->snakemake==8.18.1) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from jsonschema->snakemake==8.18.1) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from jsonschema->snakemake==8.18.1) (0.18.1)
Requirement already satisfied: fastjsonschema>=2.15 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from nbformat->snakemake==8.18.1) (2.20.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from nbformat->snakemake==8.18.1) (5.7.2)
Requirement already satisfied: traitlets>=5.1 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from nbformat->snakemake==8.18.1) (5.14.3)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython->snakemake==8.18.1)
Downloading smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB)
Requirement already satisfied: platformdirs>=2.5 in /srv/conda/envs/notebook/lib/python3.11/site-packages (from jupyter-core!=5.0.*,>=4.12->nbformat->snakemake==8.18.1) (4.2.2)
Downloading snakemake-8.18.1-py3-none-any.whl (308 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 308.4/308.4 kB 14.3 MB/s eta 0:00:00
Downloading conda_inject-1.3.2-py3-none-any.whl (4.1 kB)
Downloading dpath-2.2.0-py3-none-any.whl (17 kB)
Downloading PuLP-2.8.0-py3-none-any.whl (17.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.7/17.7 MB 37.1 MB/s eta 0:00:0000:0100:01
Downloading smart_open-7.0.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 9.6 MB/s eta 0:00:00
Downloading snakemake_interface_common-1.17.3-py3-none-any.whl (14 kB)
Downloading ConfigArgParse-1.7-py3-none-any.whl (25 kB)
Downloading snakemake_interface_executor_plugins-9.2.0-py3-none-any.whl (22 kB)
Downloading snakemake_interface_report_plugins-1.0.0-py3-none-any.whl (6.9 kB)
Downloading snakemake_interface_storage_plugins-3.3.0-py3-none-any.whl (15 kB)
Downloading reretry-0.11.8-py2.py3-none-any.whl (5.6 kB)
Downloading throttler-1.2.2-py3-none-any.whl (7.6 kB)
Downloading toposort-1.10-py3-none-any.whl (8.5 kB)
Downloading wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.7/80.7 kB 12.8 MB/s eta 0:00:00
Downloading yte-1.5.4-py3-none-any.whl (7.7 kB)
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Downloading docutils-0.21.2-py3-none-any.whl (587 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 587.4/587.4 kB 66.2 MB/s eta 0:00:00
Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 207.3/207.3 kB 26.4 MB/s eta 0:00:00
Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 6.5 MB/s eta 0:00:00
Downloading immutables-0.20-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (99 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.7/99.7 kB 13.5 MB/s eta 0:00:00
Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
Downloading argparse_dataclass-2.0.0-py3-none-any.whl (8.8 kB)
Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 12.0 MB/s eta 0:00:00
Downloading plac-1.4.3-py2.py3-none-any.whl (22 kB)
Downloading smmap-5.0.1-py3-none-any.whl (24 kB)
Building wheels for collected packages: connection-pool, datrie, stopit
Building wheel for connection-pool (setup.py) ... done
Created wheel for connection-pool: filename=connection_pool-0.0.3-py3-none-any.whl size=4061 sha256=4918f8d8a58b46512f41fbc00b04d40c5818a205139a504b11480239f6b82b15
Stored in directory: /home/jovyan/.cache/pip/wheels/2b/73/ac/bd9807cbc47e95c436b5d5afe6cca8299fdc69bf7bd9930618
Building wheel for datrie (pyproject.toml) ... done
Created wheel for datrie: filename=datrie-0.8.2-cp311-cp311-linux_x86_64.whl size=179203 sha256=38434f3f5d7b73ff12a19217ce91932b9b1e2506f4dc8ac5bdb3b4ad51c5016a
Stored in directory: /home/jovyan/.cache/pip/wheels/8b/20/f4/aeacf0184f20e22473a01257c56c74cc976e1cd838a01de6d5
Building wheel for stopit (setup.py) ... done
Created wheel for stopit: filename=stopit-1.1.2-py3-none-any.whl size=11939 sha256=63c60fd9b22d00767c5f853db696f22d50b900bd84771db6ec8a2028e2279b42
Stored in directory: /home/jovyan/.cache/pip/wheels/da/77/2d/adbc56bc4db95ad80c6d4e71cd69e2d9d122174904342e3f7f
Successfully built connection-pool datrie stopit
Installing collected packages: toposort, throttler, stopit, plac, connection-pool, appdirs, wrapt, tabulate, smmap, reretry, pulp, immutables, humanfriendly, dpath, docutils, datrie, configargparse, conda-inject, argparse-dataclass, yte, snakemake-interface-common, smart-open, gitdb, snakemake-interface-storage-plugins, snakemake-interface-report-plugins, snakemake-interface-executor-plugins, gitpython, snakemake
Successfully installed appdirs-1.4.4 argparse-dataclass-2.0.0 conda-inject-1.3.2 configargparse-1.7 connection-pool-0.0.3 datrie-0.8.2 docutils-0.21.2 dpath-2.2.0 gitdb-4.0.11 gitpython-3.1.43 humanfriendly-10.0 immutables-0.20 plac-1.4.3 pulp-2.8.0 reretry-0.11.8 smart-open-7.0.4 smmap-5.0.1 snakemake-8.18.1 snakemake-interface-common-1.17.3 snakemake-interface-executor-plugins-9.2.0 snakemake-interface-report-plugins-1.0.0 snakemake-interface-storage-plugins-3.3.0 stopit-1.1.2 tabulate-0.9.0 throttler-1.2.2 toposort-1.10 wrapt-1.16.0 yte-1.5.4
Note: you may need to restart the kernel to use updated packages.
!snakemake --version
8.18.1
Worked! Installed specified version.
Now to prepare for the Snakemake snakefile put forth...
Code put forth as example is old and was from bottom here at 'EdwardsLab's post: 'Snakemake - How to use snakemake checkpoints'.
Issue is that the code there has flag 'directory' used under input directive for the rule make_third_files and so you get The flag 'directory' used in rule make_third_files is only valid for outputs, not inputs..
The changes needed are not using directory() flag with input directives as covered here or else you get the error The flag 'directory' used in rule make_third_files is only valid for outputs, not inputs.. However, directories themseleves are allowed as input for Snakemake so just use the path to them without the flag.
Plus to get this to work, you need the two output directories as input for the default rule. (I'm not sure how it ever worked before because I don't see how rules make_some_files & make_more_files would get triggered otherwise.)
To work this out, I originally put the original provided code between the ''' below and then edit it and re-ran the next two cells, iterating on that until it worked. (I then later started a fresh session with what I worked out and re-ran to get the final-run 'clean' version seen here.)
ns='''OUTDIR = "first_directory"
SNDDIR = "second_directory"
THRDIR = "third_directory"
def combine(wildcards):
# read the first set of outputs
ck_output = checkpoints.make_some_files.get(**wildcards).output[0]
FIRSTS, = glob_wildcards(os.path.join(ck_output, "{sample}.txt"))
# read the second set of outputs
sn_output = checkpoints.make_more_files.get(**wildcards).output[0]
SECONDS, = glob_wildcards(os.path.join(sn_output, "{smpl}.txt"))
return expand(os.path.join(THRDIR, "{first}.{second}.tsv"), first=FIRSTS, second=SECONDS)
rule all:
input:
OUTDIR,
SNDDIR,
combine
checkpoint make_some_files:
output:
directory(OUTDIR)
shell:
"""
mkdir {output};
N=$(((RANDOM%5)+1));
for D in $(seq $N); do
touch {output}/$RANDOM.txt
done
"""
checkpoint make_more_files:
output:
directory(SNDDIR)
shell:
"""
mkdir {output};
N=$(((RANDOM%5)+1));
for D in $(seq $N); do
touch {output}/$RANDOM.txt
done
"""
rule make_third_files:
input:
OUTDIR,
SNDDIR,
output:
os.path.join(THRDIR, "{first}.{second}.tsv")
shell:
"""
touch {output}
"""
'''
%store ns >Snakefile
Writing 'ns' (str) to file 'Snakefile'.
!snakemake -c 1
Assuming unrestricted shared filesystem usage. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count --------------- ------- all 1 make_more_files 1 make_some_files 1 total 3 Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localcheckpoint make_more_files: output: second_directory jobid: 2 reason: Missing output files: second_directory resources: tmpdir=/tmp DAG of jobs will be updated after completion. [Tue Aug 20 17:30:46 2024] Finished job 2. 1 of 3 steps (33%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localcheckpoint make_some_files: output: first_directory jobid: 1 reason: Missing output files: first_directory resources: tmpdir=/tmp DAG of jobs will be updated after completion. [Tue Aug 20 17:30:46 2024] Finished job 1. 2 of 3 steps (67%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/32727.17967.tsv jobid: 15 reason: Missing output files: third_directory/32727.17967.tsv wildcards: first=32727, second=17967 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 15. 3 of 15 steps (20%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/20288.17078.tsv jobid: 8 reason: Missing output files: third_directory/20288.17078.tsv wildcards: first=20288, second=17078 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 8. 4 of 15 steps (27%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/24182.17967.tsv jobid: 12 reason: Missing output files: third_directory/24182.17967.tsv wildcards: first=24182, second=17967 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 12. 5 of 15 steps (33%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/9030.8514.tsv jobid: 16 reason: Missing output files: third_directory/9030.8514.tsv wildcards: first=9030, second=8514 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 16. 6 of 15 steps (40%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/24182.17078.tsv jobid: 11 reason: Missing output files: third_directory/24182.17078.tsv wildcards: first=24182, second=17078 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 11. 7 of 15 steps (47%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/32727.8514.tsv jobid: 13 reason: Missing output files: third_directory/32727.8514.tsv wildcards: first=32727, second=8514 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 13. 8 of 15 steps (53%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/20288.17967.tsv jobid: 9 reason: Missing output files: third_directory/20288.17967.tsv wildcards: first=20288, second=17967 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 9. 9 of 15 steps (60%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/9030.17078.tsv jobid: 17 reason: Missing output files: third_directory/9030.17078.tsv wildcards: first=9030, second=17078 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 17. 10 of 15 steps (67%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/9030.17967.tsv jobid: 18 reason: Missing output files: third_directory/9030.17967.tsv wildcards: first=9030, second=17967 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 18. 11 of 15 steps (73%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/32727.17078.tsv jobid: 14 reason: Missing output files: third_directory/32727.17078.tsv wildcards: first=32727, second=17078 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 14. 12 of 15 steps (80%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/20288.8514.tsv jobid: 7 reason: Missing output files: third_directory/20288.8514.tsv wildcards: first=20288, second=8514 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 7. 13 of 15 steps (87%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule make_third_files: input: first_directory, second_directory output: third_directory/24182.8514.tsv jobid: 10 reason: Missing output files: third_directory/24182.8514.tsv wildcards: first=24182, second=8514 resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 10. 14 of 15 steps (93%) done Select jobs to execute... Execute 1 jobs... [Tue Aug 20 17:30:46 2024] localrule all: input: first_directory, second_directory, third_directory/20288.8514.tsv, third_directory/20288.17078.tsv, third_directory/20288.17967.tsv, third_directory/24182.8514.tsv, third_directory/24182.17078.tsv, third_directory/24182.17967.tsv, third_directory/32727.8514.tsv, third_directory/32727.17078.tsv, third_directory/32727.17967.tsv, third_directory/9030.8514.tsv, third_directory/9030.17078.tsv, third_directory/9030.17967.tsv jobid: 0 reason: Input files updated by another job: third_directory/20288.8514.tsv, third_directory/9030.17967.tsv, third_directory/9030.8514.tsv, third_directory/9030.17078.tsv, third_directory/32727.8514.tsv, third_directory/32727.17967.tsv, third_directory/20288.17967.tsv, third_directory/32727.17078.tsv, third_directory/20288.17078.tsv, third_directory/24182.17078.tsv, third_directory/24182.17967.tsv, third_directory/24182.8514.tsv resources: tmpdir=/tmp [Tue Aug 20 17:30:46 2024] Finished job 0. 15 of 15 steps (100%) done Complete log: .snakemake/log/2024-08-20T173046.786457.snakemake.log
Works.
Let's show the contents of the directories as verificaiton.
ls first_directory/
20288.txt 24182.txt 32727.txt 9030.txt
ls second_directory/
17078.txt 17967.txt 8514.txt
ls third_directory/
20288.17078.tsv 24182.17078.tsv 32727.17078.tsv 9030.17078.tsv 20288.17967.tsv 24182.17967.tsv 32727.17967.tsv 9030.17967.tsv 20288.8514.tsv 24182.8514.tsv 32727.8514.tsv 9030.8514.tsv
The content of the working Snakefile is below for easier copying and pasting:
cat Snakefile
OUTDIR = "first_directory"
SNDDIR = "second_directory"
THRDIR = "third_directory"
def combine(wildcards):
# read the first set of outputs
ck_output = checkpoints.make_some_files.get(**wildcards).output[0]
FIRSTS, = glob_wildcards(os.path.join(ck_output, "{sample}.txt"))
# read the second set of outputs
sn_output = checkpoints.make_more_files.get(**wildcards).output[0]
SECONDS, = glob_wildcards(os.path.join(sn_output, "{smpl}.txt"))
return expand(os.path.join(THRDIR, "{first}.{second}.tsv"), first=FIRSTS, second=SECONDS)
rule all:
input:
OUTDIR,
SNDDIR,
combine
checkpoint make_some_files:
output:
directory(OUTDIR)
shell:
"""
mkdir {output};
N=$(((RANDOM%5)+1));
for D in $(seq $N); do
touch {output}/$RANDOM.txt
done
"""
checkpoint make_more_files:
output:
directory(SNDDIR)
shell:
"""
mkdir {output};
N=$(((RANDOM%5)+1));
for D in $(seq $N); do
touch {output}/$RANDOM.txt
done
"""
rule make_third_files:
input:
OUTDIR,
SNDDIR,
output:
os.path.join(THRDIR, "{first}.{second}.tsv")
shell:
"""
touch {output}
"""
Change type of the next cell to 'Code' and run cell below to clean up if want to test with running cell containing !snakemake -c 1 above again: