Snakemake for Particle Physics

An introduction to Snakemake for particle physics data analysis.

Setup Get Started

Parallel Processing Across Files

Use wildcards and expand() to run the same selection across many event files in parallel and then gather the results.

Dynamic File Discovery

Learn when ordinary wildcards are not enough and how checkpoint lets Snakemake expand the workflow at run time.

Containers and Environments

Use per-rule containers to keep software requirements explicit and reproducible.

Learning Objectives

Episode	Learning Objectives
Getting Started: Rules, Targets, and the DAG	Create a minimal `Snakefile` with `input`, `output`, and `shell`. Run Snakemake by asking for an output file. Use `rule all` to define the default workflow target. Understand dry-runs and lazy re-execution.
Scaling with Wildcards for Parallel Processing	Use wildcards to run the same rule over many files. Use `expand()` to define a whole collection of expected outputs. Build a simple workflow that runs over many files in parallel and then gathers the results. Use `script:` for a small gather step.
Dynamic File Discovery with Checkpoints	Recognise when a workflow needs a `checkpoint`. Write a checkpoint that discovers input files and records them in simple file lists. Use `checkpoints.<name>.get()` inside an input function. Interpret the DAG before and after checkpoint expansion.
Containers	Use the `container:` directive in a rule. Run Snakemake with Apptainer using the current command-line syntax. Understand why different rules can use different software environments. Recognise which container details depend on the local site configuration.
Running on HTCondor	Understand how local and batch execution can use the same `Snakefile`. Use a workflow profile to store HTCondor-specific execution settings. Recognise which resource settings matter for HTCondor jobs. Know where to find a concrete HTCondor example for further study.
Bonus: Visualising the Workflow	Use the `--dag` flag to generate a visualization of the analysis. Understand the difference between the Rule Graph and the File Graph. Use dry-runs (`-n`) to verify the execution plan.

Schedule

Time	Episode	Questions
	Setup	Before you start
00:00	Getting Started: Rules, Targets, and the DAG 25 min	How do I run my first Snakemake workflow? How does Snakemake connect rules together? How does Snakemake decide what needs to run again?
00:25	Scaling with Wildcards for Parallel Processing 25 min	How can one rule process many event files? How can Snakemake run many file-based jobs in parallel and then combine their results? How do wildcards and `expand()` help a workflow scale?
00:50	Dynamic File Discovery with Checkpoints 25 min	When are ordinary wildcards no longer enough? What does a `checkpoint` do in Snakemake? How does Snakemake expand the DAG after files are discovered at run time?
01:15	Containers 25 min	How can I run a workflow step in a controlled software environment? What changes when I add a `container:` directive to a rule? What do I need in order to run containerised rules?
01:40	Running on HTCondor 10 min	How do I run the same Snakemake workflow on HTCondor? What should go into a workflow profile? Why do resources and batching matter on HTCondor?
01:50	Bonus: Visualising the Workflow 20 min	How can I see the dependencies between my rules? What is a Directed Acyclic Graph (DAG)? How do I preview what Snakemake intends to do?

Authors and Contributors

Contributor	GitHub
Clemens Lange	@clelange