Episode

Getting Started: Rules, Targets, and the DAG

Snakemake is a Python-based workflow management system for building reproducible and scalable data analysis pipelines Mölder et al., 2021.

In this episode, we build a small event-selection workflow and use it to introduce the main Snakemake idea: you ask for the output you want, and Snakemake works out the steps needed to create it.

A First Rule

Create a small input file:

cat <<'EOF' > events.txt
Background
Signal
Background
Signal
EOF

Now create a Snakefile with one rule:

rule select_events:
    input:
        "events.txt"
    output:
        "selected_events.txt"
    shell:
        "grep 'Signal' {input} > {output}"

This rule says:

• the input is events.txt
• the output is selected_events.txt
• the command that transforms one into the other is a shell command

Running by Target

Run Snakemake by asking for the output file you want:

pixi run snakemake --cores 1 selected_events.txt

Snakemake looks at the requested target, finds a rule that can create it, and then runs that rule because selected_events.txt does not exist yet.

Thinking Backwards

Now extend the workflow with a second rule:

rule select_events:
    input:
        "events.txt"
    output:
        "selected_events.txt"
    shell:
        "grep 'Signal' {input} > {output}"


rule count_events:
    input:
        "selected_events.txt"
    output:
        "event_counts.txt"
    shell:
        "wc -l {input} > {output}"

If you now run

pixi run snakemake --cores 1 event_counts.txt

Snakemake reasons backwards:

1. You want event_counts.txt.
2. count_events can create it, but it needs selected_events.txt.
3. select_events can create selected_events.txt from events.txt.

This chain of dependencies is the Directed Acyclic Graph, or DAG. In a larger analysis, the same idea scales from two short rules to hundreds of jobs.

Defining a Default Target

If you do not specify a target on the command line, Snakemake uses the first rule in the Snakefile. By convention, we make that first rule a rule called all, which collects the final outputs we care about.

Update your Snakefile to:

rule all:
    input:
        "event_counts.txt"


rule select_events:
    input:
        "events.txt"
    output:
        "selected_events.txt"
    shell:
        "grep 'Signal' {input} > {output}"


rule count_events:
    input:
        "selected_events.txt"
    output:
        "event_counts.txt"
    shell:
        "wc -l {input} > {output}"

Now you can simply run:

pixi run snakemake --cores 1

Dry-Runs and Lazy Re-Execution

Before running a workflow, it is often useful to ask Snakemake what it would do without actually executing anything:

pixi run snakemake -n -p

The -n flag performs a dry-run, and -p prints the shell commands.

If you run the workflow again immediately afterwards, Snakemake should report that nothing needs to be done, because all requested outputs are present and up to date.

This is one of the most useful features of Snakemake: it does not rerun everything blindly. It only reruns work when an output is missing or when one of its inputs has changed.