Episode

Running on HTCondor

After a workflow runs locally, the next step is often to submit it to an HTCondor cluster. The main idea is simple: the workflow logic should stay the same. In most cases, you should not rewrite rules for HTCondor. Instead, you change how Snakemake is executed.

This is a short reference episode rather than a full live exercise. The concrete example used here comes from CERN, and lives in the snakemake-lxplus-example repository (and will probably soon be moved under the hep-workflows GitHub organization).

Same Workflow, Different Execution

The same rule can often run:

• locally on your machine
• on a login node
• on an HTCondor cluster such as LXBATCH

That is one of the strengths of Snakemake. The rule still declares its input, output, resources, and software environment. What changes is the executor and the profile that Snakemake uses at run time.

Install the Executor Plugin

In addition to snakemake itself, HTCondor execution needs the corresponding executor plugin and the HTCondor Python bindings. If you are working in the same Pixi environment that you created during setup, add them with:

pixi add python-htcondor snakemake-executor-plugin-htcondor

After that, pixi run snakemake can use the HTCondor executor from the same environment.

Use a Workflow Profile

For HTCondor execution, the cleanest approach is to keep cluster-specific settings in a workflow profile and then run Snakemake with:

pixi run snakemake --workflow-profile lxbatch

For Snakemake 9 and later, a minimal profile can be saved as workflow/profiles/lxbatch/profile.v9+.yaml:

executor: htcondor
jobs: 5000
local-cores: 10
htcondor-jobdir: .condor_jobs
default-resources:
  - getenv=True
  - htcondor_request_mem_mb=1024
  - htcondor_request_disk_mb=1024
  - classad_JobFlavour=espresso

This keeps the workflow itself readable. The Snakefile describes the analysis, while the profile describes how jobs should be submitted on the cluster.

A Minimal HTCondor Example

With that profile in place, a minimal Snakefile could look like:

rule all:
    input:
        "local_hello.txt"


rule hello_htcondor:
    output:
        "local_hello.txt"
    shell:
        "echo Hello from $(hostname) > {output}"

You can then run:

pixi run snakemake --workflow-profile lxbatch --cores 1

This uses the native snakemake-executor-plugin-htcondor and submits a simple rule through HTCondor.

The important point is not the particular example rule. The important point is that it is still an ordinary Snakemake rule. Batch execution is selected at run time by the workflow profile.

[tasks]
snakemake-htcondor = "snakemake --workflow-profile lxbatch --cores 1"

pixi run snakemake-htcondor

Resources

When jobs run on LXBATCH, resource requests become much more important than they are in a small local example. At minimum, think about:

• memory
• disk
• runtime

It is often sensible to set conservative defaults in the profile and then override them for unusually heavy rules in the workflow itself.

Why Batching Matters

On batch systems, the right unit of work is not always “one input file per job”. If the per-file tasks are very small, submitting one HTCondor job for each file can create unnecessary overhead.

The example repository therefore, includes two batching patterns:

• fixed-size batching
• cost-aware batching

Fixed-size batching is simpler, but cost-aware batching is often better when some samples are known to run much longer than others.