Discovering the SUPER in computing - dagster-slurm for reproducible research on HPC

Dagster is a modern data orchestrator that emphasises reproducibility, observability, and a strong developer experience. In parallel, most high-performance computing (HPC) centres continue to rely on Slurm for batch scheduling and resource governance. The two ecosystems rarely meet in practice: Dagster projects often target cloud or single-node deployments, while Slurm users maintain bespoke submission scripts with limited reuse or visibility. This paper introduces dagster-slurm, an open-source integration that allows the same Dagster assets to run unchanged across laptops, CI pipelines, containerised Slurm clusters, and Tier-0 supercomputers. The project packages dependencies with Pixi, submits workloads through Slurm using Dagster Pipes, and streams logs plus scheduler metrics back to the Dagster UI. The key contribution is a unified compute resource (ComputeResource) that hides SSH transport (including password-only jump hosts and OTP prompts), dependency packaging, and queue configuration while still respecting Slurm’s scheduling semantics. The project ships two production-ready execution modes - local for laptop/CI development and slurm for one-job-per-asset submissions - and two stable launchers: Bash for script-based workloads and Ray for multi-node distributed computing.

Journal of Open Source Software, 11(119), 9795