Clustrix Documentation

Run an ordinary Python function somewhere else.

Add @cluster to a function, call it normally, and Clustrix serializes it with its arguments, ships it to the compute resource you configured, runs it there, and hands you back the return value. No job script, no scp, no polling loop, no result-unpickling glue.

PyPI version Python versions License 
from clustrix import cluster, configure

configure(cluster_type="local")  # no cluster needed to try this

@cluster(cores=8, memory="16GB", time="02:00:00")
def expensive_computation(iterations=1000):
    import math

    return sum(math.sqrt(i) for i in range(iterations))

print(expensive_computation(iterations=10_000))

Change cluster_type="local" to a SLURM login node and those same lines submit a batch job. That substitutability is the point of the library.

Start here

  • Introduction – what Clustrix is, what it is not, and how it compares to hand-written sbatch scripts, Dask, Ray, joblib and plain SSH.

  • Installation – install it, with the optional extras.

  • Quickstart – a real result in five minutes, beginning with a backend that needs no cluster at all.

  • Supported Cluster Types – the four backends Clustrix supports, and the evidence that each one runs a real job.

  • Backends Clustrix does not support – if you are looking for PBS, SGE, Kubernetes or a cloud VM provider, start here.

What it does

  • One decorator. @cluster on a function is the whole interface.

  • Function packaging. Your function is serialized by value with dill and cloudpickle, so closures, nested functions and project-local modules travel with it. Source code is not required.

  • Four backends: SLURM, SSH, HuggingFace Jobs and local execution. Each has been run end to end – see Supported Cluster Types.

  • Environment replication. The remote environment is rebuilt from your local pip freeze.

  • Read-only filesystem utilities. cluster_ls, cluster_glob, cluster_stat and their siblings work against a local path or a remote one through the same call. They inspect; they do not transfer.

  • Shared-storage detection. A worker that already shares your filesystem is detected, so the payload is not copied across a network that does not need it.

  • Loop parallelization. A loop whose body carries no dependency between iterations can be distributed across nodes. The analysis is deliberately conservative and declines most real loops – see Limitations and Edge Cases.

  • A Jupyter widget. %%remote opens a configuration panel in the notebook.

  • Errors that reach you. A remote traceback is re-raised in your own process rather than left in a log file on the cluster.

Two things that sound like features and are not. @cluster(cores=N) does not split an ordinary local function across N workers: it runs once in your own process, and Clustrix warns that the number was discarded whenever you asked for more than one core. Loop parallelization is on by default, so nothing has to be switched on: cores sizes a local pool only when the loop analysis finds a supported loop and the function accepts the matching chunk argument. Clustrix also does not move your data – see Introduction.

Jupyter Notebook Integration

Clustrix registers an IPython magic that opens a configuration widget:

%%remote

Importing clustrix registers the magic but does not display the widget. A library should not inject UI as a side effect of being imported, so the widget appears on demand: run %%remote in a cell, or call clustrix.notebook_magic.display_config_widget(). Setting CLUSTRIX_AUTO_WIDGET=1 makes it display on import instead.

%%clusterfy is an alias for %%remote. It works, and it emits a DeprecationWarning.

Interactive Configuration Widget

The widget edits the same settings as clustrix.configure() and applies them to the current session.

Clustrix widget in JupyterLab, light theme

Its colours resolve through JupyterLab’s own theme variables, so it follows the notebook theme rather than carrying a second hand-maintained dark stylesheet:

The same widget in JupyterLab's dark theme

“Show advanced” reveals the package manager, Python executable, environment variables, module loads and pre-execution commands:

Widget with the Advanced panel expanded

What the widget covers

The cluster type dropdown offers local, ssh, slurm and huggingface – the same four values as clustrix.config.SUPPORTED_CLUSTER_TYPES.

  • ssh and slurm show the connection section: host, port, username, SSH key file, password, remote work directory, an environment variable to read the password from, and an “Auto setup SSH keys” button.

  • huggingface shows namespace, flavor, token and an “Allow paid GPU flavors” checkbox. GPU flavors bill by the second, so that box has to be ticked before one is accepted.

  • local needs no connection settings at all.

There are no PBS, SGE, Kubernetes, AWS, GCP, Azure or Lambda Cloud entries, because Clustrix does not support those backends; see Backends Clustrix does not support.

Table of Contents

Getting Started

User Guide

Tutorials

Interactive Notebooks

API Reference

Supported Cluster Types

Execution backends

Clustrix supports exactly four cluster_type values – the contents of clustrix.config.SUPPORTED_CLUSTER_TYPES, which is also what the CLI and the notebook widget offer. There are no others.

cluster_type

Status

Notes

slurm

Verified

A real job ran on hpc.example.edu and returned its result.

ssh

Verified

Direct execution, no scheduler. A real job ran on an 8-GPU host.

huggingface

Verified

HuggingFace Jobs. A real job ran in a container.

local

Works

Runs in the calling process. Used for development and the fast tests.

cluster_type='huggingface' means HuggingFace Jobs. There is no HuggingFace Spaces provider; see Backends Clustrix does not support.

Backends Clustrix does not support

PBS, SGE, Kubernetes, AWS, GCP, Azure and Lambda Cloud are absent, and so are the cost-monitoring and cloud pricing APIs that served them. Clustrix does not claim a backend it has not run a real job on, and none of these has one. Setting cluster_type to any of those names raises a ValueError naming the backend and its tracking issue. Each is planned for a future release; Backends Clustrix does not support has the details and the links.

Evidence

The three “Verified” rows are the backends exercised by scripts/collect_execution_evidence.py, which submits a genuine job to each reachable target, waits for it, and prints what came back. Nothing in it is mocked, and a target it cannot reach is reported as skipped rather than as passing:

python scripts/collect_execution_evidence.py            # all reachable targets
python scripts/collect_execution_evidence.py slurm gpu  # a subset

Credentials come from ~/.clustrix-dev-credentials or from the environment (CLUSTRIX_SLURM_PASSWORD, CLUSTRIX_GPU_PASSWORD, HF_TOKEN).

Indices and tables