Decorator API

The @cluster decorator is the main interface for Clustrix, allowing you to easily execute functions on remote clusters or locally with parallelization.

clustrix.decorator.cluster(_func=None, *, cores=None, memory=None, time=None, partition=None, queue=None, parallel=None, auto_gpu_parallel=None, environment=None, async_submit=None, provider=None, instance_type=None, region=None, platform=None, auto_provision=None, cluster_name=None, node_count=None, node_type=None, kubernetes_version=None, from_scratch=None, **kwargs)[source]

Decorator to execute functions on a cluster.

Parameters:

  • cores (Optional[int]) – Number of CPU cores to request

  • memory (Optional[str]) – Memory to request (e.g., “8GB”)

  • time (Optional[str]) – Time limit (e.g., “01:00:00”)

  • partition (Optional[str]) – Cluster partition to use

  • queue (Optional[str]) – Queue to submit to

  • parallel (Optional[bool]) – Whether to parallelize loops automatically

  • auto_gpu_parallel (Optional[bool]) – Whether to automatically parallelize across GPUs

  • environment (Optional[str]) – Conda environment name

  • async_submit (Optional[bool]) – Whether to submit jobs asynchronously (non-blocking)

  • provider (Optional[str]) – Cloud provider to use (‘lambda’, ‘aws’, ‘azure’, ‘gcp’, ‘huggingface’)

  • instance_type (Optional[str]) – Cloud instance type (e.g., ‘gpu_1x_a100’ for Lambda Cloud)

  • region (Optional[str]) – Cloud region (e.g., ‘us-east-1’)

  • NEW (#) – Kubernetes auto-provisioning parameters

  • platform (Optional[str]) – Execution platform (‘kubernetes’ to enable K8s execution)

  • auto_provision (Optional[bool]) – Whether to automatically provision K8s cluster if needed

  • cluster_name (Optional[str]) – Name for the auto-provisioned cluster

  • node_count (Optional[int]) – Number of worker nodes in the cluster

  • node_type (Optional[str]) – Cloud-specific node instance type

  • kubernetes_version (Optional[str]) – Kubernetes version to install

  • from_scratch (Optional[bool]) – Whether to create all infrastructure from scratch

  • **kwargs – Additional job parameters

Returns:

Decorated function that executes on cluster If async_submit=True, returns AsyncJobResult for non-blocking execution

Examples

Basic Usage

from clustrix import cluster

@cluster(cores=4, memory='8GB')
def my_function(x, y):
    return x + y

result = my_function(10, 20)

Resource Specification

@cluster(
    cores=16,
    memory='32GB',
    time='04:00:00',
    partition='gpu'
)
def gpu_computation():
    # Your GPU code here
    pass

Parallel Loop Execution

@cluster(cores=8, parallel=True)
def parallel_processing(data):
    results = []
    for item in data:  # This loop will be parallelized
        results.append(expensive_operation(item))
    return results

Local Parallelization

# Configure for local execution
import clustrix
clustrix.configure(cluster_host=None)

@cluster(cores=8, parallel=True)
def local_parallel_function(data):
    # Executes locally using multiprocessing
    return [process_item(item) for item in data]