Configuration API¶

Clustrix provides flexible configuration management for cluster settings, authentication, and execution preferences.

class clustrix.config.ClusterConfig(api_key=None, username=None, password=None, key_file=None, cluster_type='slurm', cluster_host=None, cluster_port=22, k8s_namespace='default', k8s_image='python:3.11-slim', k8s_service_account=None, k8s_pull_policy='IfNotPresent', k8s_job_ttl_seconds=3600, k8s_backoff_limit=3, k8s_remote=False, cloud_provider='manual', cloud_region=None, cloud_auto_configure=False, auto_provision_k8s=False, k8s_provider='aws', k8s_from_scratch=True, k8s_auto_cleanup=True, k8s_cluster_name=None, k8s_node_count=2, k8s_node_type=None, k8s_version='1.28', k8s_region=None, aws_profile=None, aws_access_key_id=None, aws_secret_access_key=None, aws_access_key=None, aws_secret_key=None, aws_session_token=None, aws_instance_type=None, aws_cluster_type=None, eks_cluster_name=None, aws_region=None, azure_subscription_id=None, azure_resource_group=None, azure_tenant_id=None, azure_client_id=None, azure_client_secret=None, azure_instance_type=None, aks_cluster_name=None, azure_region=None, gcp_project_id=None, gcp_zone=None, gcp_service_account_key=None, gcp_instance_type=None, gke_cluster_name=None, gcp_region=None, lambda_instance_type=None, lambda_api_key=None, hf_hardware=None, hf_token=None, hf_username=None, hf_sdk=None, default_cores=4, default_memory='8GB', default_time='01:00:00', default_partition=None, default_queue=None, remote_work_dir='/tmp/clustrix', local_work_dir=None, local_cache_dir='~/.clustrix/cache', conda_env_name=None, python_executable='python', package_manager='pip', auto_parallel=True, auto_gpu_parallel=True, max_parallel_jobs=100, max_gpu_parallel_jobs=8, job_poll_interval=30, cleanup_on_success=True, prefer_local_parallel=False, local_parallel_threshold=1000, async_submit=False, use_two_venv=True, venv_setup_timeout=300, cost_monitoring=False, use_env_password=False, password_env_var='', cache_credentials=True, credential_cache_ttl=300, ssh_port=22, environment_variables=None, module_loads=None, pre_execution_commands=None, cluster_packages=None, venv_post_install_commands=None, gpu_detection_enabled=True, auto_gpu_packages=True, cuda_version_preference=None, gpu_memory_fraction=0.9, prefer_gpu_execution=True, gpu_requirements=None, rapids_ecosystem=False, venv_info=None)[source]¶

Bases: object

Configuration settings for cluster execution.

api_key: Optional[str] = None¶

username: Optional[str] = None¶

password: Optional[str] = None¶

key_file: Optional[str] = None¶

cluster_type: str = 'slurm'¶

cluster_host: Optional[str] = None¶

cluster_port: int = 22¶

k8s_namespace: str = 'default'¶

k8s_image: str = 'python:3.11-slim'¶

k8s_service_account: Optional[str] = None¶

k8s_pull_policy: str = 'IfNotPresent'¶

k8s_job_ttl_seconds: int = 3600¶

k8s_backoff_limit: int = 3¶

k8s_remote: bool = False¶

cloud_provider: str = 'manual'¶

cloud_region: Optional[str] = None¶

cloud_auto_configure: bool = False¶

auto_provision_k8s: bool = False¶

k8s_provider: str = 'aws'¶

k8s_from_scratch: bool = True¶

k8s_auto_cleanup: bool = True¶

k8s_cluster_name: Optional[str] = None¶

k8s_node_count: int = 2¶

k8s_node_type: Optional[str] = None¶

k8s_version: str = '1.28'¶

k8s_region: Optional[str] = None¶

aws_profile: Optional[str] = None¶

aws_access_key_id: Optional[str] = None¶

aws_secret_access_key: Optional[str] = None¶

aws_access_key: Optional[str] = None¶

aws_secret_key: Optional[str] = None¶

aws_session_token: Optional[str] = None¶

aws_instance_type: Optional[str] = None¶

aws_cluster_type: Optional[str] = None¶

eks_cluster_name: Optional[str] = None¶

aws_region: Optional[str] = None¶

azure_subscription_id: Optional[str] = None¶

azure_resource_group: Optional[str] = None¶

azure_tenant_id: Optional[str] = None¶

azure_client_id: Optional[str] = None¶

azure_client_secret: Optional[str] = None¶

azure_instance_type: Optional[str] = None¶

aks_cluster_name: Optional[str] = None¶

azure_region: Optional[str] = None¶

gcp_project_id: Optional[str] = None¶

gcp_zone: Optional[str] = None¶

gcp_service_account_key: Optional[str] = None¶

gcp_instance_type: Optional[str] = None¶

gke_cluster_name: Optional[str] = None¶

gcp_region: Optional[str] = None¶

lambda_instance_type: Optional[str] = None¶

lambda_api_key: Optional[str] = None¶

hf_hardware: Optional[str] = None¶

hf_token: Optional[str] = None¶

hf_username: Optional[str] = None¶

hf_sdk: Optional[str] = None¶

default_cores: int = 4¶

default_memory: str = '8GB'¶

default_time: str = '01:00:00'¶

default_partition: Optional[str] = None¶

default_queue: Optional[str] = None¶

remote_work_dir: str = '/tmp/clustrix'¶

local_work_dir: Optional[str] = None¶

local_cache_dir: str = '~/.clustrix/cache'¶

conda_env_name: Optional[str] = None¶

python_executable: str = 'python'¶

package_manager: str = 'pip'¶

auto_parallel: bool = True¶

auto_gpu_parallel: bool = True¶

max_parallel_jobs: int = 100¶

max_gpu_parallel_jobs: int = 8¶

job_poll_interval: int = 30¶

cleanup_on_success: bool = True¶

prefer_local_parallel: bool = False¶

local_parallel_threshold: int = 1000¶

async_submit: bool = False¶

use_two_venv: bool = True¶

venv_setup_timeout: int = 300¶

cost_monitoring: bool = False¶

use_env_password: bool = False¶

password_env_var: str = ''¶

cache_credentials: bool = True¶

credential_cache_ttl: int = 300¶

ssh_port: int = 22¶

environment_variables: Optional[Dict[str, str]] = None¶

module_loads: Optional[list] = None¶

pre_execution_commands: Optional[list] = None¶

cluster_packages: Optional[list] = None¶

venv_post_install_commands: Optional[list] = None¶

gpu_detection_enabled: bool = True¶

auto_gpu_packages: bool = True¶

cuda_version_preference: Optional[str] = None¶

gpu_memory_fraction: float = 0.9¶

prefer_gpu_execution: bool = True¶

gpu_requirements: Optional[Dict[str, Any]] = None¶

rapids_ecosystem: bool = False¶

venv_info: Optional[dict] = None¶

get_env_password()[source]¶

Get password from specified environment variable.

Return type:: Optional[str]

save_to_file(config_path)[source]¶

Save this configuration instance to a file.

Return type:: None

classmethod load_from_file(config_path)[source]¶

Load configuration from a file and return a new instance.

Return type:: ClusterConfig

__init__(api_key=None, username=None, password=None, key_file=None, cluster_type='slurm', cluster_host=None, cluster_port=22, k8s_namespace='default', k8s_image='python:3.11-slim', k8s_service_account=None, k8s_pull_policy='IfNotPresent', k8s_job_ttl_seconds=3600, k8s_backoff_limit=3, k8s_remote=False, cloud_provider='manual', cloud_region=None, cloud_auto_configure=False, auto_provision_k8s=False, k8s_provider='aws', k8s_from_scratch=True, k8s_auto_cleanup=True, k8s_cluster_name=None, k8s_node_count=2, k8s_node_type=None, k8s_version='1.28', k8s_region=None, aws_profile=None, aws_access_key_id=None, aws_secret_access_key=None, aws_access_key=None, aws_secret_key=None, aws_session_token=None, aws_instance_type=None, aws_cluster_type=None, eks_cluster_name=None, aws_region=None, azure_subscription_id=None, azure_resource_group=None, azure_tenant_id=None, azure_client_id=None, azure_client_secret=None, azure_instance_type=None, aks_cluster_name=None, azure_region=None, gcp_project_id=None, gcp_zone=None, gcp_service_account_key=None, gcp_instance_type=None, gke_cluster_name=None, gcp_region=None, lambda_instance_type=None, lambda_api_key=None, hf_hardware=None, hf_token=None, hf_username=None, hf_sdk=None, default_cores=4, default_memory='8GB', default_time='01:00:00', default_partition=None, default_queue=None, remote_work_dir='/tmp/clustrix', local_work_dir=None, local_cache_dir='~/.clustrix/cache', conda_env_name=None, python_executable='python', package_manager='pip', auto_parallel=True, auto_gpu_parallel=True, max_parallel_jobs=100, max_gpu_parallel_jobs=8, job_poll_interval=30, cleanup_on_success=True, prefer_local_parallel=False, local_parallel_threshold=1000, async_submit=False, use_two_venv=True, venv_setup_timeout=300, cost_monitoring=False, use_env_password=False, password_env_var='', cache_credentials=True, credential_cache_ttl=300, ssh_port=22, environment_variables=None, module_loads=None, pre_execution_commands=None, cluster_packages=None, venv_post_install_commands=None, gpu_detection_enabled=True, auto_gpu_packages=True, cuda_version_preference=None, gpu_memory_fraction=0.9, prefer_gpu_execution=True, gpu_requirements=None, rapids_ecosystem=False, venv_info=None)¶

clustrix.config.configure(auto_install_deps=True, **kwargs)[source]¶

Configure Clustrix settings.

Parameters:

auto_install_deps (bool) – Whether to automatically install cloud provider dependencies
**kwargs – Configuration parameters matching ClusterConfig fields

Return type:

None

clustrix.config.load_config(config_path)[source]¶

Load configuration from a file (JSON or YAML).

Parameters:: config_path (str) – Path to configuration file
Return type:: None

clustrix.config.save_config(config_path)[source]¶

Save current configuration to a file.

Parameters:: config_path (str) – Path where to save configuration
Return type:: None

clustrix.config.get_config()[source]¶

Get current configuration.

Return type:: ClusterConfig

Configuration Methods¶

Programmatic Configuration¶

import clustrix

clustrix.configure(
    cluster_type='slurm',
    cluster_host='cluster.example.com',
    username='myuser',
    key_file='~/.ssh/id_rsa',
    default_cores=8,
    default_memory='16GB'
)

Configuration File¶

Create a clustrix.yml file:

cluster_type: slurm
cluster_host: cluster.example.com
username: myuser
key_file: ~/.ssh/id_rsa

default_cores: 8
default_memory: 16GB
default_time: "02:00:00"
default_partition: gpu

remote_work_dir: /scratch/myuser/clustrix
conda_env_name: myproject

auto_parallel: true
max_parallel_jobs: 50
cleanup_on_success: true

module_loads:
  - python/3.9
  - cuda/11.2

environment_variables:
  CUDA_VISIBLE_DEVICES: "0,1"

Environment Variables¶

Set configuration via environment variables:

export CLUSTRIX_CLUSTER_TYPE=slurm
export CLUSTRIX_CLUSTER_HOST=cluster.example.com
export CLUSTRIX_USERNAME=myuser

Configuration Options¶

Authentication¶

username: SSH username
password: SSH password (not recommended)
key_file: Path to SSH private key file

Cluster Settings¶

cluster_type: Type of cluster (slurm, pbs, sge, kubernetes, ssh)
cluster_host: Hostname of cluster head node
cluster_port: SSH port (default: 22)

Resource Defaults¶

default_cores: Default number of CPU cores
default_memory: Default memory allocation
default_time: Default time limit
default_partition: Default partition/queue

Execution Preferences¶

auto_parallel: Enable automatic loop parallelization
max_parallel_jobs: Maximum number of parallel jobs
prefer_local_parallel: Prefer local over remote parallel execution
cleanup_on_success: Clean up remote files after successful execution