Notebook Magic Commands¶

Clustrix provides seamless integration with Jupyter notebooks through IPython magic commands and interactive widgets.

Magic Commands¶

%%clusterfy¶

The %%clusterfy magic command creates an interactive widget interface for managing cluster configurations directly in Jupyter notebooks.

Usage:

%%clusterfy
# Interactive widget appears with full configuration interface

Features:

Configuration Management: Create, edit, and delete cluster configurations
Pre-built Templates: Default configurations for major cloud providers
Interactive Forms: GUI elements for all configuration options
Save/Load: Export/import configurations as YAML or JSON files
One-click Application: Apply configurations to current session instantly

Default Templates Include:

Local development environment
AWS GPU instances (small/large)
Google Cloud CPU instances
Azure GPU instances
SLURM HPC clusters
Kubernetes clusters

Widget Interface¶

ClusterConfigWidget¶

ClusterfyMagics¶

class clustrix.notebook_magic.ClusterfyMagics(**kwargs: Any)[source]¶

Bases: Magics

IPython magic commands for Clustrix.

IPython magic command class for the %%clusterfy command.

clusterfy(line, cell)[source]¶

Create an interactive widget for managing Clustrix configurations.

Usage:

%%clusterfy

This creates a widget interface that allows you to:

Select and manage cluster configurations
Create new configurations with validation
Save/load configurations from files
Apply configurations to the current session

magics = {'cell': {'clusterfy': 'clusterfy'}, 'line': {}}¶

registered = True¶

Configuration Defaults¶

DEFAULT_CONFIGS¶

clustrix.notebook_magic.DEFAULT_CONFIGS = {'AWS EC2 Cluster': {'aws_cluster_type': 'ec2', 'aws_instance_type': 't3.medium', 'aws_region': 'us-east-1', 'cluster_type': 'aws', 'cost_monitoring': True, 'default_cores': 2, 'default_memory': '4GB', 'package_manager': 'conda', 'remote_work_dir': '/home/ec2-user/clustrix'}, 'AWS EKS Cluster': {'aws_cluster_type': 'eks', 'aws_instance_type': 't3.medium', 'aws_region': 'us-east-1', 'cluster_type': 'aws', 'cost_monitoring': True, 'default_cores': 2, 'default_memory': '4GB', 'k8s_image': 'python:3.11', 'k8s_namespace': 'default', 'package_manager': 'pip'}, 'Azure VM Cluster': {'azure_instance_type': 'Standard_D2s_v3', 'azure_region': 'eastus', 'cluster_type': 'azure', 'cost_monitoring': True, 'default_cores': 2, 'default_memory': '8GB', 'package_manager': 'conda', 'remote_work_dir': '/home/azureuser/clustrix'}, 'Corporate PBS Cluster': {'cluster_host': 'hpc.company.com', 'cluster_type': 'pbs', 'default_cores': 8, 'default_memory': '32GB', 'default_time': '02:00:00', 'package_manager': 'pip', 'remote_work_dir': '/home/employee_id/clustrix', 'username': 'employee_id'}, 'Google Cloud VM': {'cluster_type': 'gcp', 'cost_monitoring': True, 'default_cores': 2, 'default_memory': '4GB', 'gcp_instance_type': 'e2-medium', 'gcp_region': 'us-central1', 'package_manager': 'conda', 'remote_work_dir': '/home/ubuntu/clustrix'}, 'HuggingFace Space': {'cluster_type': 'huggingface_spaces', 'cost_monitoring': True, 'default_cores': 2, 'default_memory': '16GB', 'hf_hardware': 'cpu-basic', 'hf_sdk': 'gradio'}, 'Lambda Cloud GPU': {'cluster_type': 'lambda_cloud', 'cost_monitoring': True, 'default_cores': 8, 'default_memory': '32GB', 'environment_variables': {'CUDA_VISIBLE_DEVICES': '0', 'NVIDIA_VISIBLE_DEVICES': 'all'}, 'lambda_instance_type': 'gpu_1x_a10', 'package_manager': 'conda', 'remote_work_dir': '/home/ubuntu/clustrix'}, 'Local Kubernetes': {'cluster_type': 'kubernetes', 'default_cores': 2, 'default_memory': '4GB', 'k8s_image': 'python:3.11', 'k8s_namespace': 'default', 'package_manager': 'pip'}, 'Local Multi-core': {'cluster_type': 'local', 'default_cores': -1, 'default_memory': '16GB'}, 'Local Single-core': {'cluster_type': 'local', 'default_cores': 1, 'default_memory': '16GB'}, 'SGE Research Cluster': {'cluster_host': 'submit.research.org', 'cluster_type': 'sge', 'default_cores': 24, 'default_memory': '128GB', 'default_time': '04:00:00', 'package_manager': 'conda', 'remote_work_dir': '/data/researcher/clustrix', 'username': 'researcher'}, 'SSH Remote Server': {'cluster_host': 'remote.server.com', 'cluster_port': 22, 'cluster_type': 'ssh', 'default_cores': 4, 'default_memory': '16GB', 'package_manager': 'pip', 'remote_work_dir': '/tmp/clustrix', 'username': 'user'}, 'University SLURM Cluster': {'cluster_host': 'login.hpc.university.edu', 'cluster_type': 'slurm', 'default_cores': 16, 'default_memory': '64GB', 'default_time': '01:00:00', 'package_manager': 'conda', 'remote_work_dir': '/scratch/your_username/clustrix', 'username': 'your_username'}}¶

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:: d = {} for k, v in iterable:

d[k] = v
dict(**kwargs) -> new dictionary initialized with the name=value pairs: in the keyword argument list. For example: dict(one=1, two=2)

Dictionary containing pre-built cluster configuration templates.

Available Templates:

{
    "local_dev": {
        "name": "Local Development",
        "cluster_type": "local",
        "default_cores": 4,
        "default_memory": "8GB",
        "description": "Local machine for development and testing"
    },
    "aws_gpu_small": {
        "name": "AWS GPU Small",
        "cluster_type": "ssh",
        "cluster_host": "aws-instance-ip",
        "username": "ubuntu",
        "key_file": "~/.ssh/aws_key.pem",
        "default_cores": 8,
        "default_memory": "60GB",
        "description": "AWS p3.2xlarge instance (1 V100 GPU)"
    },
    # ... more templates
}

Extension Loading¶

load_ipython_extension¶

clustrix.notebook_magic.load_ipython_extension(ipython)[source]¶

Load the extension in IPython.

Automatically loads the magic command when Clustrix is imported in a Jupyter environment.

Parameters:

ipython: The IPython instance to register the magic command with

Example:

# Magic command is automatically registered when importing clustrix
import clustrix

# Use the magic command
%%clusterfy

Requirements¶

The notebook magic functionality requires:

IPython - For magic command support
ipywidgets - For the interactive widget interface
PyYAML - For YAML configuration file support

Installation:

pip install clustrix[notebook]
# or
pip install ipywidgets pyyaml

Usage Examples¶

Programmatic Access¶

from clustrix.notebook_magic import ClusterConfigWidget

# Create widget programmatically
widget = ClusterConfigWidget()
widget.display()

Custom Configuration Templates¶

from clustrix.notebook_magic import DEFAULT_CONFIGS

# View available templates
for name, config in DEFAULT_CONFIGS.items():
    print(f"{name}: {config['description']}")

# Customize templates
custom_config = {
    "name": "My Custom Cluster",
    "cluster_type": "slurm",
    "cluster_host": "my-cluster.edu",
    "username": "myuser",
    "default_cores": 32,
    "default_memory": "128GB",
    "description": "Custom high-memory cluster"
}

Save and Load Configurations¶

The widget provides built-in save/load functionality:

Save: Use the “Save Configs” button to export configurations
Load: Use the “Load Configs” button to import configurations
File Formats: Supports both YAML and JSON formats
File Naming: Automatically detects format from file extension

Error Handling¶

The widget includes comprehensive error handling:

Import Errors: Graceful fallback when IPython/ipywidgets unavailable
File Errors: Clear error messages for save/load operations
Validation: Configuration validation before application
Connection Testing: Basic connectivity checks for remote clusters

Notes¶

The magic command is automatically registered when importing clustrix in a Jupyter environment
Widget state is preserved during the notebook session
Configurations can be shared between team members via saved files
The widget works in both Jupyter Lab and Jupyter Notebook environments