Clustrix
🚀 SSH Remote Execution Tutorial¶
This tutorial demonstrates how to use Clustrix for automated SSH-based remote execution without a job scheduler. Perfect for executing functions on remote servers, workstations, or cloud instances.
✨ New: Automated SSH Key Setup¶
Clustrix now includes 15-second automated SSH key setup that eliminates manual configuration! No more spending 15-30 minutes on SSH setup.
📋 Prerequisites¶
Access to a remote server (cloud instance, workstation, or HPC login node)
Username and password for initial authentication
Python installed on the remote server
✨ That’s it! No manual SSH key setup required
[ ]:
# Install Clustrix (uncomment if needed)
# !pip install clustrix
import clustrix
from clustrix import cluster, configure, setup_ssh_keys_with_fallback
from clustrix.config import ClusterConfig
import numpy as np
print("✅ Clustrix imported successfully!")
print("📱 Look for the interactive widget that appeared above or below.")
print("🔑 You can use the widget's SSH Key Setup section for easy configuration.")
🔑 Step 1: Automated SSH Key Setup¶
This is the magic step! Instead of manually setting up SSH keys, Clustrix does it automatically.
[ ]:
# 🔧 Configure your remote server details
# Replace these with your actual server information
config = ClusterConfig(
cluster_type="ssh",
cluster_host="your-server.example.com", # Your server hostname or IP
username="your-username", # Your username on the server
port=22, # SSH port (usually 22)
# Remote execution settings
remote_work_dir="/tmp/clustrix", # Directory for temporary files
python_executable="python3", # Python command on remote server
cleanup_on_success=True, # Clean up after successful execution
max_parallel_jobs=5, # Limit concurrent executions
)
print("✅ Server configuration created!")
print(f"🎯 Target: {config.cluster_host}")
print(f"👤 User: {config.username}")
print(f"🔌 Port: {config.port}")
print("\n🔑 Ready for automated SSH key setup...")
[ ]:
# 🚀 AUTOMATED SSH KEY SETUP
# This replaces 15-30 minutes of manual work with 15 seconds of automation!
print("🔄 Setting up SSH keys automatically...")
print("💡 You'll be prompted for your password (this is normal and secure).")
print()
ssh_result = setup_ssh_keys_with_fallback(
config=config,
cluster_alias="my_server", # Creates SSH alias for easy access
key_type="ed25519", # Modern, secure key type
force_refresh=False, # Set True to generate new keys
)
# 📊 Display results
print("\n" + "="*60)
print("🔑 SSH KEY SETUP RESULTS")
print("="*60)
if ssh_result["success"]:
print("🎉 SUCCESS! SSH keys configured automatically!")
print(f"🔑 Key path: {ssh_result['key_path']}")
print(f"📦 Key already existed: {ssh_result['key_already_existed']}")
print(f"🚀 Key deployed: {ssh_result['key_deployed']}")
print(f"🔗 Connection tested: {ssh_result['connection_tested']}")
if "ssh_config_updated" in ssh_result.get("details", {}):
print("⚙️ SSH config updated with alias")
print("\n🎯 You can now connect with: ssh my_server")
print("\n✨ What just happened:")
print(" 🔐 Generated Ed25519 SSH key pair")
print(" 📤 Deployed public key to remote server")
print(" 🧹 Cleaned up any conflicting old keys")
print(" ⚙️ Updated SSH configuration")
print(" ✅ Tested connection to verify success")
else:
print("❌ SSH key setup failed")
print(f"🔍 Error: {ssh_result.get('error', 'Unknown error')}")
if "details" in ssh_result:
print("\n🔧 Troubleshooting details:")
for key, value in ssh_result["details"].items():
print(f" {key}: {value}")
print("\n💡 Try:")
print(" - Check hostname and username are correct")
print(" - Verify network connectivity to the server")
print(" - Test manual SSH connection first")
print("\n" + "="*60)
⚙️ Step 2: Configure Clustrix¶
Now that SSH keys are set up, configure Clustrix for remote execution:
[ ]:
# Configure Clustrix with the SSH setup
configure(
cluster_type="ssh",
cluster_host=config.cluster_host,
username=config.username,
port=config.port,
# Remote environment
remote_work_dir=config.remote_work_dir,
python_executable=config.python_executable,
# Execution settings
cleanup_on_success=True,
max_parallel_jobs=5,
# Optional: Remote environment activation
# conda_env_name="myenv", # Activate conda environment
# virtualenv_path="/path/to/venv", # Activate virtual environment
)
print("✅ Clustrix configured for SSH remote execution!")
print(f"🎯 Target server: {config.cluster_host}")
print(f"📁 Remote work directory: {config.remote_work_dir}")
print(f"🐍 Python executable: {config.python_executable}")
print("\n🚀 Ready to execute functions remotely!")
🧮 Example 1: Basic Remote Computation¶
Execute a simple mathematical computation remotely:
[ ]:
@cluster
def basic_remote_computation(n=1000000):
"""
Simple computation executed on remote server.
"""
import math
import time
import platform
from datetime import datetime
print(f"🖥️ Executing on: {platform.node()}")
print(f"🐍 Python version: {platform.python_version()}")
print(f"⚡ Starting computation at {datetime.now()}")
print(f"🔢 Computing sum of squares for {n:,} numbers")
start_time = time.time()
# Compute sum of squares
total = sum(i*i for i in range(n))
# Compute some mathematical functions
sqrt_total = math.sqrt(total)
log_total = math.log(total)
end_time = time.time()
execution_time = end_time - start_time
result = {
'n': n,
'sum_of_squares': total,
'sqrt_sum': sqrt_total,
'log_sum': log_total,
'execution_time_seconds': execution_time,
'hostname': platform.node(),
'python_version': platform.python_version(),
'completion_time': datetime.now().isoformat()
}
print(f"✅ Computation completed in {execution_time:.2f} seconds")
return result
# Execute on remote server
print("🚀 Executing basic computation on remote server...")
result = basic_remote_computation(500000)
print(f"\n🎉 REMOTE COMPUTATION COMPLETE")
print(f"🖥️ Executed on: {result['hostname']}")
print(f"🐍 Python version: {result['python_version']}")
print(f"🔢 Numbers processed: {result['n']:,}")
print(f"📊 Sum of squares: {result['sum_of_squares']:,}")
print(f"📐 Square root of sum: {result['sqrt_sum']:,.2f}")
print(f"⏱️ Execution time: {result['execution_time_seconds']:.2f} seconds")
print(f"🕐 Completed at: {result['completion_time']}")
📊 Example 2: Remote Data Processing with NumPy¶
Process numerical data on the remote server:
[ ]:
@cluster
def remote_numpy_computation(matrix_size=1000, num_iterations=5):
"""
Perform numerical computations using NumPy on remote server.
"""
import numpy as np
import time
import platform
from datetime import datetime
print(f"🖥️ Remote execution on: {platform.node()}")
print(f"📊 NumPy version: {np.__version__}")
print(f"🔢 Matrix size: {matrix_size}x{matrix_size}")
print(f"🔄 Iterations: {num_iterations}")
results = []
total_start_time = time.time()
for iteration in range(num_iterations):
print(f"\n🔄 Iteration {iteration + 1}/{num_iterations}")
start_time = time.time()
# Generate random matrices
print(" 📋 Generating random matrices...")
A = np.random.randn(matrix_size, matrix_size)
B = np.random.randn(matrix_size, matrix_size)
# Matrix multiplication
print(" ✖️ Performing matrix multiplication...")
C = np.dot(A, B)
# Eigenvalue computation (smaller matrix for speed)
small_size = min(100, matrix_size)
print(f" 🧮 Computing eigenvalues ({small_size}x{small_size})...")
eigenvalues = np.linalg.eigvals(A[:small_size, :small_size])
# Statistical analysis
print(" 📈 Computing statistics...")
stats = {
'matrix_mean': float(np.mean(C)),
'matrix_std': float(np.std(C)),
'matrix_max': float(np.max(C)),
'matrix_min': float(np.min(C)),
'eigenvalue_mean': float(np.mean(eigenvalues.real)),
'eigenvalue_max': float(np.max(eigenvalues.real)),
'frobenius_norm': float(np.linalg.norm(C, 'fro')),
}
end_time = time.time()
iteration_time = end_time - start_time
iteration_result = {
'iteration': iteration + 1,
'execution_time': iteration_time,
'statistics': stats
}
results.append(iteration_result)
print(f" ⏱️ Iteration completed in {iteration_time:.2f} seconds")
total_end_time = time.time()
total_time = total_end_time - total_start_time
# Aggregate statistics
execution_times = [r['execution_time'] for r in results]
final_result = {
'computation_info': {
'matrix_size': matrix_size,
'num_iterations': num_iterations,
'hostname': platform.node(),
'numpy_version': np.__version__,
'completion_time': datetime.now().isoformat()
},
'performance': {
'total_time': total_time,
'average_iteration_time': np.mean(execution_times),
'min_iteration_time': np.min(execution_times),
'max_iteration_time': np.max(execution_times),
'operations_per_second': (num_iterations * matrix_size * matrix_size) / total_time
},
'iteration_results': results
}
print(f"\n✅ All computations completed!")
print(f"⏱️ Total execution time: {total_time:.2f} seconds")
print(f"📊 Average iteration time: {np.mean(execution_times):.2f} seconds")
return final_result
# Execute numerical computation on remote server
print("🚀 Starting remote NumPy computation...")
numpy_result = remote_numpy_computation(matrix_size=500, num_iterations=3)
print(f"\n🎉 REMOTE NUMPY COMPUTATION COMPLETE")
info = numpy_result['computation_info']
print(f"🖥️ Executed on: {info['hostname']}")
print(f"📊 NumPy version: {info['numpy_version']}")
print(f"🔢 Matrix size: {info['matrix_size']}x{info['matrix_size']}")
print(f"🔄 Iterations: {info['num_iterations']}")
perf = numpy_result['performance']
print(f"\n📈 Performance Metrics:")
print(f" ⏱️ Total time: {perf['total_time']:.2f} seconds")
print(f" 📊 Average iteration: {perf['average_iteration_time']:.2f} seconds")
print(f" ⚡ Operations/second: {perf['operations_per_second']:,.0f}")
print(f" 🏃 Fastest iteration: {perf['min_iteration_time']:.2f} seconds")
print(f" 🐌 Slowest iteration: {perf['max_iteration_time']:.2f} seconds")
# Show statistics from the last iteration
if numpy_result['iteration_results']:
last_stats = numpy_result['iteration_results'][-1]['statistics']
print(f"\n📊 Final Matrix Statistics:")
print(f" 📈 Mean: {last_stats['matrix_mean']:.4f}")
print(f" 📊 Std Dev: {last_stats['matrix_std']:.4f}")
print(f" 🔺 Max: {last_stats['matrix_max']:.4f}")
print(f" 🔻 Min: {last_stats['matrix_min']:.4f}")
print(f" 🧮 Eigenvalue Mean: {last_stats['eigenvalue_mean']:.4f}")
print(f" 📏 Frobenius Norm: {last_stats['frobenius_norm']:.2f}")
🗂️ Example 3: Remote File System Analysis¶
Analyze the file system structure on the remote server:
[ ]:
@cluster
def remote_system_analysis():
"""
Analyze system information and file system on remote server.
"""
import os
import platform
import shutil
import subprocess
import psutil # Common on many systems
from datetime import datetime
print(f"🖥️ Analyzing system: {platform.node()}")
# Basic system information
system_info = {
'hostname': platform.node(),
'system': platform.system(),
'release': platform.release(),
'version': platform.version(),
'machine': platform.machine(),
'processor': platform.processor(),
'python_version': platform.python_version(),
'architecture': platform.architecture(),
}
print(f"💻 System: {system_info['system']} {system_info['release']}")
print(f"🏗️ Architecture: {system_info['machine']}")
print(f"🐍 Python: {system_info['python_version']}")
# Memory and CPU information
try:
memory = psutil.virtual_memory()
cpu_info = {
'cpu_count': psutil.cpu_count(),
'cpu_percent': psutil.cpu_percent(interval=1),
'memory_total_gb': memory.total / (1024**3),
'memory_available_gb': memory.available / (1024**3),
'memory_percent': memory.percent,
}
print(f"⚡ CPUs: {cpu_info['cpu_count']}")
print(f"🧠 Memory: {cpu_info['memory_total_gb']:.1f} GB total, {cpu_info['memory_available_gb']:.1f} GB available")
except ImportError:
print("📊 psutil not available, skipping detailed system metrics")
cpu_info = {'error': 'psutil not available'}
# Disk usage analysis
disk_info = {}
important_paths = ['/', '/home', '/tmp', '/var', '/usr']
print("\n💾 Disk Usage Analysis:")
for path in important_paths:
if os.path.exists(path):
try:
usage = shutil.disk_usage(path)
disk_info[path] = {
'total_gb': usage.total / (1024**3),
'used_gb': usage.used / (1024**3),
'free_gb': usage.free / (1024**3),
'used_percent': (usage.used / usage.total) * 100
}
print(f" 📁 {path}: {disk_info[path]['used_gb']:.1f}GB used / {disk_info[path]['total_gb']:.1f}GB total ({disk_info[path]['used_percent']:.1f}%)")
except (OSError, PermissionError):
disk_info[path] = {'error': 'Permission denied or path inaccessible'}
# Environment analysis
env_info = {
'user': os.environ.get('USER', 'unknown'),
'home': os.environ.get('HOME', 'unknown'),
'shell': os.environ.get('SHELL', 'unknown'),
'path_entries': len(os.environ.get('PATH', '').split(':')),
'working_directory': os.getcwd(),
}
print(f"\n👤 Environment Info:")
print(f" User: {env_info['user']}")
print(f" Home: {env_info['home']}")
print(f" Shell: {env_info['shell']}")
print(f" Working Dir: {env_info['working_directory']}")
# Available Python packages
print("\n🐍 Checking Python Environment:")
common_packages = [
'numpy', 'pandas', 'scipy', 'matplotlib', 'sklearn', 'requests',
'psutil', 'jupyter', 'ipython', 'pytest', 'click', 'flask'
]
package_status = {}
for package in common_packages:
try:
__import__(package)
# Try to get version
try:
mod = __import__(package)
version = getattr(mod, '__version__', 'unknown')
package_status[package] = {'available': True, 'version': version}
except:
package_status[package] = {'available': True, 'version': 'unknown'}
except ImportError:
package_status[package] = {'available': False}
available_packages = [pkg for pkg, info in package_status.items() if info['available']]
print(f" ✅ Available packages ({len(available_packages)}/{len(common_packages)}): {', '.join(available_packages[:8])}")
# Network connectivity test
network_info = {}
try:
import socket
hostname = socket.gethostname()
ip_address = socket.gethostbyname(hostname)
network_info = {
'hostname': hostname,
'ip_address': ip_address,
'connectivity': 'basic_ok'
}
print(f"\n🌐 Network: {hostname} ({ip_address})")
except Exception as e:
network_info = {'error': str(e)}
print(f"\n🌐 Network: Error getting network info")
# Final analysis result
analysis_result = {
'analysis_metadata': {
'timestamp': datetime.now().isoformat(),
'analysis_type': 'remote_system_analysis'
},
'system_information': system_info,
'performance_info': cpu_info,
'disk_usage': disk_info,
'environment': env_info,
'python_packages': package_status,
'network_info': network_info
}
print(f"\n✅ System analysis completed!")
return analysis_result
# Analyze remote system
print("🚀 Starting remote system analysis...")
system_result = remote_system_analysis()
print(f"\n🎉 REMOTE SYSTEM ANALYSIS COMPLETE")
sys_info = system_result['system_information']
print(f"🖥️ System: {sys_info['hostname']} ({sys_info['system']} {sys_info['release']})")
print(f"🏗️ Architecture: {sys_info['machine']}")
print(f"🐍 Python: {sys_info['python_version']}")
if 'error' not in system_result['performance_info']:
perf = system_result['performance_info']
print(f"\n📊 Performance:")
print(f" ⚡ CPUs: {perf['cpu_count']}")
print(f" 🧠 Memory: {perf['memory_total_gb']:.1f} GB ({perf['memory_percent']:.1f}% used)")
print(f" 🔥 CPU Usage: {perf['cpu_percent']:.1f}%")
env = system_result['environment']
print(f"\n👤 Environment:")
print(f" User: {env['user']}")
print(f" Home: {env['home']}")
print(f" Working Dir: {env['working_directory']}")
packages = system_result['python_packages']
available = [pkg for pkg, info in packages.items() if info['available']]
print(f"\n🐍 Python Environment:")
print(f" 📦 Available packages: {len(available)}/{len(packages)}")
print(f" ✅ Key packages: {', '.join(available[:6])}")
disk = system_result['disk_usage']
print(f"\n💾 Storage:")
for path, info in disk.items():
if 'error' not in info:
print(f" 📁 {path}: {info['free_gb']:.1f} GB free")
🧪 Example 4: Remote Environment Testing¶
Test specific capabilities and benchmark performance:
[ ]:
@cluster
def benchmark_remote_performance():
"""
Benchmark computational performance on remote server.
"""
import time
import math
import platform
from datetime import datetime
print(f"🏁 Starting performance benchmarks on {platform.node()}")
benchmarks = {}
# CPU benchmark: Prime number calculation
print("\n🔢 CPU Benchmark: Prime number calculation")
start_time = time.time()
def is_prime(n):
if n < 2:
return False
for i in range(2, int(math.sqrt(n)) + 1):
if n % i == 0:
return False
return True
primes = [n for n in range(2, 10000) if is_prime(n)]
cpu_time = time.time() - start_time
benchmarks['cpu_benchmark'] = {
'test': 'prime_calculation',
'range': '2-10000',
'primes_found': len(primes),
'execution_time': cpu_time,
'primes_per_second': len(primes) / cpu_time
}
print(f" ✅ Found {len(primes)} primes in {cpu_time:.3f} seconds")
print(f" 📊 Rate: {len(primes) / cpu_time:.1f} primes/second")
# Memory benchmark: List operations
print("\n🧠 Memory Benchmark: Large list operations")
start_time = time.time()
# Create large list
large_list = list(range(1000000))
# Perform operations
reversed_list = large_list[::-1]
sorted_sample = sorted(large_list[::1000])
list_sum = sum(large_list[::100])
memory_time = time.time() - start_time
benchmarks['memory_benchmark'] = {
'test': 'list_operations',
'list_size': len(large_list),
'operations': ['reverse', 'sort_sample', 'sum_subset'],
'execution_time': memory_time,
'sum_result': list_sum
}
print(f" ✅ Processed {len(large_list):,} elements in {memory_time:.3f} seconds")
print(f" 📊 Rate: {len(large_list) / memory_time:,.0f} elements/second")
# I/O benchmark: File operations
print("\n📁 I/O Benchmark: File read/write operations")
import tempfile
import os
start_time = time.time()
# Write test
test_data = "\n".join([f"Line {i}: {i*i}" for i in range(10000)])
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
temp_file = f.name
f.write(test_data)
# Read test
with open(temp_file, 'r') as f:
read_data = f.read()
# Verify and cleanup
lines_read = len(read_data.split('\n'))
file_size = os.path.getsize(temp_file)
os.unlink(temp_file)
io_time = time.time() - start_time
benchmarks['io_benchmark'] = {
'test': 'file_read_write',
'lines_written': 10000,
'lines_read': lines_read,
'file_size_bytes': file_size,
'execution_time': io_time,
'throughput_mb_per_sec': (file_size / (1024*1024)) / io_time
}
print(f" ✅ Wrote/read {file_size:,} bytes in {io_time:.3f} seconds")
print(f" 📊 Throughput: {(file_size / (1024*1024)) / io_time:.2f} MB/second")
# Mathematical benchmark: Floating point operations
print("\n🧮 Math Benchmark: Floating point operations")
start_time = time.time()
total = 0.0
for i in range(100000):
total += math.sin(i) * math.cos(i) + math.sqrt(i + 1)
math_time = time.time() - start_time
benchmarks['math_benchmark'] = {
'test': 'trigonometric_operations',
'operations_count': 100000 * 3, # sin, cos, sqrt per iteration
'result': total,
'execution_time': math_time,
'operations_per_second': (100000 * 3) / math_time
}
print(f" ✅ Performed {100000 * 3:,} operations in {math_time:.3f} seconds")
print(f" 📊 Rate: {(100000 * 3) / math_time:,.0f} operations/second")
# Summary
total_benchmark_time = sum([b['execution_time'] for b in benchmarks.values()])
result = {
'benchmark_metadata': {
'hostname': platform.node(),
'system': platform.system(),
'machine': platform.machine(),
'python_version': platform.python_version(),
'timestamp': datetime.now().isoformat(),
'total_benchmark_time': total_benchmark_time
},
'benchmarks': benchmarks
}
print(f"\n🏁 All benchmarks completed!")
print(f"⏱️ Total benchmark time: {total_benchmark_time:.3f} seconds")
return result
# Run performance benchmarks
print("🚀 Starting remote performance benchmarks...")
benchmark_result = benchmark_remote_performance()
print(f"\n🎉 REMOTE BENCHMARKS COMPLETE")
meta = benchmark_result['benchmark_metadata']
print(f"🖥️ System: {meta['hostname']} ({meta['system']} {meta['machine']})")
print(f"🐍 Python: {meta['python_version']}")
print(f"⏱️ Total time: {meta['total_benchmark_time']:.3f} seconds")
benchmarks = benchmark_result['benchmarks']
print(f"\n📊 Benchmark Results:")
cpu = benchmarks['cpu_benchmark']
print(f" 🔢 CPU: {cpu['primes_per_second']:.1f} primes/sec")
memory = benchmarks['memory_benchmark']
print(f" 🧠 Memory: {len(memory['operations'])} ops on {memory['list_size']:,} elements in {memory['execution_time']:.3f}s")
io = benchmarks['io_benchmark']
print(f" 📁 I/O: {io['throughput_mb_per_sec']:.2f} MB/sec throughput")
math_bench = benchmarks['math_benchmark']
print(f" 🧮 Math: {math_bench['operations_per_second']:,.0f} ops/sec")
print(f"\n🏆 Remote server performance profile complete!")
🔧 SSH Connection Testing and Troubleshooting¶
Test your SSH connection and get troubleshooting information:
[ ]:
def test_ssh_connection():
"""
Test SSH connection and provide troubleshooting information.
"""
from clustrix import get_config
from clustrix.executor import ClusterExecutor
try:
print("🔍 Testing SSH connection...")
config = get_config()
if config.cluster_type != 'ssh':
print("❌ Current configuration is not for SSH.")
print("💡 Please run the SSH configuration cell above first.")
return False
print(f"🎯 Target: {config.cluster_host}:{getattr(config, 'port', 22)}")
print(f"👤 User: {config.username}")
print(f"🔑 Key: {getattr(config, 'key_file', 'auto-detected')}")
# Test basic connection
executor = ClusterExecutor(config)
executor.connect()
print("✅ SSH connection successful!")
# Test basic commands
print("\n🧪 Testing basic commands...")
commands = [
("hostname", "🖥️ Remote hostname"),
("whoami", "👤 Remote user"),
("pwd", "📁 Working directory"),
("python3 --version", "🐍 Python version"),
("uname -a", "💻 System info")
]
for cmd, description in commands:
try:
stdout, stderr = executor._execute_command(cmd)
output = (stdout or stderr or "no output").strip()
print(f" ✅ {description}: {output}")
except Exception as e:
print(f" ❌ {description}: {str(e)}")
# Test work directory
work_dir = getattr(config, 'remote_work_dir', '/tmp/clustrix')
print(f"\n📁 Testing work directory: {work_dir}")
try:
stdout, stderr = executor._execute_command(f"mkdir -p {work_dir} && echo 'Directory OK'")
if "Directory OK" in stdout:
print(f" ✅ Work directory accessible and writable")
else:
print(f" ⚠️ Work directory test inconclusive")
except Exception as e:
print(f" ❌ Work directory error: {e}")
executor.disconnect()
print("\n🎉 SSH connection test completed successfully!")
print("✅ Your SSH configuration is working correctly.")
return True
except Exception as e:
print(f"\n❌ SSH connection test failed: {e}")
print("\n🔧 Troubleshooting suggestions:")
print(" 1. Check hostname and port are correct")
print(" 2. Verify username is correct")
print(" 3. Test manual SSH: ssh user@hostname")
print(" 4. Check firewall and network connectivity")
print(" 5. Try force refresh: setup_ssh_keys_with_fallback(..., force_refresh=True)")
return False
# Run connection test
print("🔍 SSH CONNECTION TEST")
print("=" * 30)
test_success = test_ssh_connection()
if test_success:
print("\n🚀 Ready for remote execution!")
else:
print("\n🔧 Please fix SSH issues before proceeding.")
📚 Summary and Best Practices¶
🎉 What You’ve Learned¶
🔑 Automated SSH Setup: 15-second setup vs 15-30 minute manual process
⚙️ Remote Configuration: Easy Clustrix setup for SSH execution
🧮 Remote Computing: Mathematical computations on remote servers
📊 Data Processing: NumPy operations and analysis remotely
🗂️ System Analysis: File system and environment inspection
🏁 Performance Testing: Benchmarking remote server capabilities
🔧 Troubleshooting: Connection testing and problem resolution
🔒 Security Best Practices¶
✅ Use SSH keys: Automated setup creates secure Ed25519 keys
✅ Unique keys: Different keys for different servers
✅ Regular rotation: Use
force_refresh=Trueperiodically✅ Secure storage: Keys stored with proper permissions (600/644)
✅ Clean up: Enable
cleanup_on_success=True✅ Monitor access: Check SSH logs on your servers
💡 Performance Tips¶
Parallel execution: Set
max_parallel_jobsappropriatelyWork directory: Use fast storage (e.g.,
/tmpor SSD)Environment setup: Use conda/virtualenv for package management
Data transfer: Minimize large data transfers between local/remote
Connection reuse: Clustrix automatically reuses SSH connections
🎯 When to Use SSH vs Other Cluster Types¶
Choose SSH when:
Working with single servers or workstations
Need immediate execution (no queuing)
Prototyping and development
Cloud instances (AWS, GCP, Azure)
Personal computing resources
Choose SLURM/PBS/SGE when:
Large HPC clusters with job schedulers
Need resource management and fair sharing
Production workloads with resource constraints
Long-running computations requiring scheduling
Choose Kubernetes when:
Containerized execution environments
Auto-scaling and fault tolerance needed
Cloud-native applications
Microservices architecture
🚀 Next Steps¶
Try other tutorials:
SLURM Tutorial for HPC clusters
Kubernetes Tutorial for container orchestration
Cost Monitoring Tutorial for cloud costs
Explore advanced features:
Multiple cluster configurations
Custom environment setup
Filesystem utilities
Cloud provider integrations
Read documentation:
SSH Setup Guide for detailed configuration
API Documentation for advanced options
Clustrix Documentation for comprehensive guides
🎊 Congratulations!¶
You’ve successfully learned how to use Clustrix’s automated SSH setup and remote execution capabilities. You can now:
⚡ Set up SSH access in 15 seconds instead of 15-30 minutes
🚀 Execute Python functions on any SSH-accessible server
📊 Perform complex computations remotely
🔧 Troubleshoot and optimize your setup
🔒 Maintain security best practices
Happy remote computing! 🎉