Filesystem Utilities

Unified filesystem operations for local and remote clusters.

This module provides a consistent interface for filesystem operations that work both locally and on remote clusters based on the ClusterConfig object.

class clustrix.filesystem.FileInfo(size, modified, is_dir, permissions, name='')

Bases: object

File information structure.

__init__(size, modified, is_dir, permissions, name='')

Initialize FileInfo with file metadata.

property is_file

Check if this is a file (not a directory).

property modified_datetime

Get modified time as datetime object.

__repr__()

String representation of FileInfo.

__eq__(other)

Check equality with another FileInfo object.

class clustrix.filesystem.DiskUsage(total_bytes, file_count)

Bases: object

Disk usage information.

__init__(total_bytes, file_count)

Initialize DiskUsage with usage statistics.

property total_mb: float

Total size in megabytes.

property total_gb: float

Total size in gigabytes.

__repr__()

String representation of DiskUsage.

__eq__(other)

Check equality with another DiskUsage object.

class clustrix.filesystem.ClusterFilesystem(config)

Bases: object

Unified filesystem operations for local and remote clusters.

__init__(config)

Initialize filesystem with cluster configuration.

__del__()

Clean up SSH connections.

ls(path='.')

List directory contents.

Return type:

List[str]

find(pattern, path='.')

Find files matching pattern.

Return type:

List[str]

stat(path)

Get file/directory information.

Return type:

FileInfo

exists(path)

Check if file/directory exists.

Return type:

bool

isdir(path)

Check if path is a directory.

Return type:

bool

isfile(path)

Check if path is a file.

Return type:

bool

glob(pattern, path='.')

Pattern matching for files.

Return type:

List[str]

du(path='.')

Get directory usage information.

Return type:

DiskUsage

count_files(path='.', pattern='*')

Count files in directory matching pattern.

Return type:

int

clustrix.filesystem.cluster_ls(path='.', config=None)

List directory contents locally or remotely based on config.

Return type:

List[str]

clustrix.filesystem.cluster_find(pattern, path='.', config=None)

Find files matching pattern locally or remotely based on config.

Return type:

List[str]

clustrix.filesystem.cluster_stat(path, config=None)

Get file information locally or remotely based on config.

Return type:

FileInfo

clustrix.filesystem.cluster_exists(path, config=None)

Check if file/directory exists locally or remotely based on config.

Return type:

bool

clustrix.filesystem.cluster_isdir(path, config=None)

Check if path is directory locally or remotely based on config.

Return type:

bool

clustrix.filesystem.cluster_isfile(path, config=None)

Check if path is file locally or remotely based on config.

Return type:

bool

clustrix.filesystem.cluster_glob(pattern, path='.', config=None)

Pattern matching for files locally or remotely based on config.

Return type:

List[str]

clustrix.filesystem.cluster_du(path='.', config=None)

Get directory usage locally or remotely based on config.

Return type:

DiskUsage

clustrix.filesystem.cluster_count_files(path='.', pattern='*', config=None)

Count files matching pattern locally or remotely based on config.

Return type:

int

Overview

The filesystem utilities module provides a unified interface for filesystem operations that work seamlessly across local and remote clusters. All operations use the same API regardless of whether you’re working locally or on a remote cluster.

Key Features

• Unified API: Same function calls work locally and remotely

• Automatic SSH Management: Transparent connection handling for remote operations

• Path Normalization: Consistent path handling across platforms

• Data Structures: Structured returns via FileInfo and DiskUsage classes

• Config-Driven: Uses ClusterConfig to determine local vs remote execution

Core Functions

Directory Operations

clustrix.cluster_ls(path='.', config=None)

List directory contents locally or remotely based on config.

Return type:

List[str]

clustrix.cluster_find(pattern, path='.', config=None)

Find files matching pattern locally or remotely based on config.

Return type:

List[str]

clustrix.cluster_glob(pattern, path='.', config=None)

Pattern matching for files locally or remotely based on config.

Return type:

List[str]

clustrix.cluster_count_files(path='.', pattern='*', config=None)

Count files matching pattern locally or remotely based on config.

Return type:

int

File Operations

clustrix.cluster_stat(path, config=None)

Get file information locally or remotely based on config.

Return type:

FileInfo

clustrix.cluster_exists(path, config=None)

Check if file/directory exists locally or remotely based on config.

Return type:

bool

clustrix.cluster_isdir(path, config=None)

Check if path is directory locally or remotely based on config.

Return type:

bool

clustrix.cluster_isfile(path, config=None)

Check if path is file locally or remotely based on config.

Return type:

bool

Storage Operations

clustrix.cluster_du(path='.', config=None)

Get directory usage locally or remotely based on config.

Return type:

DiskUsage

Data Classes

class clustrix.filesystem.FileInfo(size, modified, is_dir, permissions, name='')

File information structure.

__init__(size, modified, is_dir, permissions, name='')

Initialize FileInfo with file metadata.

property is_file

Check if this is a file (not a directory).

property modified_datetime

Get modified time as datetime object.

__repr__()

String representation of FileInfo.

__eq__(other)

Check equality with another FileInfo object.

class clustrix.filesystem.DiskUsage(total_bytes, file_count)

Disk usage information.

__init__(total_bytes, file_count)

Initialize DiskUsage with usage statistics.

property total_mb: float

Total size in megabytes.

property total_gb: float

Total size in gigabytes.

__repr__()

String representation of DiskUsage.

__eq__(other)

Check equality with another DiskUsage object.

Core Implementation

class clustrix.filesystem.ClusterFilesystem(config)

Bases: object

Unified filesystem operations for local and remote clusters.

__init__(config)

Initialize filesystem with cluster configuration.

__del__()

Clean up SSH connections.

ls(path='.')

List directory contents.

Return type:

List[str]

find(pattern, path='.')

Find files matching pattern.

Return type:

List[str]

stat(path)

Get file/directory information.

Return type:

FileInfo

exists(path)

Check if file/directory exists.

Return type:

bool

isdir(path)

Check if path is a directory.

Return type:

bool

isfile(path)

Check if path is a file.

Return type:

bool

glob(pattern, path='.')

Pattern matching for files.

Return type:

List[str]

du(path='.')

Get directory usage information.

Return type:

DiskUsage

count_files(path='.', pattern='*')

Count files in directory matching pattern.

Return type:

int

Usage Examples

Basic Operations

from clustrix import cluster_ls, cluster_find, cluster_stat
from clustrix.config import ClusterConfig

# Configure for remote cluster
config = ClusterConfig(
    cluster_type="slurm",
    cluster_host="cluster.edu",
    username="researcher",
    remote_work_dir="/scratch/project"
)

# List directory contents
files = cluster_ls("data/", config)

# Find CSV files recursively
csv_files = cluster_find("*.csv", "datasets/", config)

# Get file information
file_info = cluster_stat("large_dataset.h5", config)
print(f"Size: {file_info.size:,} bytes")

Data-Driven Workflows

from clustrix import cluster, cluster_glob, cluster_stat

@cluster(cores=8)
def process_datasets(config):
    # Find all data files on the cluster
    data_files = cluster_glob("*.csv", "input/", config)

    results = []
    for filename in data_files:  # Loop gets parallelized automatically
        # Check file size before processing
        file_info = cluster_stat(filename, config)
        if file_info.size > 100_000_000:  # Large files
            result = process_large_file(filename, config)
        else:
            result = process_small_file(filename, config)
        results.append(result)

    return results

Local vs Remote Operations

# Local configuration
local_config = ClusterConfig(cluster_type="local", local_work_dir="./data")

# Remote configuration
remote_config = ClusterConfig(
    cluster_type="slurm",
    cluster_host="cluster.edu",
    username="researcher"
)

# Same function calls work for both
local_files = cluster_ls(".", local_config)
remote_files = cluster_ls(".", remote_config)

Pattern Matching

# Find all Python files
py_files = cluster_find("*.py", "src/", config)

# Use glob patterns
data_files = cluster_glob("data_*.{csv,json}", "input/", config)

# Count files by type
total_files = cluster_count_files(".", "*", config)
python_files = cluster_count_files(".", "*.py", config)

Directory Usage Analysis

# Get directory usage information
usage = cluster_du("/scratch/project", config)
print(f"Total size: {usage.total_gb:.2f} GB")
print(f"File count: {usage.file_count:,}")
print(f"Average file size: {usage.total_mb/usage.file_count:.1f} MB")

Error Handling

from clustrix.filesystem import FileNotFoundError

try:
    file_info = cluster_stat("nonexistent.txt", config)
except FileNotFoundError:
    print("File does not exist")

# Safe existence check
if cluster_exists("results/output.json", config):
    file_info = cluster_stat("results/output.json", config)

Best Practices

1. Use config-driven execution: Pass ClusterConfig objects to enable local/remote switching

2. Check file existence: Use cluster_exists() before operations that assume file presence

3. Handle large directories carefully: Remote operations on large directories may be slow

4. Use appropriate patterns: Leverage cluster_find() and cluster_glob() for efficient file discovery

5. Cache results: Store file listings locally when processing many files

See Also

• Filesystem Utilities Tutorial - Comprehensive tutorial with examples

• Configuration API - Configuration management

• Decorator API - Using filesystem utilities with the @cluster decorator