Parallel File Processor

Quick Start

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd

def process_csv(file_path: Path) -> dict:
    """Process a single CSV file."""
    df = pd.read_csv(file_path)
    return {'file': file_path.name, 'rows': len(df), 'columns': len(df.columns)}

# Get all CSV files
files = list(Path('data/raw/').glob('*.csv'))

# Process in parallel
results = []
with ThreadPoolExecutor(max_workers=8) as executor:
    futures = {executor.submit(process_csv, f): f for f in files}
    for future in as_completed(futures):
        results.append(future.result())

print(f"Processed {len(results)} files")

When to Use

• Processing large numbers of files (100+ files)
• Batch operations on directory contents
• Extracting data from multiple ZIP archives
• Aggregating results from parallel operations
• CPU-bound file transformations
• IO-bound file operations with proper concurrency

Related Skills

• data-pipeline-processor - Data transformation
• yaml-workflow-executor - Workflow automation
• engineering-report-generator - Report generation

Version History

• 1.1.0 (2026-01-02): Upgraded to SKILL_TEMPLATE_v2 format with Quick Start, Error Handling, Metrics, Execution Checklist, additional examples
• 1.0.0 (2024-10-15): Initial release with FileScanner, ParallelProcessor, progress tracking, result aggregation

Sub-Skills

• Example 1: Process CSV Files (+3)
• Do (+1)

Sub-Skills

• Error Handling
• Execution Checklist
• Metrics

Sub-Skills

• Core Pattern
• Core Components (+5)
• Basic Configuration
• Mode Selection (+2)