Agent skill
parallel-file-processor-mode-selection
Sub-skill of parallel-file-processor: Mode Selection (+2).
Install this agent skill to your Project
npx add-skill https://github.com/vamseeachanta/workspace-hub/tree/main/.claude/skills/_archive/development/parallel-file-processor/mode-selection
SKILL.md
Mode Selection (+2)
Mode Selection
| Workload Type | Recommended Mode | Reason |
|---|---|---|
| File I/O | THREAD_POOL |
IO-bound, threads avoid GIL issues |
| Data parsing | THREAD_POOL |
Pandas releases GIL during IO |
| CPU computation | PROCESS_POOL |
Bypasses GIL for true parallelism |
| Network requests | ASYNC |
Best for many concurrent connections |
| Simple operations | SEQUENTIAL |
Overhead may exceed benefit |
Worker Count
import os
# IO-bound (reading files, network)
io_workers = os.cpu_count() * 2
# CPU-bound (heavy computation)
cpu_workers = os.cpu_count()
# Memory-constrained (large files)
memory_workers = max(2, os.cpu_count() // 2)
Batch Size
- Small files (<1MB): Large batches (500-1000)
- Medium files (1-100MB): Medium batches (50-100)
- Large files (>100MB): Small batches (10-20) or one at a time
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
vamseeachanta/workspace-hub
gsd-complete-milestone
Archive completed milestone and prepare for next version
vamseeachanta/workspace-hub
gsd-reapply-patches
Reapply local modifications after a GSD update
vamseeachanta/workspace-hub
gsd-verify-work
Validate built features through conversational UAT
vamseeachanta/workspace-hub
gsd-thread
Manage persistent context threads for cross-session work
vamseeachanta/workspace-hub
clinical-trial-protocol
Generate clinical trial protocols for medical devices or drugs through a modular, waypoint-based architecture with research-only and full protocol modes.
vamseeachanta/workspace-hub
single-cell-rna-qc
Performs quality control on single-cell RNA-seq data (.h5ad or .h5 files) using scverse best practices with MAD-based filtering and comprehensive visualizations.
Didn't find tool you were looking for?