Agent skill
spark
Apache Spark distributed computing. Use for big data processing.
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/other/other/spark
SKILL.md
Apache Spark
Spark is the king of Big Data. v4.0 (2024/2025) makes Spark Connect the default, allowing thin clients (like VS Code) to connect to massive clusters easily.
When to Use
- Data Engineering: ETL at Petabyte scale.
- Streaming: Structured Streaming for real-time analytics.
- Legacy ML:
spark.ml(though mostly replaced by XGBoost/Torch).
Core Concepts
Spark Connect
Decouples client (your laptop) from server (the cluster). Allows using Spark from Go/Rust/TypeScript.
Catalyst Optimizer
Optimizes your SQL/DataFrame queries before execution.
RDD
The low-level API. Almost never used directly in modern Spark.
Best Practices (2025)
Do:
- Use PySpark: It is now a first-class citizen with Python UDF profiling.
- Use Delta Lake / Iceberg: Spark works best with modern table formats.
- Use
pandas_udf: For vectorized Python UDFs.
Don't:
- Don't use
rdd.map: It is slow (Python serialization). Use DataFrames.
References
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
Didn't find tool you were looking for?