Agent skill
polars-expertise
This skill should be used when the user asks about Polars DataFrame library (Apache Arrow) for Python or Rust. Triggers: "polars expressions", "lazy vs eager", "scan_parquet streaming", "convert pandas to polars", "pyspark to polars", "kdb to polars", "group_by_dynamic", "rolling_mean", "polars window functions", "asof join", "polars GPU", "polars parquet", "LazyFrame". Time series: OHLCV resampling, rolling windows, financial data patterns. Performance: native expressions over map_elements, early projection, categorical types, streaming.
Install this agent skill to your Project
npx add-skill https://github.com/DeevsDeevs/agent-system/tree/main/polars-expertise
SKILL.md
Polars
High-performance DataFrame library built on Apache Arrow. Supports Python and Rust with expression-based API, lazy evaluation, and automatic parallelization.
Quick Start
Python
uv pip install polars
# GPU support: uv pip install polars[gpu]
import polars as pl
# Eager: immediate execution
df = pl.DataFrame({"symbol": ["AAPL", "GOOG"], "price": [150.0, 140.0]})
df.filter(pl.col("price") > 145).select("symbol", "price")
# Lazy: optimized execution (preferred for large data)
lf = pl.scan_parquet("trades.parquet")
result = lf.filter(pl.col("volume") > 1000).group_by("symbol").agg(
pl.col("price").mean().alias("avg_price")
).collect()
Rust
# Cargo.toml - select features you need
[dependencies]
polars = { version = "0.46", features = ["lazy", "parquet", "temporal"] }
use polars::prelude::*;
fn main() -> PolarsResult<()> {
// Eager
let df = df![
"symbol" => ["AAPL", "GOOG"],
"price" => [150.0, 140.0]
]?;
// Lazy (preferred)
let lf = LazyFrame::scan_parquet("trades.parquet", Default::default())?;
let result = lf
.filter(col("volume").gt(lit(1000)))
.group_by([col("symbol")])
.agg([col("price").mean().alias("avg_price")])
.collect()?;
Ok(())
}
Core Pattern: Expressions
Everything in Polars is an expression. Expressions are composable, lazy, and parallelized.
# Expression building blocks
pl.col("price") # column reference
pl.col("price") * pl.col("volume") # arithmetic
pl.col("price").mean().over("symbol") # window function
pl.when(cond).then(a).otherwise(b) # conditional
Expressions execute in contexts: select(), with_columns(), filter(), group_by().agg()
When to Use Lazy
Use Lazy (scan_*, .lazy()) |
Use Eager (read_*) |
|---|---|
| Large files (> RAM) | Small data, exploration |
| Complex pipelines | Simple one-off ops |
| Need query optimization | Interactive notebooks |
| Streaming required | Immediate feedback |
Lazy benefits: predicate pushdown, projection pushdown, parallel execution, streaming.
Style: Use .alias() for Column Naming
Always use .alias("name") instead of name=expr kwargs:
# GOOD: Explicit .alias() - works everywhere, composable
df.with_columns(
(pl.col("price") * pl.col("volume")).alias("value"),
pl.col("price").mean().over("symbol").alias("avg_price")
)
# AVOID: Kwarg style - less flexible, doesn't chain
df.with_columns(
value=pl.col("price") * pl.col("volume"), # avoid
avg_price=pl.col("price").mean().over("symbol") # avoid
)
.alias() is explicit, chains with other methods, and works consistently in all contexts.
Anti-Patterns - AVOID
# BAD: Python functions kill parallelization
df.with_columns(pl.col("x").map_elements(lambda x: x * 2)) # SLOW
# GOOD: Native expressions are parallel
df.with_columns((pl.col("x") * 2).alias("x")) # FAST
# BAD: Row iteration
for row in df.iter_rows(): # SLOW
process(row)
# GOOD: Columnar operations
df.with_columns(process_expr) # FAST
# BAD: Late projection
lf.filter(...).collect().select("a", "b") # reads all columns
# GOOD: Early projection
lf.select("a", "b").filter(...).collect() # reads only needed columns
Performance Checklist
- Using
scan_*(lazy) for large files? - Projecting columns early in pipeline?
- Using native expressions (no
map_elements)? - Categorical dtype for low-cardinality strings?
- Appropriate integer sizes (i32 vs i64)?
- Streaming for out-of-memory data? (
collect(engine="streaming"))
Reference Navigator
Python References
| Topic | File | When to Load |
|---|---|---|
| Expressions, types, lazy/eager | python/core_concepts.md | Understanding fundamentals |
| Select, filter, group_by, window | python/operations.md | Common operations |
| CSV, Parquet, streaming I/O | python/io_guide.md | Reading/writing data |
| Joins, pivots, reshaping | python/transformations.md | Combining/reshaping data |
| Performance, patterns | python/best_practices.md | Optimization |
Rust References
| Topic | File | When to Load |
|---|---|---|
| DataFrame, Series, ChunkedArray | rust/core_concepts.md | Rust API fundamentals |
| Expression API in Rust | rust/operations.md | Operations syntax |
| Readers, writers, streaming | rust/io_guide.md | I/O operations |
| Feature flags, crates | rust/features.md | Cargo setup |
| Allocators, SIMD, nightly | rust/performance.md | Performance tuning |
| Zero-copy, FFI, Arrow | rust/arrow_interop.md | Arrow integration |
Shared References
| Topic | File | When to Load |
|---|---|---|
| SQL queries on DataFrames | sql_interface.md | SQL syntax needed |
| Query optimization, streaming | lazy_deep_dive.md | Understanding lazy engine |
| NVIDIA GPU acceleration | gpu_support.md | GPU setup/usage |
Migration Guides
| From | File | When to Load |
|---|---|---|
| pandas | migration_pandas.md | Converting pandas code |
| PySpark | migration_spark.md | Converting Spark code |
| q/kdb+ | migration_qkdb.md | Converting kdb code |
Time Series / Financial Data Quick Patterns
# OHLCV resampling
df.group_by_dynamic("timestamp", every="1m").agg(
pl.col("price").first().alias("open"),
pl.col("price").max().alias("high"),
pl.col("price").min().alias("low"),
pl.col("price").last().alias("close"),
pl.col("volume").sum()
)
# Rolling statistics
df.with_columns(
pl.col("price").rolling_mean(window_size=20).alias("sma_20"),
pl.col("price").rolling_std(window_size=20).alias("volatility")
)
# As-of join for market data alignment
trades.join_asof(quotes, on="timestamp", by="symbol", strategy="backward")
Load python/best_practices.md for comprehensive time series patterns.
Runnable Examples
| Example | File | Purpose |
|---|---|---|
| Financial OHLCV | examples/financial_ohlcv.py | OHLCV resampling, rolling stats, VWAP |
| Pandas Migration | examples/pandas_migration.py | Side-by-side pandas vs polars |
| Streaming Large Files | examples/streaming_large_file.py | Out-of-memory processing patterns |
Development Tips
Use LSP for navigating Polars code:
- Python: Pyright/Pylance provides excellent type inference for Polars expressions
- Rust: rust-analyzer understands Polars types and expression chains
LSP operations like goToDefinition and hover help explore Polars API without leaving the editor.
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
arxiv-search
Search arXiv preprint repository for research papers in physics, mathematics, computer science, quantitative biology, finance, and statistics. Use when finding academic papers, preprints, ML research, scientific publications. Triggers: arxiv, preprint, research paper, academic paper, scientific literature.
mft-research-experts
Run research orchestration for data quality, factor geometry, hypothesis validation, and incident forensics. Use when you need SHIP/KILL/ITERATE decisions with strict validation. Triggers: mft-strategist, data-sentinel, factor-geometer, skeptic, forensic-auditor, research pipeline, hypothesis validation, post-mortem.
datetime
Get current date and time in various formats. Use whenever you need the current date, time, timestamps, or formatted datetime values for any purpose (logging, file naming, scheduling, comparisons, etc.)
97-dev
Apply timeless programming wisdom from "97 Things Every Programmer Should Know" when writing, reviewing, or refactoring code. Use for design decisions, code quality checks, professional development guidance, testing strategies, and workflow optimization.
dev-experts
Apply opinionated developer personas for architecture decisions, production debugging, language-specific code review, comprehensive reviewer passes, and test strategy. Use when you need an architect plan, devops investigation, Rust/Python/C++ review, grumpy reviewer audit, or tester-driven test plan. Triggers: architect, devops, rust-dev, python-dev, cpp-dev, reviewer, tester, pre-merge review, refactor for maintainability.
venue-expert
This skill should be used when the user asks about "market microstructure", "exchange mechanics", "order book", "auction", "NBBO", "Reg NMS", "trading venue", "halt", "LULD", "tick size", "maker-taker", "price-time priority", "SIP", "direct feed", "TRF", "wholesaler", "PFOF", "best execution", "trade-through", "ISO", "opening cross", "closing cross", "NOII", "ITCH", "OUCH", or mentions specific exchanges (Nasdaq, NYSE, CME, Binance, SHFE, DCE, CZCE, CFFEX, INE, etc.). For Chinese futures: "CTP", "综合交易平台", "夜盘", "night session", "看穿式监管", "position limits", "持仓限额", queue position in Chinese markets, or Chinese product codes (rb, cu, sc, if, ic, i, j, ta, ma, etc.). Provides hierarchical venue expertise for research and debugging trading systems.
Didn't find tool you were looking for?