Agent skill
pandas-data-processing-1-memory-efficiency
Sub-skill of pandas-data-processing: 1. Memory Efficiency (+3).
Install this agent skill to your Project
npx add-skill https://github.com/vamseeachanta/workspace-hub/tree/main/.claude/skills/_archive/data/scientific/pandas-data-processing/1-memory-efficiency
SKILL.md
1. Memory Efficiency (+3)
1. Memory Efficiency
Use appropriate data types:
# ❌ Bad: Default float64
df = pd.DataFrame({'value': np.random.randn(1000000)})
print(f"Memory: {df.memory_usage(deep=True).sum() / 1e6:.1f} MB")
# ✅ Good: Use float32 when precision allows
df_optimized = pd.DataFrame({'value': np.random.randn(1000000).astype(np.float32)})
print(f"Memory: {df_optimized.memory_usage(deep=True).sum() / 1e6:.1f} MB") # 50% reduction
# ✅ Use categorical for repeated strings
df['category'] = pd.Categorical(['A', 'B', 'C'] * 100000)
Chunking for large files:
def process_large_csv_in_chunks(
csv_file: Path,
chunksize: int = 100_000
) -> pd.DataFrame:
"""Process large CSV in chunks to avoid memory issues."""
chunks = []
for chunk in pd.read_csv(csv_file, chunksize=chunksize):
# Process each chunk
chunk_processed = chunk[chunk['Value'] > 0] # Example filter
chunks.append(chunk_processed)
# Combine all chunks
result = pd.concat(chunks, ignore_index=True)
return result
2. Vectorization
Always prefer vectorized operations:
# ❌ Bad: Loop
df['result'] = 0
for i in range(len(df)):
df.loc[i, 'result'] = df.loc[i, 'a'] + df.loc[i, 'b']
# ✅ Good: Vectorized
df['result'] = df['a'] + df['b']
# ✅ Better: NumPy for complex operations
df['result'] = np.where(
df['a'] > 0,
df['a'] + df['b'],
df['a'] - df['b']
)
3. Index Usage
Use index for time series:
# ✅ Set datetime index
df['Time'] = pd.to_datetime(df['Time'])
df.set_index('Time', inplace=True)
# Fast slicing
subset = df['2025-01-01':'2025-01-31']
# Fast resampling
daily_mean = df.resample('D').mean()
4. Data Validation
Validate data before processing:
def validate_engineering_data(df: pd.DataFrame) -> bool:
"""Validate engineering data integrity."""
# Check for missing values
if df.isnull().any().any():
print("⚠ Warning: Missing values detected")
print(df.isnull().sum())
# Check for duplicates
if df.duplicated().any():
print("⚠ Warning: Duplicate rows detected")
print(f"Duplicates: {df.duplicated().sum()}")
# Check data types
print("Data types:")
print(df.dtypes)
# Check value ranges
numeric_cols = df.select_dtypes(include=[np.number]).columns
for col in numeric_cols:
if (df[col] < 0).any():
print(f"⚠ Warning: Negative values in {col}")
return True
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
vamseeachanta/workspace-hub
gsd-complete-milestone
Archive completed milestone and prepare for next version
vamseeachanta/workspace-hub
gsd-reapply-patches
Reapply local modifications after a GSD update
vamseeachanta/workspace-hub
gsd-verify-work
Validate built features through conversational UAT
vamseeachanta/workspace-hub
gsd-thread
Manage persistent context threads for cross-session work
vamseeachanta/workspace-hub
clinical-trial-protocol
Generate clinical trial protocols for medical devices or drugs through a modular, waypoint-based architecture with research-only and full protocol modes.
vamseeachanta/workspace-hub
single-cell-rna-qc
Performs quality control on single-cell RNA-seq data (.h5ad or .h5 files) using scverse best practices with MAD-based filtering and comprehensive visualizations.
Didn't find tool you were looking for?