Agent skill

memory-optimization

Optimize Python code for reduced memory usage and improved memory efficiency. Use when asked to reduce memory footprint, fix memory leaks, optimize data structures for memory, handle large datasets efficiently, or diagnose memory issues. Covers object sizing, generator patterns, efficient data structures, and memory profiling strategies.

View SKILL.md on GitHub Repository
Stars 897
Forks 232

Install this agent skill to your Project

npx add-skill https://github.com/benchflow-ai/skillsbench/tree/main/tasks/parallel-tfidf-search/environment/skills/memory-optimization

SKILL.md

Memory Optimization Skill

Transform Python code to minimize memory usage while maintaining functionality.

Workflow

  1. Profile to identify memory bottlenecks (largest allocations, leak patterns)
  2. Analyze data structures and object lifecycles
  3. Select optimization strategies based on access patterns
  4. Transform code with memory-efficient alternatives
  5. Verify memory reduction without correctness loss

Memory Optimization Decision Tree

What's consuming memory?

Large collections:
├── List of objects → __slots__, namedtuple, or dataclass(slots=True)
├── List built all at once → Generator/iterator pattern
├── Storing strings → String interning, categorical encoding
└── Numeric data → NumPy arrays instead of lists

Data processing:
├── Loading full file → Chunked reading, memory-mapped files
├── Intermediate copies → In-place operations, views
├── Keeping processed data → Process-and-discard pattern
└── DataFrame operations → Downcast dtypes, sparse arrays

Object lifecycle:
├── Objects never freed → Check circular refs, use weakref
├── Cache growing unbounded → LRU cache with maxsize
├── Global accumulation → Explicit cleanup, context managers
└── Large temporary objects → Delete explicitly, gc.collect()

Transformation Patterns

Pattern 1: Class to slots

Reduces per-instance memory by 40-60%:

Before:

python
class Point:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

After:

python
class Point:
    __slots__ = ('x', 'y', 'z')

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

Pattern 2: List to Generator

Avoid materializing entire sequences:

Before:

python
def get_all_records(files):
    records = []
    for f in files:
        records.extend(parse_file(f))
    return records

all_data = get_all_records(files)
for record in all_data:
    process(record)

After:

python
def get_all_records(files):
    for f in files:
        yield from parse_file(f)

for record in get_all_records(files):
    process(record)

Pattern 3: Downcast Numeric Types

Reduce NumPy/Pandas memory by 2-8x:

Before:

python
df = pd.read_csv('data.csv')  # Default int64, float64

After:

python
def optimize_dtypes(df):
    for col in df.select_dtypes(include=['int']):
        df[col] = pd.to_numeric(df[col], downcast='integer')
    for col in df.select_dtypes(include=['float']):
        df[col] = pd.to_numeric(df[col], downcast='float')
    return df

df = optimize_dtypes(pd.read_csv('data.csv'))

Pattern 4: String Deduplication

For repeated strings:

Before:

python
records = [{'status': 'active', 'type': 'user'} for _ in range(1000000)]

After:

python
import sys

STATUS_ACTIVE = sys.intern('active')
TYPE_USER = sys.intern('user')

records = [{'status': STATUS_ACTIVE, 'type': TYPE_USER} for _ in range(1000000)]

Or with Pandas:

python
df['status'] = df['status'].astype('category')

Pattern 5: Memory-Mapped File Processing

Process files larger than RAM:

python
import mmap
import numpy as np

# For binary data
with open('large_file.bin', 'rb') as f:
    mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    # Process chunks without loading entire file

# For NumPy arrays
arr = np.memmap('large_array.dat', dtype='float32', mode='r', shape=(1000000, 100))

Pattern 6: Chunked DataFrame Processing

python
def process_large_csv(filepath, chunksize=10000):
    results = []
    for chunk in pd.read_csv(filepath, chunksize=chunksize):
        result = process_chunk(chunk)
        results.append(result)
        del chunk  # Explicit cleanup
    return pd.concat(results)

Data Structure Memory Comparison

Structure Memory per item Use case
list of dict ~400+ bytes Flexible, small datasets
list of class ~300 bytes Object-oriented, small
list of __slots__ class ~120 bytes Many similar objects
namedtuple ~80 bytes Immutable records
numpy.ndarray 8 bytes (float64) Numeric, vectorized ops
pandas.DataFrame ~10-50 bytes/cell Tabular, analysis

Memory Leak Detection

Common leak patterns and fixes:

Pattern Cause Fix
Growing cache No eviction policy @lru_cache(maxsize=1000)
Event listeners Not unregistered Weak references or explicit removal
Circular references Objects reference each other weakref, break cycles
Global lists Append without cleanup Bounded deque, periodic clear
Closures Capture large objects Capture only needed values

Profiling Commands

python
# Object size
import sys
sys.getsizeof(obj)  # Shallow size only

# Deep size with pympler
from pympler import asizeof
asizeof.asizeof(obj)  # Includes referenced objects

# Memory profiler decorator
from memory_profiler import profile
@profile
def my_function():
    pass

# Tracemalloc for allocation tracking
import tracemalloc
tracemalloc.start()
# ... code ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

Verification Checklist

Before finalizing optimized code:

  • Memory usage reduced (measure with profiler)
  • Functionality preserved (same outputs)
  • No new memory leaks introduced
  • Performance acceptable (generators may add iteration overhead)
  • Code remains readable and maintainable

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

benchflow-ai/skillsbench

csv-processing

Use this skill when reading sensor data from CSV files, writing simulation results to CSV, processing time-series data with pandas, or handling missing values in datasets.

897 232
Explore
benchflow-ai/skillsbench

pid-controller

Use this skill when implementing PID control loops for adaptive cruise control, vehicle speed regulation, throttle/brake management, or any feedback control system requiring proportional-integral-derivative control.

897 232
Explore
benchflow-ai/skillsbench

yaml-config

Use this skill when reading or writing YAML configuration files, loading vehicle parameters, or handling config file parsing with proper error handling.

897 232
Explore
benchflow-ai/skillsbench

simulation-metrics

Use this skill when calculating control system performance metrics such as rise time, overshoot percentage, steady-state error, or settling time for evaluating simulation results.

897 232
Explore
benchflow-ai/skillsbench

vehicle-dynamics

Use this skill when simulating vehicle motion, calculating safe following distances, time-to-collision, speed/position updates, or implementing vehicle state machines for cruise control modes.

897 232
Explore
benchflow-ai/skillsbench

web-interface-guidelines

Vercel's comprehensive UI guidelines for building accessible, performant web interfaces. Use this skill when reviewing or building UI components for compliance with best practices around accessibility, performance, animations, and visual stability.

897 232
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results