Agent skill

data_cleaning

Clean messy tabular datasets with deduplication, missing value imputation, outlier handling, and text processing. Use when dealing with dirty data that has duplicates, nulls, or inconsistent formatting.

Stars 897
Forks 232

Install this agent skill to your Project

npx add-skill https://github.com/benchflow-ai/skillsbench/tree/main/tasks/trend-anomaly-causal-inference/environment/skills/data_cleaning

SKILL.md

Data Cleaning Framework

Comprehensive, modular data cleaning framework for tabular datasets. Provides strategy-based cleaning operations including deduplication, missing value imputation, text processing, and outlier handling through a configurable pipeline.

Core Components

CleaningStrategies

Collection of static methods for specific cleaning operations:

Deduplication

  • remove_duplicates(df, subset=None) - Remove duplicate rows given the context of the prompt or the context of the dataset, Carefully evaluate if it still makes sense to allow duplicates exist in the downstream analysis. You should always check the data and remove them based on the logic of meta data and downstream goal.

Missing Value Handling

Logic: If the column is not critical to affect the dowstream task and the cost of dropping the whole line is not neccessary, you should try to impute them, otherwise you can drop.

  • drop_missing(df, columns) - Drop rows with missing critical values (that will affect the downstream task's peformance) in specified columns.
  • impute_median(df, columns) - Impute with median for numerical columns if the column is numerical data
  • impute_knn(df, target_features, n_neighbors=5) - KNN-based imputation such that the target column value potentially can be inferred by other columns. I.e. you have to carefully see the semantic and logical relationship between columns in terms of real life cases and make decisions to use this method for better data imputation performance.
  • impute_mode(df, columns) - Impute with mode for categorical columns if the target column is relatively independent to other columns and should just be filled with an acceptable value.

Text Processing

  • process_text(df, columns, operation) - Text operations: some columns might need conduct text related processing to get numbers or remove unnecessary content, you need to dertermine if this is needed based on preview the column values regarding the down stream goal. (extract_numbers, clean_whitespace, extract_email, lowercase, remove_special)

Outlier Handling

Logic: Again you need to make sure if removing outliers is costly or not in terms of the sample size or critical columns under your investigation. Keep the rows by replacing the outlier value with a more reasonable value, or simply remove the whole row in one of the two ways if it barely has any impact.

  • cap_outliers_iqr(df, columns, multiplier=1.5) - Winsorization using IQR to keep reasonable signals of the data;
  • remove_outliers_iqr(df, columns, multiplier=1.5) - Remove outliers using IQR
  • remove_outliers_zscore(df, columns, threshold=3.0) - Remove outliers using Z-score

DataCleaningPipeline

Orchestrates multiple cleaning steps with logging and progress tracking.

Usage

python
from data_cleaning import CleaningStrategies, DataCleaningPipeline

# Create pipeline
pipeline = DataCleaningPipeline(name="Transaction Data")

# Add cleaning steps
pipeline.add_step(
    CleaningStrategies.remove_duplicates,
    subset=['customer_id', 'order_date'],
    description="Remove duplicate orders"
).add_step(
    CleaningStrategies.drop_missing,
    columns=['customer_id', 'amount'],
    description="Drop rows with missing critical fields"
).add_step(
    CleaningStrategies.impute_knn,
    target_features={
        'income': {'features': ['age', 'education'], 'type': 'numeric'}
    },
    description="KNN imputation for income"
).add_step(
    CleaningStrategies.cap_outliers_iqr,
    columns=['amount'],
    multiplier=1.5,
    description="Cap price outliers"
)

# Execute pipeline
df_clean = pipeline.execute(raw_df, verbose=True)

# Get execution log
log_df = pipeline.get_log()

Input

  • Any tabular DataFrame requiring cleaning

Output

  • Cleaned DataFrame with execution log available

Best Practices

  • Chain multiple steps using method chaining
  • Use verbose=True to monitor pipeline progress
  • Review execution log for quality assurance
  • Extend CleaningStrategies with domain-specific methods as needed

Expand your agent's capabilities with these related and highly-rated skills.

benchflow-ai/skillsbench

csv-processing

Use this skill when reading sensor data from CSV files, writing simulation results to CSV, processing time-series data with pandas, or handling missing values in datasets.

897 232
Explore
benchflow-ai/skillsbench

pid-controller

Use this skill when implementing PID control loops for adaptive cruise control, vehicle speed regulation, throttle/brake management, or any feedback control system requiring proportional-integral-derivative control.

897 232
Explore
benchflow-ai/skillsbench

yaml-config

Use this skill when reading or writing YAML configuration files, loading vehicle parameters, or handling config file parsing with proper error handling.

897 232
Explore
benchflow-ai/skillsbench

simulation-metrics

Use this skill when calculating control system performance metrics such as rise time, overshoot percentage, steady-state error, or settling time for evaluating simulation results.

897 232
Explore
benchflow-ai/skillsbench

vehicle-dynamics

Use this skill when simulating vehicle motion, calculating safe following distances, time-to-collision, speed/position updates, or implementing vehicle state machines for cruise control modes.

897 232
Explore
benchflow-ai/skillsbench

web-interface-guidelines

Vercel's comprehensive UI guidelines for building accessible, performant web interfaces. Use this skill when reviewing or building UI components for compliance with best practices around accessibility, performance, animations, and visual stability.

897 232
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results