Agent skill
data_cleaning
Clean messy tabular datasets with deduplication, missing value imputation, outlier handling, and text processing. Use when dealing with dirty data that has duplicates, nulls, or inconsistent formatting.
Install this agent skill to your Project
npx add-skill https://github.com/benchflow-ai/skillsbench/tree/main/tasks/trend-anomaly-causal-inference/environment/skills/data_cleaning
SKILL.md
Data Cleaning Framework
Comprehensive, modular data cleaning framework for tabular datasets. Provides strategy-based cleaning operations including deduplication, missing value imputation, text processing, and outlier handling through a configurable pipeline.
Core Components
CleaningStrategies
Collection of static methods for specific cleaning operations:
Deduplication
remove_duplicates(df, subset=None)- Remove duplicate rows given the context of the prompt or the context of the dataset, Carefully evaluate if it still makes sense to allow duplicates exist in the downstream analysis. You should always check the data and remove them based on the logic of meta data and downstream goal.
Missing Value Handling
Logic: If the column is not critical to affect the dowstream task and the cost of dropping the whole line is not neccessary, you should try to impute them, otherwise you can drop.
drop_missing(df, columns)- Drop rows with missing critical values (that will affect the downstream task's peformance) in specified columns.impute_median(df, columns)- Impute with median for numerical columns if the column is numerical dataimpute_knn(df, target_features, n_neighbors=5)- KNN-based imputation such that the target column value potentially can be inferred by other columns. I.e. you have to carefully see the semantic and logical relationship between columns in terms of real life cases and make decisions to use this method for better data imputation performance.impute_mode(df, columns)- Impute with mode for categorical columns if the target column is relatively independent to other columns and should just be filled with an acceptable value.
Text Processing
process_text(df, columns, operation)- Text operations: some columns might need conduct text related processing to get numbers or remove unnecessary content, you need to dertermine if this is needed based on preview the column values regarding the down stream goal. (extract_numbers, clean_whitespace, extract_email, lowercase, remove_special)
Outlier Handling
Logic: Again you need to make sure if removing outliers is costly or not in terms of the sample size or critical columns under your investigation. Keep the rows by replacing the outlier value with a more reasonable value, or simply remove the whole row in one of the two ways if it barely has any impact.
cap_outliers_iqr(df, columns, multiplier=1.5)- Winsorization using IQR to keep reasonable signals of the data;remove_outliers_iqr(df, columns, multiplier=1.5)- Remove outliers using IQRremove_outliers_zscore(df, columns, threshold=3.0)- Remove outliers using Z-score
DataCleaningPipeline
Orchestrates multiple cleaning steps with logging and progress tracking.
Usage
from data_cleaning import CleaningStrategies, DataCleaningPipeline
# Create pipeline
pipeline = DataCleaningPipeline(name="Transaction Data")
# Add cleaning steps
pipeline.add_step(
CleaningStrategies.remove_duplicates,
subset=['customer_id', 'order_date'],
description="Remove duplicate orders"
).add_step(
CleaningStrategies.drop_missing,
columns=['customer_id', 'amount'],
description="Drop rows with missing critical fields"
).add_step(
CleaningStrategies.impute_knn,
target_features={
'income': {'features': ['age', 'education'], 'type': 'numeric'}
},
description="KNN imputation for income"
).add_step(
CleaningStrategies.cap_outliers_iqr,
columns=['amount'],
multiplier=1.5,
description="Cap price outliers"
)
# Execute pipeline
df_clean = pipeline.execute(raw_df, verbose=True)
# Get execution log
log_df = pipeline.get_log()
Input
- Any tabular DataFrame requiring cleaning
Output
- Cleaned DataFrame with execution log available
Best Practices
- Chain multiple steps using method chaining
- Use verbose=True to monitor pipeline progress
- Review execution log for quality assurance
- Extend CleaningStrategies with domain-specific methods as needed
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
csv-processing
Use this skill when reading sensor data from CSV files, writing simulation results to CSV, processing time-series data with pandas, or handling missing values in datasets.
pid-controller
Use this skill when implementing PID control loops for adaptive cruise control, vehicle speed regulation, throttle/brake management, or any feedback control system requiring proportional-integral-derivative control.
yaml-config
Use this skill when reading or writing YAML configuration files, loading vehicle parameters, or handling config file parsing with proper error handling.
simulation-metrics
Use this skill when calculating control system performance metrics such as rise time, overshoot percentage, steady-state error, or settling time for evaluating simulation results.
vehicle-dynamics
Use this skill when simulating vehicle motion, calculating safe following distances, time-to-collision, speed/position updates, or implementing vehicle state machines for cruise control modes.
web-interface-guidelines
Vercel's comprehensive UI guidelines for building accessible, performant web interfaces. Use this skill when reviewing or building UI components for compliance with best practices around accessibility, performance, animations, and visual stability.
Didn't find tool you were looking for?