Agent skill

data-processor

Process and transform arrays of data with common operations like filtering, mapping, and aggregation

View SKILL.md on GitHub Repository
Stars 232
Forks 15

Install this agent skill to your Project

npx add-skill https://github.com/aiskillstore/marketplace/tree/main/skills/artemisai/data-processor

SKILL.md

Data Processor Skill

A general-purpose data processing skill for transforming arrays of objects. This skill demonstrates the token efficiency benefits of code execution - instead of describing transformations in natural language, write code once and reuse it.

What This Skill Does

Processes arrays of data with common transformations:

  • Filter records based on conditions
  • Map fields to new values
  • Aggregate data (sum, average, count, etc.)
  • Sort and group data
  • Remove duplicates
  • Merge datasets

When to Use This Skill

Use this skill when you need to:

  • Transform large datasets (hundreds or thousands of records)
  • Apply consistent business logic to data
  • Aggregate or summarize data
  • Clean or normalize data
  • Combine data from multiple sources

Token Efficiency: Processing 1000 records in code uses ~500 tokens. Describing the same operations in natural language would use ~50,000 tokens.

Implementation

javascript
/**
 * Data Processor - General purpose data transformation
 * @param {Array} data - Array of objects to process
 * @param {Object} operations - Operations to apply
 * @returns {Object} Processed data and statistics
 */
async function processData(data, operations = {}) {
  if (!Array.isArray(data)) {
    throw new Error('Data must be an array');
  }
  
  let result = [...data];
  const stats = {
    inputCount: data.length,
    operations: [],
  };
  
  // Filter operation
  if (operations.filter) {
    const beforeCount = result.length;
    result = result.filter(operations.filter);
    stats.operations.push({
      type: 'filter',
      recordsRemoved: beforeCount - result.length
    });
  }
  
  // Map operation (transform fields)
  if (operations.map) {
    result = result.map(operations.map);
    stats.operations.push({ type: 'map' });
  }
  
  // Sort operation
  if (operations.sort) {
    const { field, order = 'asc' } = operations.sort;
    result.sort((a, b) => {
      const aVal = a[field];
      const bVal = b[field];
      const comparison = aVal < bVal ? -1 : aVal > bVal ? 1 : 0;
      return order === 'asc' ? comparison : -comparison;
    });
    stats.operations.push({ type: 'sort', field, order });
  }
  
  // Aggregate operation
  if (operations.aggregate) {
    const { field, operation: aggOp } = operations.aggregate;
    const values = result.map(r => r[field]).filter(v => v != null);
    
    let aggregateResult;
    switch (aggOp) {
      case 'sum':
        aggregateResult = values.reduce((sum, v) => sum + v, 0);
        break;
      case 'average':
        aggregateResult = values.reduce((sum, v) => sum + v, 0) / values.length;
        break;
      case 'count':
        aggregateResult = values.length;
        break;
      case 'min':
        aggregateResult = Math.min(...values);
        break;
      case 'max':
        aggregateResult = Math.max(...values);
        break;
      default:
        throw new Error(`Unknown aggregate operation: ${aggOp}`);
    }
    
    stats.aggregateResult = {
      field,
      operation: aggOp,
      value: aggregateResult
    };
  }
  
  // Remove duplicates
  if (operations.unique) {
    const { field } = operations.unique;
    const seen = new Set();
    const beforeCount = result.length;
    result = result.filter(item => {
      const key = item[field];
      if (seen.has(key)) return false;
      seen.add(key);
      return true;
    });
    stats.operations.push({
      type: 'unique',
      field,
      duplicatesRemoved: beforeCount - result.length
    });
  }
  
  stats.outputCount = result.length;
  
  return {
    data: result,
    stats
  };
}

module.exports = processData;

Examples

Example 1: Filter and Sort

javascript
const processData = require('/skills/data-processor.js');

const salesData = [
  { id: 1, amount: 150, status: 'completed' },
  { id: 2, amount: 200, status: 'pending' },
  { id: 3, amount: 175, status: 'completed' },
  { id: 4, amount: 225, status: 'completed' }
];

const result = await processData(salesData, {
  filter: (record) => record.status === 'completed',
  sort: { field: 'amount', order: 'desc' }
});

console.log(result);
// Output:
// {
//   data: [
//     { id: 4, amount: 225, status: 'completed' },
//     { id: 3, amount: 175, status: 'completed' },
//     { id: 1, amount: 150, status: 'completed' }
//   ],
//   stats: {
//     inputCount: 4,
//     operations: [
//       { type: 'filter', recordsRemoved: 1 },
//       { type: 'sort', field: 'amount', order: 'desc' }
//     ],
//     outputCount: 3
//   }
// }

Example 2: Aggregate Data

javascript
const processData = require('/skills/data-processor.js');

const orders = [
  { orderId: 1, total: 100 },
  { orderId: 2, total: 150 },
  { orderId: 3, total: 200 }
];

const result = await processData(orders, {
  aggregate: { field: 'total', operation: 'sum' }
});

console.log(result.stats.aggregateResult);
// Output: { field: 'total', operation: 'sum', value: 450 }

Example 3: Complex Transformation

javascript
const processData = require('/skills/data-processor.js');

const customers = [
  { name: '  John Doe  ', email: 'JOHN@EXAMPLE.COM', age: 30 },
  { name: 'Jane Smith', email: 'jane@example.com', age: 25 },
  { name: '  John Doe  ', email: 'JOHN@EXAMPLE.COM', age: 30 } // duplicate
];

const result = await processData(customers, {
  map: (customer) => ({
    name: customer.name.trim(),
    email: customer.email.toLowerCase(),
    age: customer.age
  }),
  unique: { field: 'email' },
  filter: (customer) => customer.age >= 25,
  sort: { field: 'age', order: 'asc' }
});

console.log(result.data);
// Output:
// [
//   { name: 'Jane Smith', email: 'jane@example.com', age: 25 },
//   { name: 'John Doe', email: 'john@example.com', age: 30 }
// ]

Integration with MCP Tools

This skill works great in combination with MCP tools:

javascript
// Fetch data from an MCP tool
const rawData = await callMCPTool('database__query', {
  query: 'SELECT * FROM customers WHERE created_date > "2024-01-01"'
});

// Process with the skill
const processData = require('/skills/data-processor.js');
const result = await processData(rawData, {
  filter: (r) => r.status === 'active',
  sort: { field: 'revenue', order: 'desc' },
  aggregate: { field: 'revenue', operation: 'sum' }
});

// Save results
await callMCPTool('storage__save', {
  key: 'processed_customers',
  value: result.data
});

// Return summary to agent (not full data)
return {
  processedRecords: result.stats.outputCount,
  totalRevenue: result.stats.aggregateResult.value
};

Tips and Best Practices

  1. Save Intermediate Results: For large datasets, save to /workspace after each major operation
  2. Return Summaries: Send statistics to the agent, not full datasets
  3. Chain Operations: Combine multiple operations for complex transformations
  4. Validate Input: Always check data types and handle edge cases
  5. Reuse This Skill: Save to /skills and use across multiple tasks

Related Skills

  • validator - Validate data before processing
  • exporter - Export processed data to various formats
  • aggregator - Advanced statistical aggregations

Performance Notes

This skill can process:

  • 1,000 records: < 50ms
  • 10,000 records: < 200ms
  • 100,000 records: < 2s

All operations use efficient JavaScript array methods with O(n) or O(n log n) complexity.

Inspired by: The Anthropic skills pattern for token-efficient data processing. See Code Execution with MCP for the philosophy behind this approach.

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

aiskillstore/marketplace

perigon-backend

Perigon ASP.NET Core + EF Core + Aspire conventions

232 15
Explore
aiskillstore/marketplace

perigon-agent

Pointers for Copilot/agents to apply Perigon conventions

232 15
Explore
aiskillstore/marketplace

perigon-angular

Angular 21+ standalone/Material/signal conventions for Perigon WebApp

232 15
Explore
aiskillstore/marketplace

fastapi-mastery

Comprehensive FastAPI development skill covering REST API creation, routing, request/response handling, validation, authentication, database integration, middleware, and deployment. Use when working with FastAPI projects, building APIs, implementing CRUD operations, setting up authentication/authorization, integrating databases (SQL/NoSQL), adding middleware, handling WebSockets, or deploying FastAPI applications. Triggered by requests involving .py files with FastAPI code, API endpoint creation, Pydantic models, or FastAPI-specific features.

232 15
Explore
aiskillstore/marketplace

context7-efficient

Token-efficient library documentation fetcher using Context7 MCP with 86.8% token savings through intelligent shell pipeline filtering. Fetches code examples, API references, and best practices for JavaScript, Python, Go, Rust, and other libraries. Use when users ask about library documentation, need code examples, want API usage patterns, are learning a new framework, need syntax reference, or troubleshooting with library-specific information. Triggers include questions like "Show me React hooks", "How do I use Prisma", "What's the Next.js routing syntax", or any request for library/framework documentation.

232 15
Explore
aiskillstore/marketplace

browser-use

Browser automation using Playwright MCP. Navigate websites, fill forms, click elements, take screenshots, and extract data. Use when tasks require web browsing, form submission, web scraping, UI testing, or any browser interaction.

232 15
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results