Create a New Source

Prerequisites

Before starting, read these skills:

• understanding-flow - How sources fit in architecture
• understanding-sources - Source interface
• understanding-transformers - Transformer chaining from sources
• understanding-events - Event structure sources emit
• understanding-mapping - Transform raw input to events
• testing-strategy - How to test
• writing-documentation - Documentation standards (for Phase 7)

Source Types

Source Categories

Choose Your Template

Process Overview

1. Research     → Understand input format, find SDK/types
2. Examples     → Create input examples in dev entry FIRST
3. Mapping      → Define input → walkerOS event transformation
4. Scaffold     → Copy template and configure
5. Implement    → Build using examples as test fixtures
6. Test         → Verify against example variations
7. Document     → Write README

Phase 1: Research

Goal: Understand the input format before writing any code.

1.1 Identify Input Source

• What triggers events? - HTTP POST, webhook, DOM mutation, dataLayer push
• What data is received? - Request body, headers, query params
• Authentication? - API keys, signatures, tokens

1.2 Find Official Resources

# Search npm for official types
npm search @[platform]
npm info @types/[platform]

# Check for official SDK
npm search [platform]-sdk

1.3 Document Input Schema

1.4 Check Existing Patterns

# List existing sources
ls packages/web/sources/
ls packages/server/sources/

# Reference implementations
# - dataLayer: DOM-based, array interception
# - express: HTTP middleware
# - fetch: Generic HTTP handler (simplest server pattern)
# - gcp: Cloud Functions specific

Gate: Research Complete

• Input trigger identified (HTTP, webhook, DOM, dataLayer)
• Input schema documented (required/optional fields)
• Fields mapped to walkerOS event structure

Phase 2: Create Input Examples (BEFORE Implementation)

Goal: Define realistic input data in dev entry FIRST.

2.1 Scaffold Directory Structure

mkdir -p packages/server/sources/[name]/src/{examples,schemas,types}

2.2 Create Example Files

2.3 Export via dev.ts

export * as schemas from './schemas';
export * as examples from './examples';

Gate: Examples Valid

• All example files compile (npm run build)
• Can trace: input → expected output for each example
• Edge cases included (minimal input, invalid input)

Phase 3: Define Mapping

Goal: Document transformation from input format to walkerOS events.

Use mapping.ts as your template.

Verify Mapping Logic

Input: inputs.pageViewInput (event: 'page_view')
  ↓ Match mapping rule: page_view.*
  ↓ Apply rule.name: 'page_view' → 'page view'
  ↓ Apply rule.data.map transformations
  ↓ properties.page_title → title
  ↓ properties.page_path → path
Output: Should match outputs.pageViewEvent

Gate: Mapping Verified

• Mapping rules cover main input event types
• Each rule.name transforms to correct walkerOS event name
• Each mapping rule traces correctly to expected output

Phase 4: Scaffold

Template sources:

• Web: packages/web/sources/dataLayer/
• Server: packages/server/sources/fetch/ (simplest pattern)

cp -r packages/server/sources/fetch packages/server/sources/[name]
cd packages/server/sources/[name]

# Update package.json: name, description, repository.directory

Directory structure:

packages/server/sources/[name]/
├── src/
│   ├── index.ts           # Main export
│   ├── index.test.ts      # Tests against examples
│   ├── dev.ts             # Exports schemas and examples
│   ├── examples/
│   ├── schemas/
│   └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md

Transformer Chain Integration

Sources can wire to transformer chains via next in the init config:

sources: {
  mySource: {
    code: sourceMySource,
    config: { settings: { /* ... */ } },
    next: 'validate'  // Events go through validator before collector
  }
}

Phase 5: Implement

Now write code to transform inputs to expected outputs.

Template Files

Key Patterns

1. Context destructuring: Extract config, env, logger, id from context
2. Schema validation: Use Zod schemas to validate settings and provide defaults
3. Forward to collector: Call env.push() to send events to the collector
4. Error logging: Use logger?.error() for errors only, not routine operations
5. Return Source.Instance: Return { type, config, push } object

Gate: Implementation Compiles

• npm run build passes
• npm run lint passes

Phase 6: Test Against Examples

Verify implementation produces expected outputs.

Test Template

Use the test template: index.test.ts

Key Test Patterns

1. Use createSourceContext() helper - Standardizes context creation
2. Mock env.push - Verify events are forwarded to collector
3. Use examples for test data - Don't hardcode test values
4. Test error paths - Verify graceful error handling and logging

Gate: Tests Pass

• npm run test passes
• Tests verify against example outputs (not hardcoded values)
• Invalid input handled gracefully (no crashes)

Phase 7: Document

Follow the writing-documentation skill for:

• README structure and templates
• Example validation against apps/quickstart/
• Quality checklist before publishing

Key requirements for source documentation:

• Input format table documenting expected fields
• Event name mapping table (source format → walkerOS format)
• Configuration options table
• Working code example with imports
• Installation instructions

Validation Checklist

Beyond understanding-development requirements (build, test, lint, no any):

• dev.ts exports schemas and examples
• Examples include edge cases (minimal, invalid input)
• Invalid input returns gracefully (no crashes, clear error)
• Tests use examples for assertions (not hardcoded values)

Reference Files

Related Skills

• understanding-sources - Source interface and push pattern
• understanding-events - Event structure
• testing-strategy - Testing with env mocking
• writing-documentation - Documentation standards