Agent skill

new-project

Build a new AI agent with Olakai monitoring from scratch — project setup, SDK integration, KPI configuration, and end-to-end validation

Stars 12,831
Forks 1,123

Install this agent skill to your Project

npx add-skill https://github.com/andrewyng/context-hub/tree/main/content/olakai/skills/new-project

Metadata

Additional technical details for this skill

tags
olakai,new-project,agent,monitoring,kpi,governance
source
maintainer
revision
1
updated on
2026-03-10

SKILL.md

Build a New AI Agent Project with Olakai

This skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.

Prerequisites

Before starting, ensure:

  1. Olakai CLI installed: npm install -g olakai-cli
  2. CLI authenticated: olakai login
  3. API key for SDK (generated per-agent via CLI — see Step 2.2)

Why Custom KPIs Are Essential

Olakai's core value is tracking business-specific KPIs for your AI agents. Without KPIs, you're tracking events without gaining actionable insights.

What you can measure with KPIs:

  • Business outcomes (items processed, success rates, revenue impact)
  • Operational data (step counts, retry rates, execution time)
  • Quality indicators (error rates, user satisfaction signals)

Without KPIs configured:

  • No dashboard KPIs beyond basic token counts
  • No aggregated performance views
  • No alerting thresholds
  • No ROI calculations

Every agent should have 2-4 KPIs that answer: "How do I know this agent is performing well?"

KPIs created here belong to this specific agent only. If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.

Understanding the customData to KPI Pipeline

Before diving into implementation, understand how data flows through Olakai:

SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData

How It Works

  1. customData (SDK): Raw JSON you send with each event
  2. CustomDataConfig (Platform): Schema defining which fields are processed
  3. Context Variables: CustomDataConfig fields become available for formulas
  4. KPI Formula: Expression that computes a value (e.g., SuccessRate * 100)
  5. kpiData (Response): Computed KPI values returned with each event

Critical Rules

Rule Consequence
Only CustomDataConfig fields become variables Unregistered customData fields are NOT usable in KPIs
Formula evaluation is case-insensitive stepCount, STEPCOUNT, StepCount all work in formulas
NUMBER configs need numeric values Don't send "5" (string), send 5 (number)
KPIs are unique per agent Each KPI belongs to exactly one agent — create separately for each

Built-in Context Variables (Always Available)

Variable Type Description
Prompt string The prompt text sent to the LLM
Response string The LLM response text
Documents count number Number of attached documents
PII detected boolean Whether PII was detected
PHI detected boolean Whether PHI was detected
CODE detected boolean Whether code was detected
SECRET detected boolean Whether secrets were detected

Step 1: Design the Agent Architecture

1.1 Determine Agent Type

Agentic AI (Multi-step autonomous workflows):

  • Research agents, document processors, data pipelines
  • Track as SINGLE events aggregating all internal LLM calls
  • Focus on workflow-level KPIs (total tokens, total time, success/failure)

Assistive AI (Interactive chatbots/copilots):

  • Customer support agents, coding assistants, Q&A systems
  • Track EACH interaction as separate events
  • Focus on conversation-level KPIs (per-message tokens, response quality)

1.2 Design Your KPI Schema (CRITICAL)

Design your KPIs BEFORE writing any SDK code. This ensures only meaningful data is sent and tracked.

Step A: Identify Business Questions

What do stakeholders need to know about this agent?

  • "How many items does it process per run?"
  • "What's the success/failure rate?"
  • "How efficient is each execution?"

Step B: Map Questions to Data Fields

Business Question Field Name Type KPI Formula Aggregation
Throughput ItemsProcessed NUMBER ItemsProcessed SUM
Reliability SuccessRate NUMBER SuccessRate * 100 AVERAGE
Error count SuccessRate NUMBER IF(SuccessRate < 1, 1, 0) SUM
Correlation ExecutionId STRING (for filtering only) -

Step C: Plan Your customData Structure

typescript
// ONLY include fields you'll register as CustomDataConfigs
customData: {
  // Business KPIs
  ItemsProcessed: number,  // Count of items handled
  SuccessRate: number,     // 0-1 success ratio

  // Performance KPIs
  StepCount: number,       // Number of workflow steps

  // Identification (for filtering, not KPIs)
  ExecutionId: string,     // Correlation ID
}

IMPORTANT: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but cannot be used in KPIs.

What NOT to Include in customData

The Olakai platform automatically tracks these — do NOT duplicate them:

Already Tracked Where Don't Send As customData
Session ID Main payload sessionId
Agent ID API key association agentId
User email userEmail parameter email, userEmail
Timestamp Event metadata timestamp, createdAt
Request time requestTime parameter duration, latency
Token count tokens parameter tokenCount
Model Auto-detected model, modelName
Provider Client config provider

customData is ONLY for:

  1. KPI variables — Fields you'll use in formula calculations
  2. Tagging/filtering — Fields you'll filter by in queries

Step 2: Configure Olakai Platform

2.1 Create a Workflow (Required)

Every agent MUST belong to a workflow, even if it's the only agent in that workflow.

bash
olakai workflows create --name "Your Workflow Name" --json
# Output: { "id": "wfl_xxx...", "name": "Your Workflow Name" }

2.2 Create the Agent in Olakai

bash
olakai agents create \
  --name "Your Agent Name" \
  --description "What this agent does" \
  --workflow WORKFLOW_ID \
  --with-api-key \
  --json

# Returns agent details including apiKey:
# {
#   "id": "cmkbteqn501kyjy4yu6p6xrrx",
#   "name": "Your Agent Name",
#   "workflowId": "wfl_xxx...",
#   "apiKey": "sk_agent_xxxxx..."   <-- Use this in your SDK
# }

Agent-Workflow Hierarchy:

Workflow: "Customer Support Pipeline"
├── Agent: "Ticket Classifier"
├── Agent: "Response Generator"
└── Agent: "Quality Checker"

Workflow: "Document Processing"
└── Agent: "Document Summarizer"  ← single-agent workflows are valid

2.3 Create Custom Data Configurations (BEFORE Writing SDK Code)

This step MUST be completed before Step 3 (SDK Integration). Only fields registered here can be used in KPI formulas.

ONLY create configs for data you'll use in KPIs or for filtering. Don't create configs for data already tracked automatically.

bash
# For numeric fields (can be used in KPI calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ItemsProcessed" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "SuccessRate" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "StepCount" --type NUMBER

# For string fields (for filtering/grouping, not calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ExecutionId" --type STRING

# Verify all configs are created
olakai custom-data list --agent-id YOUR_AGENT_ID

2.4 Create KPI Definitions

Quick Start with Templates

Instead of writing formulas from scratch, use predefined classifier templates:

bash
# List available templates
olakai kpis templates

# Create a classifier KPI from a template
olakai kpis create --name "User Satisfaction" \
  --calculator-id classifier --template-id sentiment_scorer \
  --scope CHAT --agent-id $AGENT_ID

# Create a time-saved estimator
olakai kpis create --name "Time Saved" \
  --calculator-id classifier --template-id time_saved_estimator \
  --scope CHAT --agent-id $AGENT_ID

Custom Formula KPIs

bash
# Variable passthrough
olakai kpis create \
  --name "Items Processed" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "ItemsProcessed" \
  --unit "items" \
  --aggregation SUM

# Percentage calculation
olakai kpis create \
  --name "Success Rate" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "SuccessRate * 100" \
  --unit "%" \
  --aggregation AVERAGE

# Conditional counting
olakai kpis create \
  --name "Error Count" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "IF(SuccessRate < 1, 1, 0)" \
  --unit "errors" \
  --aggregation SUM

# Validate formulas before creating
olakai kpis validate --formula "ItemsProcessed" --agent-id YOUR_AGENT_ID

Step 3: Implement SDK Integration

3.1 TypeScript Implementation

Install dependencies:

bash
npm install @olakai/sdk openai

Initialize and track:

typescript
import { olakaiConfig, olakai } from "@olakai/sdk";
import OpenAI from "openai";

// Initialize Olakai
olakaiConfig({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: process.env.NODE_ENV === "development",
});

// Create LLM client
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// Use wrapped client — monitoring happens automatically
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: userPrompt }],
});

Agentic workflow with manual event tracking:

taskExecutionId — Cross-Agent Task Correlation. Generate ONE taskExecutionId per task and share it across all agents in a multi-agent workflow. This links events from different agents into a single logical task for analytics.

typescript
async function runAgent(input: string): Promise<string> {
  const startTime = Date.now();
  const executionId = crypto.randomUUID();
  const taskExecutionId = crypto.randomUUID();
  let totalTokens = 0;
  let stepCount = 0;
  let itemsProcessed = 0;

  try {
    // Step 1: Planning
    stepCount++;
    const plan = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `Plan: ${input}` }],
    });
    totalTokens += plan.usage?.total_tokens ?? 0;

    // Step 2: Process items
    const items = parseItems(plan.choices[0].message.content);
    for (const item of items) {
      stepCount++;
      const result = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: `Process: ${item}` }],
      });
      totalTokens += result.usage?.total_tokens ?? 0;
      itemsProcessed++;
    }

    // Step 3: Summarize
    stepCount++;
    const summary = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: "Summarize results" }],
    });
    totalTokens += summary.usage?.total_tokens ?? 0;

    const finalResponse = summary.choices[0].message.content ?? "";

    // Track the complete workflow as a single event
    // Only send fields that have CustomDataConfigs (from Step 2.3)
    olakai("event", "ai_activity", {
      prompt: input,
      response: finalResponse,
      tokens: totalTokens,
      requestTime: Date.now() - startTime,
      taskExecutionId,
      task: "Data Processing & Analysis",
      customData: {
        ExecutionId: executionId,
        StepCount: stepCount,
        ItemsProcessed: itemsProcessed,
        SuccessRate: 1.0,
      },
    });

    return finalResponse;
  } catch (error) {
    // Track failed execution — same fields, different values
    olakai("event", "ai_activity", {
      prompt: input,
      response: `Error: ${error instanceof Error ? error.message : "Unknown"}`,
      tokens: totalTokens,
      requestTime: Date.now() - startTime,
      taskExecutionId,
      task: "Data Processing & Analysis",
      customData: {
        ExecutionId: executionId,
        StepCount: stepCount,
        ItemsProcessed: itemsProcessed,
        SuccessRate: 0,
      },
    });
    throw error;
  }
}

3.2 Python Implementation

Install dependencies:

bash
pip install olakai-sdk openai

Initialize and track:

python
import os
from olakaisdk import olakai_config, olakai, OlakaiEventParams
from openai import OpenAI

# Initialize Olakai
olakai_config(os.getenv("OLAKAI_API_KEY"))

# Create OpenAI client
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

Agentic workflow:

python
import time
import uuid

def run_agent(input_text: str) -> str:
    start_time = time.time()
    execution_id = str(uuid.uuid4())
    task_execution_id = str(uuid.uuid4())
    total_tokens = 0
    step_count = 0
    items_processed = 0

    try:
        # Your workflow steps here...
        step_count += 1
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": input_text}]
        )
        total_tokens += response.usage.total_tokens
        final_response = response.choices[0].message.content

        # Track successful execution
        olakai("event", "ai_activity", OlakaiEventParams(
            prompt=input_text,
            response=final_response,
            tokens=total_tokens,
            requestTime=int((time.time() - start_time) * 1000),
            taskExecutionId=task_execution_id,
            task="Data Processing & Analysis",
            customData={
                "ExecutionId": execution_id,
                "StepCount": step_count,
                "ItemsProcessed": items_processed,
                "SuccessRate": 1.0,
            }
        ))

        return final_response

    except Exception as e:
        olakai("event", "ai_activity", OlakaiEventParams(
            prompt=input_text,
            response=f"Error: {str(e)}",
            tokens=total_tokens,
            requestTime=int((time.time() - start_time) * 1000),
            taskExecutionId=task_execution_id,
            task="Data Processing & Analysis",
            customData={
                "ExecutionId": execution_id,
                "StepCount": step_count,
                "ItemsProcessed": items_processed,
                "SuccessRate": 0,
            }
        ))
        raise

3.3 REST API Direct Integration

For other languages or custom integrations:

bash
curl -X POST "https://app.olakai.ai/api/monitoring/prompt" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "prompt": "User input here",
    "response": "Agent response here",
    "app": "your-agent-name",
    "task": "Data Processing & Analysis",
    "tokens": 1500,
    "requestTime": 5000,
    "customData": {
      "ExecutionId": "abc-123",
      "StepCount": 5,
      "ItemsProcessed": 10,
      "SuccessRate": 1.0
    }
  }'

Step 4: Test-Validate-Iterate Cycle

Always validate your implementation by running a test and inspecting the actual event data.

4.1 Run Your Agent

Execute your agent with test data to generate at least one event.

4.2 Fetch and Inspect the Event

bash
olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json
olakai activity get EVENT_ID --json

4.3 Validate Each Component

Check customData is present and correct:

bash
olakai activity get EVENT_ID --json | jq '.customData'

Check KPIs are numeric (not strings):

bash
olakai activity get EVENT_ID --json | jq '.kpiData'

CORRECT — numeric values:

json
{
  "Items Processed": 10,
  "Success Rate": 100
}

WRONG — string values (broken formula):

json
{
  "Items Processed": "itemsProcessed"
}

Fix: olakai kpis update KPI_ID --formula "YourVariable"

WRONG — null values: Fix by verifying:

  1. CustomDataConfig exists: olakai custom-data list --agent-id ID
  2. Field name case matches exactly (case-sensitive)
  3. SDK actually sends the field in customData

4.4 Validation Flow

1. Run agent (generate event)
           ↓
2. Fetch event: olakai activity get ID --json
           ↓
3. Check customData present? NO → Fix SDK code
           ↓
4. Check kpiData numeric? NO → Fix formula
           ↓
5. Check kpiData not null? NO → Create CustomDataConfig or fix field name
           ↓
✅ All validations pass — implementation complete

Step 5: Production Checklist

Before deploying to production:

  • API key stored securely in environment variables
  • Error handling wraps all LLM calls
  • Failed executions still report events (with SuccessRate: 0)
  • All custom data fields have corresponding CustomDataConfig entries
  • KPI formulas validated and showing numeric values (not strings)
  • SDK configured with appropriate retries and timeouts
  • Sensitive data redaction enabled if needed

KPI Formula Reference

Supported Operators

Category Operators
Arithmetic +, -, *, /
Comparison <, <=, =, <>, >=, >
Logical AND, OR, NOT
Conditional IF(condition, true_val, false_val), MAP(value, match1, out1, default)
Math ABS, MAX, MIN, AVERAGE, TRUNC
Null handling ISNA(value), ISDEFINED(value), NA()

Common Formula Patterns

bash
--formula "ItemsProcessed"                              # passthrough
--formula "SuccessRate * 100"                            # percentage (0-1 to 0-100)
--formula "IF(SuccessRate < 1, 1, 0)"                    # conditional counting
--formula "IF(PII detected, 1, 0)"                       # built-in variable
--formula "IF(ISDEFINED(MyField), MyField, 0)"           # null-safe
--formula "IF(AND(StepCount > 5, SuccessRate < 0.9), 1, 0)"  # compound conditions

Aggregation Types

Aggregation Use For Example
SUM Totals, counts Total items processed across all runs
AVERAGE Rates, percentages Average success rate

Task Categories Reference

Use these predefined task categories for the task field:

Category Example Use
Research & Intelligence Competitive intelligence, market research
Data Processing & Analysis Data extraction, statistical analysis
Content Development Blog writing, technical documentation
Content Refinement Editing, proofreading
Customer Experience Complaint resolution, ticket triage
Software Development Code generation, code review, debugging
Strategic Planning Roadmap development, scenario planning

Quick Reference

bash
# CLI Commands
olakai login                                              # Authenticate
olakai workflows create --name "Name" --json              # Create workflow
olakai agents create --name "Name" --workflow ID --with-api-key  # Register agent
olakai custom-data create --agent-id ID --name X --type NUMBER   # Create custom field
olakai kpis create --formula "X" --agent-id ID            # Create KPI
olakai activity list --agent-id ID                        # View events
typescript
// TypeScript SDK
import { olakaiConfig, olakai } from "@olakai/sdk";
olakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });

olakai("event", "ai_activity", {
  prompt: "input",
  response: "output",
  tokens: 1500,
  task: "Data Processing & Analysis",
  customData: { StepCount: 3, Success: 1 },
});
python
# Python SDK
from olakaisdk import olakai_config, olakai, OlakaiEventParams
olakai_config(os.getenv("OLAKAI_API_KEY"))

olakai("event", "ai_activity", OlakaiEventParams(
    prompt="input",
    response="output",
    tokens=1500,
    task="Data Processing & Analysis",
    customData={"StepCount": 3, "Success": 1},
))

Expand your agent's capabilities with these related and highly-rated skills.

andrewyng/context-hub

login-flows

Common login automation patterns for web apps using Playwright

12,831 1,123
Explore
andrewyng/context-hub

document-extraction

Use this skill for intelligent document processing and content extraction using LandingAI's Agentic Document Extraction (ADE). Trigger when users need to (1) Parse documents (PDFs, images, spreadsheets, presentations) into structured Markdown with layout understanding, (2) Extract specific structured data from documents using schemas (invoice fields, form data, table data, etc.), (3) Classify and separate multi-document batches by type (invoices vs receipts, statements vs forms, etc.), (4) Process large documents asynchronously (up to 1GB/1000 pages), (5) Get visual grounding (bounding boxes, page numbers) for extracted content — use when users mention bounding boxes, word locations, grounding, highlighting extracted content, or showing where data appears in a document. Use this skill when the task involves understanding document content for a set of documents. In particular this skill can help you write code that run on sets of documents. This will increase speed, and reduce the cost of loading the documents on the Agent context window because you can use a single script to extract the information needed.

12,831 1,123
Explore
andrewyng/context-hub

document-workflows

Use this skill for building end-to-end document processing workflows and pipelines using LandingAI ADE. Trigger when users need to: (1) Process batches of documents in parallel or async, (2) Build classify-then-extract pipelines for mixed document types, (3) Prepare parsed documents for RAG systems with chunking and vector DB ingestion, (4) Load extraction results into databases like Snowflake or export to CSV/DataFrames, (5) Visualize extraction results: draw bounding box overlays on pages, crop chunk images, or highlight/annotate specific words or phrases found in documents, (6) Build Streamlit or web UIs for document processing, (7) Find and highlight specific terms within document sections using word-level grounding (e.g. highlight "L2S" in the Introduction, redact PII, annotate extracted values on the original page). This skill complements the document-extraction skill which covers ADE SDK basics. Use document-extraction to write code that executes parse/extract/split operations with more precision and less cost than adding the document image to the prompt and asking the LLM to find the relevant info. Use document-workflows when composing those operations into pipelines, or when you need visualization, annotation, or word-level grounding on parsed documents.

12,831 1,123
Explore
andrewyng/context-hub

integrate

Add Olakai monitoring to existing AI code — wrap your LLM client, configure custom KPIs, and validate the integration end-to-end

12,831 1,123
Explore
andrewyng/context-hub

tavily-best-practices

Build production-ready Tavily integrations with best practices for web search, content extraction, crawling, and research in agentic workflows, RAG systems, and autonomous agents

12,831 1,123
Explore
andrewyng/context-hub

electronics-sourcing

Guide for AI agents to source electronic components using parts-mcp — tool sequencing, decision patterns, and multi-step workflows

12,831 1,123
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results