Script Kit AI Testing

Two distinct AI interfaces exist. Know which you're testing.

Two AI Interfaces

1. AI Window (Separate Floating Window)

• Full chat app: sidebar with chat history, message bubbles, model dropdown, search, Cmd+K command bar
• Source: src/ai/window/
• Opens via openAi stdin command or "Ask AI" built-in

2. Inline Chat Prompt (ChatPrompt in Main Window)

• Input at TOP, full-width message containers (not bubbles), conversation turns
• Source: src/prompts/chat/
• Opens via Tab key with filter text, or SDK chat() call

Stdin Commands for AI Testing

AI Window Commands

{"type":"openAi"}                                          // Open empty AI window
{"type":"openAiWithMockData"}                              // Open with sample conversations
{"type":"showAiCommandBar"}                                // Open Cmd+K actions menu
{"type":"setAiInput","text":"Hello AI","submit":true}      // Set input text & optionally submit
{"type":"setAiSearch","text":"python"}                      // Filter chat list sidebar
{"type":"simulateAiKey","key":"enter"}                     // Simulate keypress in AI window
{"type":"captureWindow","title":"Script Kit AI","path":".test-screenshots/ai.png"}  // Screenshot

Inline Chat Prompt Commands

{"type":"run","path":"/abs/path/to/chat-test.ts"}          // Run a chat() test script

Quick Start: AI Window Visual Test

cargo build && echo '{"type":"openAiWithMockData"}' | SCRIPT_KIT_AI_LOG=1 ./target/debug/script-kit-gpui 2>&1

Then capture and verify:

# In a separate terminal or via named pipe
echo '{"type":"captureWindow","title":"Script Kit AI","path":".test-screenshots/ai-window.png"}'

Always read the PNG file after capture to verify visually.

Quick Start: Inline Chat Prompt Test

// tests/smoke/test-chat-inline.ts
import '../../scripts/kit-sdk';

function log(test: string, status: string, extra: any = {}) {
  console.log(JSON.stringify({ test, status, timestamp: new Date().toISOString(), ...extra }));
}

const name = "chat-inline";
log(name, "running");
const start = Date.now();
try {
  await chat({
    placeholder: "Ask anything...",
    messages: [{ role: 'user', content: 'Hello' }],
    useBuiltinAi: true,
  });
  log(name, "pass", { duration_ms: Date.now() - start });
} catch (e) {
  log(name, "fail", { error: String(e), duration_ms: Date.now() - start });
}
process.exit(0);

Run:

cargo build && echo '{"type":"run","path":"'"$(pwd)"'/tests/smoke/test-chat-inline.ts"}' | \
  SCRIPT_KIT_AI_LOG=1 ./target/debug/script-kit-gpui 2>&1

Mock Data Mode

openAiWithMockData inserts sample conversations into the AI window database so you can visually test the sidebar, chat list, and message rendering without needing real API keys.

Use this for:

• Sidebar layout iteration
• Chat history rendering
• Search/filter behavior
• Visual regression checks

API Key Configuration for Testing

Set environment variables to enable providers:

SCRIPT_KIT_ANTHROPIC_API_KEY=sk-...   # Anthropic (default provider)
SCRIPT_KIT_OPENAI_API_KEY=sk-...      # OpenAI
SCRIPT_KIT_CLAUDE_CODE_ENABLED=1       # Enable Claude Code provider

Without API keys, the inline chat shows a setup card instead of the chat input. Test this state with:

unset SCRIPT_KIT_ANTHROPIC_API_KEY SCRIPT_KIT_OPENAI_API_KEY
cargo build && echo '{"type":"run","path":"'"$(pwd)"'/tests/smoke/test-ai-setup-card.ts"}' | \
  SCRIPT_KIT_AI_LOG=1 ./target/debug/script-kit-gpui 2>&1

Visual Iteration Workflow

1. Make UI change in src/ai/window/ or src/prompts/chat/
2. cargo build
3. Open with mock data: echo '{"type":"openAiWithMockData"}' | SCRIPT_KIT_AI_LOG=1 ./target/debug/script-kit-gpui 2>&1
4. Capture screenshot: {"type":"captureWindow","title":"Script Kit AI","path":".test-screenshots/ai-iteration.png"}
5. Read the PNG to verify
6. Repeat until satisfied

For inline chat, use the test script pattern above with captureScreenshot() from the SDK.

Named Pipe Pattern (Interactive Iteration)

For sending multiple commands to a running instance:

mkfifo /tmp/skpipe
cargo build && cat /tmp/skpipe | SCRIPT_KIT_AI_LOG=1 ./target/debug/script-kit-gpui 2>&1 &

# Send commands one at a time
echo '{"type":"openAiWithMockData"}' > /tmp/skpipe
sleep 1
echo '{"type":"captureWindow","title":"Script Kit AI","path":".test-screenshots/ai-1.png"}' > /tmp/skpipe
sleep 0.5
echo '{"type":"setAiInput","text":"test message"}' > /tmp/skpipe
sleep 0.5
echo '{"type":"captureWindow","title":"Script Kit AI","path":".test-screenshots/ai-2.png"}' > /tmp/skpipe

rm /tmp/skpipe

Protocol Messages Reference

AI Window Protocol (src/protocol/message/variants/ai.rs)

Inline Chat Protocol (src/protocol/message/variants/prompts_media.rs)

Existing Test Files

Key Source Files

Testing Checklist

• AI window opens: {"type":"openAi"}
• AI window with mock data: {"type":"openAiWithMockData"}
• Sidebar shows chat list with dates (Today, This Week)
• Search filters chat list: {"type":"setAiSearch","text":"..."}
• Input field shows placeholder "Ask anything..."
• Input accepts text: {"type":"setAiInput","text":"..."}
• Submit triggers streaming: {"type":"setAiInput","text":"...","submit":true}
• Model dropdown shows available providers
• Command bar opens: {"type":"showAiCommandBar"}
• Inline ChatPrompt via chat({ useBuiltinAi: true })
• Setup card when no API keys configured
• Error messages display correctly (ChatSetError)
• Screenshots capture AI window: {"type":"captureWindow","title":"Script Kit AI",...}

Anti-Patterns

• Running scripts via CLI args (must use stdin JSON)
• Testing AI streaming without SCRIPT_KIT_AI_LOG=1 (log output explodes)
• Forgetting to read the PNG after captureWindow
• Testing with real API keys when mock data suffices (slow, costs money)
• Confusing AI window (src/ai/window/) with inline ChatPrompt (src/prompts/chat/)
• Not setting useBuiltinAi: true when testing built-in AI mode