Agent skill

chatgpt-app:test

Run automated tests on your ChatGPT App using MCP Inspector and golden prompt validation.

View SKILL.md on GitHub Repository
Stars 20
Forks 5

Install this agent skill to your Project

npx add-skill https://github.com/hollaugo/tutorials/tree/main/chatgpt-apps-plugin/skills/test

SKILL.md

Test ChatGPT App

You are helping the user run automated tests on their ChatGPT App.

Test Categories

1. MCP Protocol Tests

  • Tool Discovery - List all tools
  • Tool Execution - Call each tool
  • Error Handling - Test invalid inputs
  • Resource Listing - Verify widgets exist

2. Schema Tests

  • Validate input schemas
  • Check type constraints
  • Test required fields

3. Widget Tests

  • HTML is valid
  • JavaScript bundles load
  • React components render
  • Theme switching works

4. Golden Prompt Tests

  • Direct prompts trigger correct tools
  • Indirect prompts trigger correct tools
  • Negative prompts don't trigger tools

Workflow

  1. Start Server in Test Mode

    bash
    npm run start:stdio

  2. Run MCP Inspector Tests Use chatgpt-test-runner agent.

  3. Run Schema Validation Check all tool schemas.

  4. Run Golden Prompt Tests Validate prompts from .chatgpt-app/golden-prompts.json.

  5. Generate Report Save to .chatgpt-app/test-report.json.

Results Format

## Test Results

### MCP Protocol Tests
✓ Tool discovery (15ms)
✓ list-items execution (45ms)
✓ create-item execution (52ms)

### Golden Prompt Tests
Direct: 15/15 passed
Indirect: 14/15 passed
Negative: 9/9 rejected

---
**Overall: 38/39 tests passed**

Fixing Failures

For each failure, explain:

  • What failed
  • Why it failed
  • How to fix it with code example

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

hollaugo/tutorials

chatgpt-app:new

Create a new ChatGPT App from concept to working code. Guides through conceptualization, design, implementation, testing, and deployment.

20 5
Explore
hollaugo/tutorials

chatgpt-app:add-auth

Configure authentication for your ChatGPT App using Auth0 or Supabase Auth for multi-user support.

20 5
Explore
hollaugo/tutorials

chatgpt-app:add-tool

Add a new MCP tool to your ChatGPT App. Guides through tool design, schema creation, and code generation.

20 5
Explore
hollaugo/tutorials

chatgpt-app:deploy

Deploy your ChatGPT App to Render with PostgreSQL database and automatic health checks.

20 5
Explore
hollaugo/tutorials

chatgpt-app:golden-prompts

Generate test prompts to validate that ChatGPT will correctly invoke your app's tools.

20 5
Explore
hollaugo/tutorials

chatgpt-app:add-widget

Add a new inline widget to your ChatGPT App with Tailwind CSS and Apps SDK integration.

20 5
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results