ReddRadar
Agent skill
A/B Test Design
Statistical experiment design and analysis capabilities for product experimentation
Install this agent skill to your Project
npx add-skill https://github.com/a5c-ai/babysitter/tree/main/library/specializations/product-management/skills/ab-test-design
SKILL.md
A/B Test Design Skill
Overview
Specialized skill for statistical experiment design and analysis capabilities. Enables product teams to design rigorous experiments, calculate sample sizes, and interpret results with statistical confidence.
Capabilities
Experiment Design
- Calculate required sample sizes for experiments
- Design experiment variants and hypotheses
- Define success metrics and guardrail metrics
- Create experiment documentation templates
- Design multi-variant tests (A/B/n)
- Plan sequential and Bayesian experiments
Statistical Analysis
- Validate statistical significance of results
- Calculate practical significance and effect sizes
- Detect interaction effects and segments
- Perform power analysis
- Calculate confidence intervals
- Handle multiple comparison corrections
Decision Support
- Recommend ship/iterate/kill decisions
- Identify segment-specific impacts
- Assess long-term vs short-term effects
- Generate experiment reports
- Track experiment velocity metrics
Target Processes
This skill integrates with the following processes:
product-market-fit.js- Validation experiments for PMF hypothesesconversion-funnel-analysis.js- Funnel optimization experimentsbeta-program.js- A/B testing during beta phases
Input Schema
{
"type": "object",
"properties": {
"experimentType": {
"type": "string",
"enum": ["ab", "multivariate", "sequential", "bandit"],
"description": "Type of experiment to design"
},
"hypothesis": {
"type": "string",
"description": "Hypothesis to test"
},
"primaryMetric": {
"type": "object",
"properties": {
"name": { "type": "string" },
"baseline": { "type": "number" },
"mde": { "type": "number", "description": "Minimum detectable effect" }
}
},
"guardrailMetrics": {
"type": "array",
"items": { "type": "string" },
"description": "Metrics that should not regress"
},
"trafficAllocation": {
"type": "number",
"description": "Percentage of traffic for experiment"
},
"confidenceLevel": {
"type": "number",
"default": 0.95,
"description": "Statistical confidence level"
}
},
"required": ["experimentType", "hypothesis", "primaryMetric"]
}
Output Schema
{
"type": "object",
"properties": {
"experimentPlan": {
"type": "object",
"properties": {
"name": { "type": "string" },
"hypothesis": { "type": "string" },
"variants": { "type": "array", "items": { "type": "object" } },
"sampleSize": { "type": "number" },
"duration": { "type": "string" },
"metrics": { "type": "object" }
}
},
"powerAnalysis": {
"type": "object",
"properties": {
"requiredSampleSize": { "type": "number" },
"estimatedDuration": { "type": "string" },
"power": { "type": "number" }
}
},
"implementation": {
"type": "object",
"properties": {
"trackingEvents": { "type": "array", "items": { "type": "string" } },
"segmentation": { "type": "array", "items": { "type": "string" } },
"rolloutPlan": { "type": "string" }
}
},
"analysisFramework": {
"type": "object",
"properties": {
"primaryAnalysis": { "type": "string" },
"secondaryAnalyses": { "type": "array", "items": { "type": "string" } },
"decisionCriteria": { "type": "object" }
}
}
}
}
Usage Example
const experimentDesign = await executeSkill('ab-test-design', {
experimentType: 'ab',
hypothesis: 'Adding social proof to pricing page increases conversion by 10%',
primaryMetric: {
name: 'pricing_page_conversion',
baseline: 0.05,
mde: 0.10
},
guardrailMetrics: ['revenue_per_visitor', 'bounce_rate'],
trafficAllocation: 50,
confidenceLevel: 0.95
});
Dependencies
- Statistical libraries for power analysis
- Experimentation platform integrations (Optimizely, LaunchDarkly, etc.)
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
a5c-ai/babysitter
gsd-tools
Central utility skill for GSD operations. Provides config parsing, slug generation, timestamps, path operations, and orchestrates calls to other specialized skills. Acts as the unified entry point that the original gsd-tools.cjs provided via its lib/ modules (commands, config, core, init).
a5c-ai/babysitter
model-profile-resolution
Resolve model profile (quality/balanced/budget) at orchestration start and map agents to specific models. Enables cost/quality tradeoffs by selecting appropriate AI models for each agent role.
a5c-ai/babysitter
verification-suite
Plan structure validation, phase completeness checks, reference integrity verification, and artifact existence confirmation. Provides the structured verification layer ensuring GSD artifacts are well-formed and complete.
a5c-ai/babysitter
state-management
STATE.md reading, writing, and field-level updates. Provides cross-session state persistence via .planning/STATE.md with structured fields for current task, completed phases, blockers, decisions, and quick tasks.
a5c-ai/babysitter
git-integration
Git commit patterns, formats, and conventions for GSD methodology. Provides atomic commits per task, structured commit messages, planning file commits, branch management, and milestone tag operations.
a5c-ai/babysitter
frontmatter-parsing
YAML frontmatter parsing and manipulation for .planning/ documents. Provides read, write, update, query, and validation operations on frontmatter blocks in GSD markdown artifacts.
Didn't find tool you were looking for?