Autonomous Deep Research Loop

Iterative research protocol with fresh context per iteration, externalized state, and convergence detection for deep technical investigation.

Runtime path resolution:

• OpenCode/Copilot runtime: .opencode/agent/*.md
• Claude runtime: .claude/agents/*.md
• Codex runtime: .codex/agents/*.toml

1. WHEN TO USE

When to Use This Skill

Use this skill when:

• Deep investigation requiring multiple rounds of discovery
• Topic spans 3+ technical domains or sources
• Initial findings need progressive refinement
• Overnight or unattended research sessions
• Research where prior findings inform subsequent queries

When NOT to Use

• Simple, single-question research (use direct codebase search or /spec_kit:plan)
• Known-solution documentation (use /spec_kit:plan)
• Implementation tasks (use /spec_kit:implement)
• Quick codebase searches (use @context or direct Grep/Glob)
• Fewer than 3 sources needed (single-pass research suffices)

Keyword Triggers

autoresearch, deep research, autonomous research, research loop, iterative research, multi-round research, deep investigation, comprehensive research

For iterative code review and quality auditing, see sk-deep-review.

2. SMART ROUTING

Resource Loading Levels

Smart Router Pseudocode

from pathlib import Path

SKILL_ROOT = Path(__file__).resolve().parent
RESOURCE_BASES = (SKILL_ROOT / "references", SKILL_ROOT / "assets")
DEFAULT_RESOURCE = "references/quick_reference.md"

INTENT_SIGNALS = {
    "LOOP_SETUP": {"weight": 4, "keywords": ["autoresearch", "deep research", "research loop", "autonomous research"]},
    "ITERATION": {"weight": 4, "keywords": ["iteration", "next round", "continue research", "research cycle"]},
    "CONVERGENCE": {"weight": 3, "keywords": ["convergence", "stop condition", "diminishing returns", "stuck"]},
    "STATE": {"weight": 3, "keywords": ["state file", "JSONL", "strategy", "resume", "auto-resume"]},
}

NOISY_SYNONYMS = {
    "LOOP_SETUP": {"run research": 2.0, "investigate deeply": 1.8, "overnight research": 1.5},
    "ITERATION": {"another pass": 1.5, "keep searching": 1.4, "dig deeper": 1.6},
    "CONVERGENCE": {"good enough": 1.4, "stop when": 1.5, "diminishing": 1.6},
    "STATE": {"pick up where": 1.5, "continue from": 1.4, "resume": 1.8},
}

RESOURCE_MAP = {
    "LOOP_SETUP": ["references/loop_protocol.md", "references/state_format.md", "assets/deep_research_config.json"],
    "ITERATION": ["references/loop_protocol.md", "references/convergence.md"],
    "CONVERGENCE": ["references/convergence.md"],
    "STATE": ["references/state_format.md", "assets/deep_research_strategy.md"],
}

LOADING_LEVELS = {
    "ALWAYS": [DEFAULT_RESOURCE],
    "ON_DEMAND_KEYWORDS": ["full protocol", "all templates", "complete reference"],
    "ON_DEMAND": ["references/loop_protocol.md", "references/state_format.md", "references/convergence.md"],
}

Scoped Guard

def _guard_in_skill():
    """Verify this skill is active before loading resources."""
    if not hasattr(_guard_in_skill, '_active'):
        _guard_in_skill._active = True
    return _guard_in_skill._active

def discover_markdown_resources(base_path: Path) -> list[str]:
    """Discover all .md files in the assets directory."""
    return sorted(str(p.relative_to(base_path)) for p in (base_path / "references").glob("*.md"))

Phase Detection

Detect the current research phase from dispatch context to load appropriate resources:

3. HOW IT WORKS

Architecture: 3-Layer Integration

User invokes: /spec_kit:deep-research "topic"
                    |
                    v
    ┌─────────────────────────────────┐
    │  /spec_kit:deep-research command│  Layer 1: Command
    │  (YAML workflow + loop config)  │  Manages loop lifecycle
    └──────────────┬──────────────────┘
                   |
                   v
    ┌─────────────────────────────────┐
    │     YAML Loop Engine            │  Layer 2: Workflow
    │  - Init (config, strategy)       │  Dispatch, evaluate, decide
    │  - Loop (dispatch + converge)   │
    │  - Synthesize (final output)     │
    │  - Save (memory context)        │
    └──────────────┬──────────────────┘
                   |  dispatches per iteration
                   v
    ┌─────────────────────────────────┐
    │    @deep-research (LEAF agent)  │  Layer 3: Agent
    │  - Reads: state + strategy      │  Fresh context each time
    │  - Executes ONE research cycle  │
    │  - Writes: findings + state      │
    │  - Tools: WebFetch, Grep, etc.  │
    └──────────────┬──────────────────┘
                   |
                   v
    ┌─────────────────────────────────┐
    │        State Files (disk)       │  Externalized State
    │  deep-research-config.json       │  Persists across iterations
    │  deep-research-state.jsonl      │
    │  deep-research-strategy.md      │
    │  research/iterations/iteration-NNN.md │
    │  research/research.md (workflow-owned │
    │  progressive synthesis)         │
    └─────────────────────────────────┘

Core Innovation: Fresh Context Per Iteration

Each agent dispatch gets a fresh context window. State continuity comes from files, not memory. This solves context degradation in long research sessions.

Adapted from: karpathy/autoresearch (loop concept), AGR (fresh context "Ralph Loop"), pi-autoresearch (JSONL state), autoresearch-opencode (context injection).

Data Flow

Init --> Create config.json, strategy.md, state.jsonl
  |
Loop --> Read state --> Check convergence --> Dispatch @deep-research
  |                                              |
  |                                              v
  |                                         Agent executes:
  |                                         1. Read state files
  |                                         2. Determine focus
  |                                         3. Research (3-5 actions)
  |                                         4. Write iteration-NNN.md
  |                                         5. Update strategy.md
  |                                         6. Append state.jsonl
  |                                              |
  +<--- Evaluate results <-----------------------+
  |
  +--- Continue? --> Yes: next iteration
  |                  No: exit loop
  v
Synthesize --> Compile final research/research.md
  |
Save --> generate-context.js --> verify memory artifact

Key Concepts

4. RULES

ALWAYS

1. Read state first -- Agent must read JSONL and strategy.md before any research action
2. One focus per iteration -- Pick ONE research sub-topic from strategy.md "Next Focus"
3. Externalize findings -- Write to iteration-NNN.md, not held in agent context
4. Update strategy -- Append to "What Worked"/"What Failed", update "Next Focus"
5. Report newInfoRatio -- Every iteration JSONL record must include newInfoRatio
6. Respect exhausted approaches -- Never retry approaches in the "Exhausted" list
7. Cite sources -- Every finding must cite [SOURCE: url] or [SOURCE: file:line]
8. Use generate-context.js for memory saves -- Never manually create memory files
9. Treat research/research.md as workflow-owned -- Iteration findings feed synthesis; the workflow owns the canonical research/research.md
10. Document ruled-out directions per iteration -- Every iteration must include what was tried and failed
11. Report newInfoRatio + 1-sentence novelty justification -- Every JSONL iteration record must include both
12. Quality guards must pass before convergence -- Source diversity, focus alignment, and no single-weak-source checks must pass before STOP can trigger

NEVER

1. Dispatch sub-agents -- @deep-research is LEAF-only (NDP compliance)
2. Hold findings in context -- Write everything to files
3. Exceed TCB -- Target 8-11 tool calls per iteration (max 12)
4. Ask the user -- Autonomous execution; make best-judgment decisions
5. Skip convergence checks -- Every iteration must be evaluated
6. Modify config after init -- Config is read-only after initialization
7. Overwrite prior findings -- Append to research/research.md, never replace

Iteration Status Enum

complete | timeout | error | stuck | insight | thought

• insight: Low newInfoRatio but important conceptual breakthrough
• thought: Analytical-only iteration, no evidence gathering

EXPERIMENTAL / REFERENCE-ONLY FEATURES

These concepts remain documented for future design work, but they are not part of the live executable contract for /spec_kit:deep-research:

1. Wave orchestration -- parallel question fan-out, pruning, and breakthrough logic
2. Checkpoint commits -- per-iteration git commits
3. Segment transitions / :restart -- multi-segment session partitioning
4. Alternate CLI dispatch -- process-isolated claude -p or similar dispatch modes

ESCALATE IF

1. 3+ consecutive timeouts -- Infrastructure issue, not research problem
2. State file corruption unrecoverable -- Cannot reconstruct from JSONL or iteration files
3. All approaches exhausted with questions remaining -- Research may need human guidance
4. Security concern in findings -- Proprietary code or credentials discovered
5. All recovery tiers exhausted -- No automatic recovery path remaining

5. REFERENCES

Core Documentation

Templates

6. SUCCESS CRITERIA

Loop Completion

• Research loop ran to convergence or max iterations
• All state files present and consistent (config, JSONL, strategy)
• research/research.md produced with findings from all iterations
• Memory context saved via generate-context.js

Quality Gates

Convergence Report

Every completed loop produces a convergence report:

• Stop reason (converged, max_iterations, all_questions_answered, stuck_unrecoverable)
• Total iterations completed
• Questions answered ratio
• Average newInfoRatio trend

7. INTEGRATION POINTS

Framework Integration

This skill operates within the behavioral framework defined in CLAUDE.md.

Key integrations:

• Gate 2: Skill routing via skill_advisor.py (keywords: autoresearch, deep research)
• Gate 3: File modifications require spec folder question per CLAUDE.md Gate 3
• Memory: Context preserved via Spec Kit Memory MCP (generate-context.js)
• Orchestrator: @orchestrate dispatches @deep-research as LEAF agent

Memory Integration

Before research:
  memory_context({ input: topic, mode: "deep", intent: "understand" })
  --> Loads prior research into strategy.md "Known Context"

During research (each iteration):
  Agent writes research/iterations/iteration-NNN.md
  Agent updates research/deep-research-strategy.md
  Agent appends research/deep-research-state.jsonl

After research:
  node .opencode/skill/system-spec-kit/scripts/dist/memory/generate-context.js [spec-folder]
  # No additional indexing step is part of the live workflow contract.

Command Integration

8. RELATED RESOURCES

Worked Examples

Deep Research on Unknown Topic:

1. /spec_kit:deep-research:auto "WebSocket reconnection strategies across browsers"
2. Init creates config, strategy with 5 key questions
3. Iterations 1-3: Broad survey, official docs, codebase patterns
4. Iterations 4-6: Deep dive into specific strategies, edge cases
5. Iteration 7: Convergence detected after recent newInfoRatio values stay below the configured threshold
6. Synthesis produces 17-section research/research.md
7. Memory saved via generate-context.js

Narrow Research with Early Convergence:

1. /spec_kit:deep-research:auto "What CSS properties trigger GPU compositing?"
2. Init creates config with 2 key questions
3. Iteration 1: Finds definitive answer from official specs
4. All questions answered after iteration 1
5. Loop stops cleanly, research/research.md produced

Stuck Recovery Example:

1. Iterations 4-6 all have newInfoRatio below the configured threshold
2. Stuck recovery triggers at iteration 7
3. Recovery widens focus to least-explored question
4. Iteration 7 finds new angle, newInfoRatio jumps to 0.4
5. Loop continues productively

Design Origins

Agents

Commands

For one-page cheat sheet: See quick_reference.md

For code review capabilities, see sk-deep-review.