Ingest Compliance Document

Single command to ingest a defense/compliance PDF through the full pipeline:

PDF → extractor → cui-marker → doc2qra → taxonomy → memory

Commands

Pipeline Stages

Stage 1: Extract (extractor skill)

• Profile document to detect preset (nist_controls, requirements_spec, etc.)
• Extract sections, tables, figures via deterministic pipeline
• Output: structured JSON + markdown

Stage 2: CUI Check (cui-marker skill)

• Scan extracted text for CUI indicators
• Apply appropriate markings to ArangoDB document metadata
• Flag legacy markings (FOUO, SBU) for conversion

Stage 3: QRA Extraction (doc2qra skill)

• Generate QRA triplets via LLM
• Grounding validation filters hallucinations

Stage 4: Taxonomy (taxonomy skill)

• Extract federated taxonomy tags (bridges: Precision, Resilience, etc.)
• Map NIST control families to taxonomy domains

Stage 5: Memory Storage (memory skill)

• Store QRAs with taxonomy tags in ArangoDB
• Create graph edges between controls and requirements
• Enable multi-hop recall for compliance queries

Stage 6: Control Extraction (automatic via extractor s12)

• Extractor's s12_framework_mapper runs automatically during Stage 1
• Extracts NIST, CWE, ATT&CK, SPARTA, D3FEND, ISO control references from document chunks
• Creates chunk_control_edges (evidence: document references control)
• Creates requirement_control_edges for requirements (SHALL/MUST language, tables, specs)
• Queues proof_jobs for /lean4-prove formal verification of requirement→control claims
• 3-tier extraction: regex (wide net) → RapidFuzz (validate vs 7K control catalog) → classifier (future)
• Enables /memory recall to traverse from compliance documents directly to the controls they reference

Supported Document Types

Example

# Ingest NIST SP 800-171
./run.sh ingest ~/Downloads/sp800-171r2.pdf

# Output:
# [1/5] Extracting... (detected preset: nist_controls, 113 pages)
# [2/5] CUI scanning... (detected: INFOSEC, marking applied)
# [3/5] Generating QRAs... (110 controls → 342 QRAs)
# [4/5] Extracting taxonomy... (14 families mapped)
# [5/5] Storing to memory... (342 QRAs stored, 156 edges created)
# Done. 342 QRAs ingested to scope: compliance