Scrape Strategy Skill (v2.0)

You are an expert at web scraping strategy, understanding when and how to extract content from protected websites.

8-Tier Scraping System

Primr uses an 8-tier fallback system for web scraping:

Key Features

Sticky Tier

Once a tier works for a host, it's tried first for subsequent pages from that host.

Circuit Breaker

After 3 consecutive failures of the same tier for a host, that tier is skipped.

Cookie Handoff

Cookies obtained by browser tiers are reused by faster HTTP tiers.

Soft Block Detection

Checks actual content, not just HTTP status. Catches "200 OK" traps where the response is a block page.

Tier Selection Heuristics

When to Expect Tier 1-2 (Playwright)

• Modern SPA sites (React, Vue, Angular)
• Sites with dynamic content loading
• Sites with accordions or expandable sections

When to Expect Tier 3 (curl_cffi)

• Sites with TLS fingerprint detection
• Cloudflare-protected sites (some)
• Sites that block Python user agents

When to Expect Tier 4-5 (DrissionPage)

• Sites with aggressive bot detection
• Sites requiring JavaScript challenges
• Sites with CAPTCHA (limited success)

When to Expect Tier 6-7 (HTTP)

• Simple static sites
• API endpoints
• Sites without JavaScript requirements

When to Expect Tier 8 (Vision)

• Sites with complex layouts
• PDF-heavy content
• Sites where text extraction fails

Error Handling Patterns

Soft Block Detection

Indicators of soft blocks:
- Content length < 1000 bytes
- Contains "access denied", "blocked", "captcha"
- Missing expected content markers
- Redirect to login/challenge page

Tier Escalation

On failure:
1. Log failure reason
2. Check circuit breaker status
3. Try next tier if available
4. After 3 consecutive same-error failures, stop escalation

Recovery Strategies

Interpreting Scrape Results

Success Metrics

+ 34/46 pages scraped

34 = pages successfully scraped
46 = total pages selected
12 = pages that failed (blocked, timeout, etc.)

Quality Assessment

• 70%+ success rate: Good coverage
• 50-70% success rate: Acceptable for protected sites
• <50% success rate: Consider deep mode instead

Tier Statistics

tier_stats:
  playwright: 20
  playwright_aggressive: 8
  curl_cffi: 4
  drissionpage_stealth: 2

Higher numbers in lower tiers = more protected site.

Mode Selection Guidance

Use Scrape Mode When:

• Target is company's own website
• Need first-party content
• Budget is limited
• Time is limited

Use Deep Mode When:

• Site is heavily protected (>50% failure rate)
• Need external sources
• Company website is sparse
• Need market/competitive context

Use Full Mode When:

• Comprehensive report needed
• Budget allows (~$1.50)
• Time allows (~30 min)
• Both first-party and external sources valuable

Example Workflow

User: "The site seems heavily protected"

Agent:
1. Check prior scrape results:
   - Success rate: 35%
   - Most pages blocked at tier 4
   
2. Recommend strategy:
   "This site has aggressive bot protection. I recommend:
    - Use deep mode for external research
    - Or accept partial scrape results
    
    Deep mode will gather information from external sources
    without hitting the protected site directly."

3. If user chooses deep mode:
   estimate_run(company, url, "deep")
   → Cost: $0.80, Time: ~12 minutes

Constraints

• Patient Timeout: 90s max per page (allows multiple tier attempts)
• Concurrency: 3 concurrent pages default
• Circuit Breaker: 3 failures before tier skip
• Smart Escalation: Stops after 3 consecutive same-error failures