Agent skill
blog-scraper
Scrape blog posts via RSS feeds (free, no API key) with Apify fallback for JS-heavy sites. Use when you need to monitor competitor blogs, track industry content, or aggregate blog posts by keyword.
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/other/other/blog-scraper
SKILL.md
Blog Scraper
Scrape blog posts via RSS/Atom feeds (free) with optional Apify fallback for JS-heavy sites.
Quick Start
For RSS mode (free), only dependency is pip install requests. No API key needed.
# Scrape a blog's RSS feed
python3 skills/blog-scraper/scripts/scrape_blogs.py \
--urls "https://example.com/blog" --days 30
# Multiple blogs with keyword filter
python3 skills/blog-scraper/scripts/scrape_blogs.py \
--urls "https://blog1.com,https://blog2.com" --keywords "AI,marketing" --output summary
# Force Apify for JS-heavy sites
python3 skills/blog-scraper/scripts/scrape_blogs.py \
--urls "https://example.com" --mode apify
How It Works
Auto Mode (default)
- For each URL, tries to discover an RSS/Atom feed:
- Checks HTML
<link rel="alternate">tags - Probes common paths:
/feed,/rss,/atom.xml,/feed.xml,/rss.xml,/blog/feed,/index.xml
- Checks HTML
- Parses discovered feeds (supports RSS 2.0 and Atom)
- If any URLs fail, falls back to Apify
jupri/rss-xml-scraper(if token available) - Applies date and keyword filtering client-side
RSS Mode
Only tries RSS feeds, no Apify fallback.
Apify Mode
Uses Apify actor directly, skipping RSS discovery.
CLI Reference
| Flag | Default | Description |
|---|---|---|
--urls |
required | Blog URL(s), comma-separated |
--keywords |
none | Keywords to filter (comma-separated, OR logic) |
--days |
30 | Only include posts from last N days |
--max-posts |
50 | Max posts to return |
--mode |
auto | auto (RSS + fallback), rss (RSS only), apify (Apify only) |
--output |
json | Output format: json or summary |
--token |
env var | Apify token (only needed for Apify mode/fallback) |
--timeout |
300 | Max seconds for Apify run |
Cost
- RSS mode: Free (no API, no tokens)
- Apify mode: Uses
jupri/rss-xml-scraper— minimal Apify credits
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
Didn't find tool you were looking for?