Agent skill
web-fetch
Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.
Install this agent skill to your Project
npx add-skill https://github.com/0xBigBoss/claude-code/tree/main/.claude/skills/web-fetch
SKILL.md
Web Content Fetching
Fetch web content in this order:
- Prefer markdown-native endpoints (
content-type: text/markdown) - Use selector-based HTML extraction for known sites
- Use the bundled Bun fallback script when selectors fail
Prerequisites
Verify required tools before extracting:
command -v curl >/dev/null || echo "curl is required"
command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"
command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"
Install Bun dependencies for the bundled script:
cd ~/.claude/skills/web-fetch && bun install
Default Workflow
Use this as the default flow for any URL:
URL="<url>"
CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
if echo "$CONTENT_TYPE" | grep -q "markdown"; then
curl -sL "$URL"
else
curl -sL "$URL" \
| html2markdown \
--include-selector "article,main,[role=main]" \
--exclude-selector "nav,header,footer,script,style"
fi
Known Site Selectors
| Site | Include Selector | Exclude Selector |
|---|---|---|
| platform.claude.com | #content-container |
- |
| docs.anthropic.com | #content-container |
- |
| developer.mozilla.org | article |
- |
| github.com (docs) | article |
nav,.sidebar |
| Generic | article,main,[role=main] |
nav,header,footer,script,style |
Example:
curl -sL "<url>" \
| html2markdown \
--include-selector "#content-container" \
--exclude-selector "nav,header,footer"
Finding the Right Selector
When a site isn't in the patterns list:
# Check what content containers exist
curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selector
curl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line count
curl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l
Universal Fallback Script
When selectors produce poor output, run the bundled parser:
bun ~/.claude/skills/web-fetch/fetch.ts "<url>"
If already in the skill directory:
bun fetch.ts "<url>"
Options Reference
--include-selector "CSS" # Keep only matching elements
--exclude-selector "CSS" # Remove matching elements
--domain "https://..." # Convert relative links to absolute
Troubleshooting
Empty output with selectors: The page might be markdown-native. Check headers first:
curl -sIL "<url>" | grep -i '^content-type:'
Wrong content selected: The site may have multiple article/main regions:
curl -s "<url>" | grep -o '<article[^>]*>'
html2markdown not found: Install it, then retry selector-based extraction.
bun or script deps missing: Run cd ~/.claude/skills/web-fetch && bun install.
Missing code blocks: Check if the site uses non-standard code formatting.
Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
0xBigBoss/claude-code
canton-nodes
Canton validator node reference data. Use for participant IDs, database names, port availability, and architecture context.
0xBigBoss/claude-code
writing-plans
Use when requirements are clear enough to plan and the work spans multiple steps, files, or verification stages
0xBigBoss/claude-code
writing-skills
Use when creating or editing a skill and you need it to be discoverable, concise, and native to the target harness
0xBigBoss/claude-code
dispatching-parallel-agents
Use when there are multiple independent subtasks that can progress in parallel without overlapping ownership or blocking the next local step
0xBigBoss/claude-code
testing-best-practices
Use when designing tests, writing test cases, or planning test strategy for a module. Covers unit, integration, and e2e layering.
0xBigBoss/claude-code
git-best-practices
Use when creating commits, managing branches, opening PRs, or rewriting history. Not for non-git implementation tasks or repo-specific release policy decisions.
Didn't find tool you were looking for?