Agent skill
web-scraping-crawling
Scrape, crawl, and extract content from web pages. Use when the user needs to extract text, articles, or structured data from websites, or crawl multiple pages from a site.
Install this agent skill to your Project
npx add-skill https://github.com/binome-dev/humcp/tree/main/src/tools/web_scraping
SKILL.md
Web Scraping & Crawling Tools
Tools for scraping individual pages, crawling entire websites, and extracting structured data.
Free Tools (No API Key)
Crawl4AI - Browser-based scraping
result = await crawl4ai_scrape(
url="https://example.com",
extract_markdown=True,
max_length=5000
)
newspaper4k - Article extraction
result = await newspaper_extract(
url="https://example.com/article",
max_length=10000
)
trafilatura - Content extraction
result = await trafilatura_extract(
url="https://example.com",
include_comments=False,
output_format="txt" # or "markdown", "json", "xml"
)
API-Powered Tools
Firecrawl (FIRECRAWL_API_KEY)
# Single page
result = await firecrawl_scrape(url="https://example.com")
# Multi-page crawl
result = await firecrawl_crawl(url="https://example.com", limit=10)
Spider (SPIDER_API_KEY)
# Single page
result = await spider_scrape(url="https://example.com")
# Multi-page crawl
result = await spider_crawl(url="https://example.com", limit=10)
Jina Reader (JINA_API_KEY optional)
# Read a URL
result = await jina_reader(url="https://example.com")
# Web search
result = await jina_search(query="latest AI news")
ScrapeGraph (SGAI_API_KEY)
# AI-powered structured extraction
result = await scrapegraph_scrape(
url="https://example.com",
prompt="Extract all product names and prices"
)
# Convert page to markdown
result = await scrapegraph_markdownify(url="https://example.com")
AgentQL (AGENTQL_API_KEY)
result = await agentql_query(
url="https://example.com",
query='{ text_content[] }'
)
Browserbase (BROWSERBASE_API_KEY, BROWSERBASE_PROJECT_ID)
result = await browserbase_scrape(url="https://example.com")
BrightData (BRIGHTDATA_API_KEY)
result = await brightdata_scrape(url="https://example.com")
Oxylabs (OXYLABS_USERNAME, OXYLABS_PASSWORD)
result = await oxylabs_scrape(
url="https://example.com",
source="universal",
render_javascript=False
)
Linkup (LINKUP_API_KEY)
result = await linkup_search(
query="machine learning tutorials",
depth="standard" # or "deep"
)
Apify (APIFY_API_TOKEN)
result = await apify_run_actor(
actor_id="apify/web-scraper",
input_data={"startUrls": [{"url": "https://example.com"}]}
)
Response Format
All tools return a consistent response:
{
"success": true,
"data": {
"url": "https://example.com",
"title": "Page Title",
"content": "Extracted text content...",
"markdown": "# Markdown content...",
"metadata": {"authors": ["..."], "publish_date": "..."}
}
}
Crawl tools return multiple pages:
{
"success": true,
"data": {
"pages": [
{"url": "...", "title": "...", "content": "..."}
],
"total_pages": 5
}
}
When to Use
| Need | Tool |
|---|---|
| Simple article text | newspaper_extract |
| Clean main content | trafilatura_extract |
| JavaScript-heavy pages | crawl4ai_scrape, browserbase_scrape |
| Multi-page crawl | firecrawl_crawl, spider_crawl |
| Structured data extraction | scrapegraph_scrape, agentql_query |
| Bypass anti-bot protection | brightdata_scrape, oxylabs_scrape |
| Web search | jina_search, linkup_search |
| Pre-built scraper workflows | apify_run_actor |
| Page to markdown | firecrawl_scrape, scrapegraph_markdownify, jina_reader |
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
llm
Generate text, call tools, and get structured JSON output using LLM providers (Claude, OpenAI, Gemini, Ollama). Use when the user needs to call an LLM API for text generation, function calling, or structured data extraction.
calendar-tools
Manages calendar bookings and video conferencing. Use when the user needs to check availability, create bookings on Cal.com, or schedule and manage Zoom meetings.
http-api-client
Make HTTP requests to any URL or API endpoint. Use when the user needs to call a REST API, fetch data from a URL, send webhooks, or test HTTP endpoints. Supports GET, POST, PUT, PATCH, DELETE methods with custom headers and JSON body.
processing-data
Processes CSV files and pandas DataFrames. Use when working with CSV files, tabular data, spreadsheets, or when the user asks to query, analyze, or manipulate structured data.
persistent-memory
Store, search, and retrieve persistent memories and conversation history using Mem0 or Zep. Use when the user needs to remember facts, maintain context across sessions, or manage conversational memory.
storage
Use these tools for S3-compatible object storage operations with MinIO
Didn't find tool you were looking for?