Agent skill

crawl4ai

Web crawler that converts URLs to clean markdown. Use when user wants to fetch a webpage, extract web content, convert URL to markdown, or scrape website content.

View SKILL.md on GitHub Repository
Stars 0
Forks 0

Install this agent skill to your Project

npx add-skill https://github.com/alexjx/skills/tree/main/skills/crawl4ai

SKILL.md

Crawl4AI Web Crawler

Converts web pages to clean, LLM-friendly markdown using crawl4ai.

Requirements

Setup

Before first use, run the setup script:

bash
skills/crawl4ai/scripts/setup.sh

This will:

  1. Check for uv installation
  2. Run uv sync to create .venv/ and install dependencies
  3. Install Playwright Chromium browser (python -m playwright install chromium --with-deps)
  4. Run crawl4ai-doctor to verify installation

Usage

To crawl a URL and get markdown (from the skill directory):

bash
cd skills/crawl4ai
uv run python scripts/crawl.py "https://example.com"
# Or after setup:
uv run crawl "https://example.com"

Options:

  • --output FILE - Save markdown to file instead of stdout
  • --include-links - Include hyperlinks in markdown output
  • --include-images - Include image references
  • --no-headless - Show browser window (default: headless)
  • --timeout SECONDS - Page load timeout (default: 30)
  • --verbose - Enable crawler progress logs

Examples

Basic crawl

bash
cd skills/crawl4ai
uv run python scripts/crawl.py "https://docs.python.org/3/"

Save to file

bash
uv run python scripts/crawl.py "https://example.com" --output content.md

With all options

bash
uv run python scripts/crawl.py "https://example.com" \
  --include-links \
  --include-images \
  --timeout 60 \
  --output result.md

Troubleshooting

If crawling fails:

  1. Run uv run crawl4ai-doctor to diagnose
  2. Ensure playwright browsers are installed: uv run python -m playwright install chromium
  3. Check the URL is accessible and not blocked by robots.txt

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

alexjx/skills

agent-creator

Create Claude Code subagents. Use when user wants to create a subagent, specialized agent, or custom AI assistant for Claude Code.

0 0
Explore
alexjx/skills

skill-creator

Create new Claude Code skills in this repository. Use when user wants to add a new skill, create a skill, implement a skill, or extend Claude's capabilities with a custom workflow.

0 0
Explore
alexjx/skills

commit

Guide for creating atomic commits with conventional commit format. Use when user wants to commit changes.

0 0
Explore
alexjx/skills

agent-harness

Framework for orchestrating multiple AI agent sessions to complete complex, long-running software projects. Use when the user wants to set up or use the long-running agent harness pattern for managing multi-session projects, breaking work into discrete sessions with persistent state tracking, or implementing incremental progress workflows across multiple Claude Code sessions.

0 0
Explore
mattpocock/skills

edit-article

Edit and improve articles by restructuring sections, improving clarity, and tightening prose. Use when user wants to edit, revise, or improve an article draft.

111,310 9,758
Explore
mattpocock/skills

handoff

Compact the current conversation into a handoff document for another agent to pick up.

111,310 9,758
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results