MCPs
WebScraping.AI MCP Server

WebScraping.AI MCP Server

MCP server for advanced web scraping and AI-driven data extraction

Stars

Forks

Watchers

Issues

WebScraping.AI MCP Server implements the Model Context Protocol to provide web data extraction and question answering functionalities. It integrates with WebScraping.AI to offer robust tools for retrieving, rendering, and parsing web content, including structured data and natural language answers from web pages. It supports JavaScript rendering, proxy management, device emulation, and custom extraction configurations, making it suitable for both individual and team deployments in AI-assisted workflows.

Key Features

Question answering about web page content

Structured and plain text data extraction from websites

HTML content retrieval with optional JavaScript rendering

Concurrent web request management with rate limiting

Support for multiple proxy types and country selection

Custom JavaScript execution on target web pages

Device emulation (desktop, mobile, tablet)

Integration with Cursor and Claude Desktop for AI workflows

Account usage and concurrency monitoring

CSS selector-based content extraction

Use Cases

Automated extraction of article content from news sites

Gathering structured product data from e-commerce platforms

Answering user questions directly from web sources

Extracting specific information using CSS selectors

Bypassing bot protections with proxy rotation and country selection

Rendering and scraping content from JavaScript-heavy websites

Integrating web scraping into AI agent workflows for contextual responses

Monitoring and analyzing competitor websites

Running custom JavaScript to manipulate or retrieve page elements

Scaling web scraping tasks with concurrency and rate limiting

README

WebScraping.AI MCP Server

A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.

Features

Question answering about web page content
Structured data extraction from web pages
HTML content retrieval with JavaScript rendering
Plain text extraction from web pages
CSS selector-based content extraction
Multiple proxy types (datacenter, residential) with country selection
JavaScript rendering using headless Chrome/Chromium
Concurrent request management with rate limiting
Custom JavaScript execution on target pages
Device emulation (desktop, mobile, tablet)
Account usage monitoring

Installation

Running with npx

bash

env WEBSCRAPING_AI_API_KEY=your_api_key npx -y webscraping-ai-mcp

Manual Installation

bash

# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

# Install dependencies
npm install

# Run
npm start

Configuring in Cursor

Note: Requires Cursor version 0.45.6+

The WebScraping.AI MCP server can be configured in two ways in Cursor:

Project-specific Configuration (recommended for team projects): Create a .cursor/mcp.json file in your project directory:

json

{
  "servers": {
    "webscraping-ai": {
      "type": "command",
      "command": "npx -y webscraping-ai-mcp",
      "env": {
        "WEBSCRAPING_AI_API_KEY": "your-api-key",
        "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
      }
    }
  }
}

Global Configuration (for personal use across all projects): Create a ~/.cursor/mcp.json file in your home directory with the same configuration format as above.

If you are using Windows and are running into issues, try using cmd /c "set WEBSCRAPING_AI_API_KEY=your-api-key && npx -y webscraping-ai-mcp" as the command.

This configuration will make the WebScraping.AI tools available to Cursor's AI agent automatically when relevant for web scraping tasks.

Running on Claude Desktop

Add this to your claude_desktop_config.json:

json

{
  "mcpServers": {
    "mcp-server-webscraping-ai": {
      "command": "npx",
      "args": ["-y", "webscraping-ai-mcp"],
      "env": {
        "WEBSCRAPING_AI_API_KEY": "YOUR_API_KEY_HERE",
        "WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
      }
    }
  }
}

Configuration

Environment Variables

Required

WEBSCRAPING_AI_API_KEY: Your WebScraping.AI API key
- Required for all operations
- Get your API key from WebScraping.AI

Optional Configuration

WEBSCRAPING_AI_CONCURRENCY_LIMIT: Maximum number of concurrent requests (default: 5)
WEBSCRAPING_AI_DEFAULT_PROXY_TYPE: Type of proxy to use (default: residential)
WEBSCRAPING_AI_DEFAULT_JS_RENDERING: Enable/disable JavaScript rendering (default: true)
WEBSCRAPING_AI_DEFAULT_TIMEOUT: Maximum web page retrieval time in ms (default: 15000, max: 30000)
WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT: Maximum JavaScript rendering time in ms (default: 2000)

Configuration Examples

For standard usage:

bash

# Required
export WEBSCRAPING_AI_API_KEY=your-api-key

# Optional - customize behavior (default values)
export WEBSCRAPING_AI_CONCURRENCY_LIMIT=5
export WEBSCRAPING_AI_DEFAULT_PROXY_TYPE=residential # datacenter or residential
export WEBSCRAPING_AI_DEFAULT_JS_RENDERING=true
export WEBSCRAPING_AI_DEFAULT_TIMEOUT=15000
export WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT=2000

Available Tools

1. Question Tool (`webscraping_ai_question`)

Ask questions about web page content.

json

{
  "name": "webscraping_ai_question",
  "arguments": {
    "url": "https://example.com",
    "question": "What is the main topic of this page?",
    "timeout": 30000,
    "js": true,
    "js_timeout": 2000,
    "wait_for": ".content-loaded",
    "proxy": "datacenter",
    "country": "us"
  }
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": "The main topic of this page is examples and documentation for HTML and web standards."
    }
  ],
  "isError": false
}

2. Fields Tool (`webscraping_ai_fields`)

Extract structured data from web pages based on instructions.

json

{
  "name": "webscraping_ai_fields",
  "arguments": {
    "url": "https://example.com/product",
    "fields": {
      "title": "Extract the product title",
      "price": "Extract the product price",
      "description": "Extract the product description"
    },
    "js": true,
    "timeout": 30000
  }
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": {
        "title": "Example Product",
        "price": "$99.99",
        "description": "This is an example product description."
      }
    }
  ],
  "isError": false
}

3. HTML Tool (`webscraping_ai_html`)

Get the full HTML of a web page with JavaScript rendering.

json

{
  "name": "webscraping_ai_html",
  "arguments": {
    "url": "https://example.com",
    "js": true,
    "timeout": 30000,
    "wait_for": "#content-loaded"
  }
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": "<html>...[full HTML content]...</html>"
    }
  ],
  "isError": false
}

4. Text Tool (`webscraping_ai_text`)

Extract the visible text content from a web page.

json

{
  "name": "webscraping_ai_text",
  "arguments": {
    "url": "https://example.com",
    "js": true,
    "timeout": 30000
  }
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": "Example Domain\nThis domain is for use in illustrative examples in documents..."
    }
  ],
  "isError": false
}

5. Selected Tool (`webscraping_ai_selected`)

Extract content from a specific element using a CSS selector.

json

{
  "name": "webscraping_ai_selected",
  "arguments": {
    "url": "https://example.com",
    "selector": "div.main-content",
    "js": true,
    "timeout": 30000
  }
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": "<div class=\"main-content\">This is the main content of the page.</div>"
    }
  ],
  "isError": false
}

6. Selected Multiple Tool (`webscraping_ai_selected_multiple`)

Extract content from multiple elements using CSS selectors.

json

{
  "name": "webscraping_ai_selected_multiple",
  "arguments": {
    "url": "https://example.com",
    "selectors": ["div.header", "div.product-list", "div.footer"],
    "js": true,
    "timeout": 30000
  }
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": [
        "<div class=\"header\">Header content</div>",
        "<div class=\"product-list\">Product list content</div>",
        "<div class=\"footer\">Footer content</div>"
      ]
    }
  ],
  "isError": false
}

7. Account Tool (`webscraping_ai_account`)

Get information about your WebScraping.AI account.

json

{
  "name": "webscraping_ai_account",
  "arguments": {}
}

Example response:

json

{
  "content": [
    {
      "type": "text",
      "text": {
        "requests": 5000,
        "remaining": 4500,
        "limit": 10000,
        "resets_at": "2023-12-31T23:59:59Z"
      }
    }
  ],
  "isError": false
}

Common Options for All Tools

The following options can be used with all scraping tools:

timeout: Maximum web page retrieval time in ms (15000 by default, maximum is 30000)
js: Execute on-page JavaScript using a headless browser (true by default)
js_timeout: Maximum JavaScript rendering time in ms (2000 by default)
wait_for: CSS selector to wait for before returning the page content
proxy: Type of proxy, datacenter or residential (residential by default)
country: Country of the proxy to use (US by default). Supported countries: us, gb, de, it, fr, ca, es, ru, jp, kr, in
custom_proxy: Your own proxy URL in "http://user:password@host:port" format
device: Type of device emulation. Supported values: desktop, mobile, tablet
error_on_404: Return error on 404 HTTP status on the target page (false by default)
error_on_redirect: Return error on redirect on the target page (false by default)
js_script: Custom JavaScript code to execute on the target page

Error Handling

The server provides robust error handling:

Automatic retries for transient errors
Rate limit handling with backoff
Detailed error messages
Network resilience

Example error response:

json

{
  "content": [
    {
      "type": "text",
      "text": "API Error: 429 Too Many Requests"
    }
  ],
  "isError": true
}

Integration with LLMs

This server implements the Model Context Protocol, making it compatible with any MCP-enabled LLM platforms. You can configure your LLM to use these tools for web scraping tasks.

Example: Configuring Claude with MCP

javascript

const { Claude } = require('@anthropic-ai/sdk');
const { Client } = require('@modelcontextprotocol/sdk/client/index.js');
const { StdioClientTransport } = require('@modelcontextprotocol/sdk/client/stdio.js');

const claude = new Claude({
  apiKey: process.env.ANTHROPIC_API_KEY
});

const transport = new StdioClientTransport({
  command: 'npx',
  args: ['-y', 'webscraping-ai-mcp'],
  env: {
    WEBSCRAPING_AI_API_KEY: 'your-api-key'
  }
});

const client = new Client({
  name: 'claude-client',
  version: '1.0.0'
});

await client.connect(transport);

// Now you can use Claude with WebScraping.AI tools
const tools = await client.listTools();
const response = await claude.complete({
  prompt: 'What is the main topic of example.com?',
  tools: tools
});

Development

bash

# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

# Install dependencies
npm install

# Run tests
npm test

# Add your .env file
cp .env.example .env

# Start the inspector
npx @modelcontextprotocol/inspector node src/index.js

Contributing

Fork the repository
Create your feature branch
Run tests: npm test
Submit a pull request

License

MIT License - see LICENSE file for details

Star History

Repository Owner

webscraping-ai

Organization

Repository Details

Language JavaScript

Default Branch master

Size 79 KB

Contributors 1

MCP Verified Nov 12, 2025

Programming Languages

JavaScript

96.78%

Dockerfile

3.22%

Related MCPs

Discover similar Model Context Protocol servers

Maigret MCP Server

OSINT username and URL search server for the Model Context Protocol.

Maigret MCP Server provides Model Context Protocol (MCP) integration for the Maigret OSINT tool, enabling AI and context-aware applications to search for usernames across hundreds of social networks and analyze URLs. Designed for seamless operation with MCP-compatible clients like Claude Desktop, it supports multiple output formats and advanced filtering options. The server can be installed via Docker or npm, offers Docker-based deployment for consistent performance, and facilitates responsible OSINT research.

⭐ 205
MCP
BurtTheCoder/mcp-maigret

Baseline MCP Server

Brings web platform API support status to AI models via Model Context Protocol.

Baseline MCP Server serves as a Model Context Protocol (MCP) server, providing API endpoints to retrieve the baseline support status of web platform features. It fetches data from the Web Platform Dashboard API and delivers web API compatibility details, including browser support levels, baseline status, and usage data, to MCP clients or AI models. The server is configurable for various MCP clients including Claude Desktop and Visual Studio Code, and can also be deployed via Docker or Deno environments. It enables integration of up-to-date, machine-readable browser compatibility information with large language models and development tools.

⭐ 36
MCP
yamanoku/baseline-mcp-server

Jira MCP

AI-Ready Jira Server for Real-World Developer Workflows

Jira MCP is an opinionated server implementation of the Model Context Protocol (MCP) tailored for Jira automation and integration. It provides advanced tools to manage Jira issues, sprints, development data, and project versions, reflecting workflows of real engineering and QA teams. Designed for seamless AI and tool integrations, it supports context-rich retrieval of Jira information for enhancing development transparency and automation.

⭐ 69
MCP
nguyenvanduocit/jira-mcp

ScreenMonitorMCP v2

Real-time screen monitoring and visual analysis for AI assistants via MCP.

ScreenMonitorMCP v2 is a Model Context Protocol (MCP) server enabling AI assistants to capture, analyze, and interact with screen content in real time. It supports instant screenshots, live streaming, advanced vision-based analysis, and provides performance monitoring across Windows, macOS, and Linux. Integration with clients like Claude Desktop is streamlined, offering easy configuration and broad compatibility. The tool leverages AI vision models to provide intelligent insights into screen content and system health.

⭐ 64
MCP
inkbytefo/ScreenMonitorMCP

ZoomEye MCP Server

Real-time cyberspace asset intelligence for AI assistants via Model Context Protocol.

ZoomEye MCP Server implements the Model Context Protocol (MCP) to provide network asset intelligence to AI assistants and development tools. It enables querying of global internet assets through ZoomEye's cyber asset search engine using structured parameters and dorks. The server includes features like caching, error handling, and compatibility with leading MCP environments, supporting real-time cyber asset data integration for various AI and developer platforms.

⭐ 50
MCP
zoomeye-ai/mcp_zoomeye

PDF Tools MCP

Comprehensive PDF manipulation via MCP protocol.

PDF Tools MCP provides an extensive suite of PDF manipulation operations using the Model Context Protocol framework. It supports both local and remote PDF tasks, such as rendering pages, merging, extracting metadata, retrieving text, and combining documents. The tool registers endpoints through the MCP protocol, enabling seamless server-based PDF processing for various clients. Built with Python, it emphasizes secure handling and compatibility with Claude Desktop via the Smithery ecosystem.

⭐ 31
MCP
danielkennedy1/pdf-tools-mcp

View all Alternatives

Didn't find tool you were looking for?

Search AI Tools

WebScraping.AI MCP Server

Key Features

Use Cases

README

WebScraping.AI MCP Server

Features

Installation

Running with npx

Manual Installation

Configuring in Cursor

Running on Claude Desktop

Configuration

Environment Variables

Required

Optional Configuration

Configuration Examples

Available Tools

1. Question Tool (`webscraping_ai_question`)

2. Fields Tool (`webscraping_ai_fields`)

3. HTML Tool (`webscraping_ai_html`)

4. Text Tool (`webscraping_ai_text`)

5. Selected Tool (`webscraping_ai_selected`)

6. Selected Multiple Tool (`webscraping_ai_selected_multiple`)

7. Account Tool (`webscraping_ai_account`)

Common Options for All Tools

Error Handling

Integration with LLMs

Example: Configuring Claude with MCP

Development

Contributing

License

Star History

Repository Owner

Repository Details

Programming Languages

Tags

Related MCPs

Search AI Tools

WebScraping.AI MCP Server

Key Features

Use Cases

README

WebScraping.AI MCP Server

Features

Installation

Running with npx

Manual Installation

Configuring in Cursor

Running on Claude Desktop

Configuration

Environment Variables

Required

Optional Configuration

Configuration Examples

Available Tools

1. Question Tool (webscraping_ai_question)

2. Fields Tool (webscraping_ai_fields)

3. HTML Tool (webscraping_ai_html)

4. Text Tool (webscraping_ai_text)

5. Selected Tool (webscraping_ai_selected)

6. Selected Multiple Tool (webscraping_ai_selected_multiple)

7. Account Tool (webscraping_ai_account)

Common Options for All Tools

Error Handling

Integration with LLMs

Example: Configuring Claude with MCP

Development

Contributing

License

Star History

Repository Owner

Repository Details

Programming Languages

Tags

Related MCPs

1. Question Tool (`webscraping_ai_question`)

2. Fields Tool (`webscraping_ai_fields`)

3. HTML Tool (`webscraping_ai_html`)

4. Text Tool (`webscraping_ai_text`)

5. Selected Tool (`webscraping_ai_selected`)

6. Selected Multiple Tool (`webscraping_ai_selected_multiple`)

7. Account Tool (`webscraping_ai_account`)