MCPs tagged with # web crawling

Driflyte MCP Server acts as a bridge between AI-powered assistants and diverse, topic-aware content sources by exposing a Model Context Protocol (MCP) server. It enables retrieval-augmented generation workflows by crawling, indexing, and serving topic-specific documents from web pages and GitHub repositories. The system is extensible, with planned support for additional knowledge sources, and is designed for easy integration with popular AI tools such as ChatGPT, Claude, and VS Code.

• ⭐ 9
• MCP
• serkan-ozal/driflyte-mcp-server

mcp-server-webcrawl provides an AI-oriented server that enables advanced filtering, analysis, and search over data from various web crawlers. Designed for seamless integration with large language models, it supports boolean search, filtering by resource types and HTTP status, and is compatible with popular crawling formats. It facilitates AI clients, such as Claude Desktop, with prompt routines and customizable workflows, making it easy to manage, query, and analyze archived web content. The tool supports integration with multiple crawler outputs and offers templates for automated routines.

• ⭐ 32
• MCP
• pragmar/mcp-server-webcrawl

Search1API MCP Server is an implementation of the Model Context Protocol (MCP) that provides search and crawl services using the Search1API. It allows seamless integration with MCP-compatible clients, including LibreChat and various developer tools, by managing API key configuration through multiple methods. Built with Node.js, it supports both standalone operation and Docker-based deployment for integration in broader AI toolchains.

• ⭐ 157
• MCP
• fatwang2/search1api-mcp

Graphlit MCP Server provides a Model Context Protocol interface, enabling seamless integration between MCP clients and the Graphlit platform. It supports ingestion from a wide array of sources such as Slack, Discord, Google Drive, email, Jira, and GitHub, turning them into a searchable, RAG-ready knowledge base. Built-in tools allow for document, media extraction, web crawling, and web search, as well as advanced retrieval and publishing functionalities. The server facilitates easy configuration, sophisticated data operations, and automated notifications for diverse workflows.

• ⭐ 369
• MCP
• graphlit/graphlit-mcp-server