What is Crawlspace?
Crawlspace provides a comprehensive platform designed for developers who need to build and deploy web crawlers efficiently. It offers horizontally-scaling architecture, allowing users to crawl millions of pages cost-effectively. The platform integrates Large Language Models (LLMs) and query selectors to extract structured JSON data according to custom schemas, ensuring data accuracy and relevance for applications and AI agents.
Emphasis is placed on responsible crawling practices, with built-in adherence to robots.txt files and default rate-limiting. Crawlspace features a platform-wide TTL cache to minimize redundant traffic to origin websites. Integrated storage options include SQLite for structured data, an S3-compatible bucket for unstructured data, and a vector database for semantic data, simplifying the data management pipeline for developers.
Features
- Scalable Crawling: Affordably crawl millions of pages with horizontally-scaling architecture.
- AI-Powered Scraping: Utilize LLMs or query selectors to extract JSON data conforming to custom schemas.
- Compliance Built-in: Automatically respects robots.txt and applies rate-limiting.
- Platform-Wide Cache: Reduces redundant traffic by pulling from a shared TTL cache.
- Integrated Storage: Includes SQLite, S3-compatible bucket, and vector database per crawler.
- Serverless Deployment: Deploy crawlers without managing infrastructure.
- TypeScript-First: Write type-safe crawler code with npm package support.
- JavaScript Rendering: Capable of rendering single-page applications (SPAs).
- Scheduling: Set crawlers to run on daily, hourly, or minute-by-minute schedules (depending on plan).
- Secrets Management: Securely use credentials for crawling pages behind authentication.
Use Cases
- Find prospective customers by scraping websites for specific technology usage.
- Implement Retrieval-Augmented Generation (RAG) by crawling and embedding documentation.
- Conduct market research by pulling live data like pricing from across the web.
- Monitor community growth by tracking metrics like Discord server members.
- Build image databases by saving image files and associated metadata (e.g., alt text).
- Replicate third-party API data to bypass rate limits or track changes.
FAQs
-
How does Crawlspace shield websites from redundant bot traffic?
Crawlspace uses a platform-wide TTL cache for unauthenticated requests. Duplicate requests within a time window are served from the cache, preventing traffic to the origin site. Additionally, crawlers default to respecting robots.txt. -
Should I use CommonCrawl instead of Crawlspace?
Use CommonCrawl if you need data that can be up to a month old, are comfortable parsing terabytes in WARC/WAT/WET formats, and are okay with data location constraints or transfer costs from us-east-1. -
Can Crawlspace crawl social media websites?
Typically no, as sites like LinkedIn or X forbid crawling in their robots.txt. Data enrichment platforms are recommended for social media data. -
Can Crawlspace bypass CAPTCHA or anti-bot challenges?
No, Crawlspace, similar to GoogleBot, does not solve CAPTCHA challenges. -
Can I use AI models from providers like OpenAI and Anthropic?
Yes, you can use third-party AI models by placing your API tokens in the crawler's .env file and accessing them via the provided 'env' object in your code.
