Diggernaut Turn Website Content into Structured Datasets Easily

What is Diggernaut?

Diggernaut provides a robust cloud platform designed for web scraping, data extraction, and various ETL (Extract, Transform, Load) tasks. It enables users to automate the collection of data from websites by creating specialized bots called 'diggers'. These diggers navigate target websites, extract specified information, normalize it, and store it in the cloud as structured datasets.

The platform caters to users with varying technical skills. A Visual Extractor tool offers a point-and-click interface for building digger configurations without coding. For more complex tasks, developers can utilize a dedicated meta-language. Users have the flexibility to run their diggers within the Diggernaut cloud environment or compile them into executable files for local execution on Windows, Linux, or Mac systems. Extracted data can be downloaded in popular formats like CSV, XLS, and JSON, or accessed programmatically via a REST API.

Features

Visual Extractor: Build scrapers using a point-and-click interface without coding.
Meta-language Scripting: Create complex data extraction configurations using a specialized language.
Cloud Execution: Run web scrapers directly on the Diggernaut platform.
Local Execution: Compile diggers into executable files for Windows, Linux, or Mac.
Multiple Output Formats: Download extracted data as CSV, XLS, or JSON.
REST API Access: Retrieve scraped data programmatically.
OCR Module: Extract text directly from images on web pages (cloud execution only).
Integrations: Connects with services like Zapier, Tableau, CAPTCHA solvers, and proxy providers.

Use Cases

Extracting product prices, reviews, and ratings from e-commerce sites.
Aggregating event information from various online sources.
Collecting news headlines and articles from media websites.
Gathering statistical data from diverse web sources.
Monitoring government data, reports, licenses, and permits.
Scraping opinions and comments from forums or social media.
Extracting real estate property details including recorder, assessor, or tax information.

FAQs

Why would I need Diggernaut?

Diggernaut significantly speeds up data collection from websites, saving hours or days compared to manual methods, allowing you to focus on more important tasks.
Can one digger scrape multiple sites or pages?

Yes, a single digger can navigate multiple pages across one or more websites to gather data, provided the target data structures are consistent.
Can I run diggers on my own server without being charged?

Yes, you can compile your digger into an executable file for Windows, Linux, or Mac and run it on your own systems free of charge for as long and as often as needed.
What technology powers the diggers?

Diggers use Golang as their backend, which allows for compilation into fast and efficient executable files compared to interpreted languages.
Can Diggernaut extract text from images?

Yes, Diggernaut has an OCR module to extract text from images. This feature is available only for diggers running in the Diggernaut cloud, not for compiled local diggers.