pathway.com - Powering your RAG and ETL at scale with Live Data

pathway.com Powering your RAG and ETL at scale with Live Data

What is pathway.com?

Pathway is a robust and high-performance data processing framework designed to build and manage AI/ML applications utilizing live data and real-time pipelines. It facilitates seamless data ingestion from over 300 sources, complete with automatic synchronization.

Pathway supports a range of functionalities, including real-time feature generation, live vector search, and anomaly alerts. The framework allows building applications to derive accurate AI insights from terabytes of connected documents and data tables.

Features

High-performance input/output connectors: Kafka, S3, cloud file systems, databases
Predefined API connectors: 300+ data sources
REST API endpoint: Serving query/answer and realtime features with sub-millisecond latency
Python programming API: Table API
SQL programming API
Incremental stream and table operations: join, filter, group-by, reduce
Advanced join types: temporal joins, windows, and ranges
User Defined Functions: Call external libraries
Async data processing: Call APIs, libraries, LLM services
Data schema support: Python/mypy compatible typing
LLM extension pack: Unstructured data parsing toolkit (data source sync, parsing, extraction, indexing)
Advanced data indexing: Vector with HNSW, BM24, hybrid
Horizontal Scalability: Enterprise Feature

Use Cases

Real-time GPS data analytics
Logistics and automotive IoT analytics
Fraud detection
RAG for sales and marketing
Search in slide decks
Log monitoring
Social media sentiment analysis
High-accuracy RAG
Sharepoint AI search
Delta lake ETL
Monitored instances

FAQs

What is the contract duration for the Department Pilot plan?

The contract duration for the Department Pilot plan can be 30, 60, or 90 days.
Which cloud providers are supported by Pathway RAG pipelines?

Pathway RAG pipelines support AWS, Azure, Google Cloud, and on-premises deployments.
What data sources can be used in department pilot plan?

Supported data sources include Google Drive, Microsoft SharePoint, and the Local File System.