Build software better, together

pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl 8000 machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

Updated Nov 10, 2026
Python

elastic / logstash

Star

Logstash - transport and process your logs, events, or other data

java streaming jruby logging real-time-processing etl-framework

Updated Nov 10, 2026
Java

cloudquery / cloudquery

Star

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Updated Nov 10, 2026
Go

noflo / noflo

Star

Flow-based programming for JavaScript

fbp noflo flow-based-programming etl-framework nocode

Updated Nov 8, 2026
JavaScript

apache / hamilton

Star

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

python data-science machine-learning etl pandas orchestration data-engineering data-analysis software-engineering feature-engineering dataframe hacktoberfest dag lineage etl-framework etl-pipeline rag mlops llmops

Updated Nov 9, 2026
Jupyter Notebook

san089 / goodreads_etl_pipeline

Star

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Updated Mar 9, 2020
Python

singer-io / getting-started

Star

This repository is a getting started guide to Singer.

python etl data-analysis singer etl-framework

Updated Aug 8, 2025
Makefile

marsupialtail / quokka

Star

Making data lake work for time series

sql distributed etl-framework data-lake-analytics mlops

Updated Aug 21, 2024
Python

apache / seatunnel-web

Star

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

real-time offline high-performance apache data-integration sql-engine data-pipeline etl-framework seatunnel

Updated Jan 22, 2026
Java

stitchfix / hamilton

Star

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

python data-science machine-learning etl numpy pandas data-engineering data-platform software-engineering feature-engineering dataframe dag hamiltonian etl-framework hamilton featurization etl-pipeline stitch-fix

Updated Jul 3, 2023
Python

Cinchoo / ChoETL

Star

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

yaml parser json csv csharp avro etl dotnet xml writer flat reader parquet keyvalue parquet-files etl-framework cinchoo-etl

Updated Oct 19, 2026
C#

YotpoLtd / metorikku

Star

A simplified, lightweight ETL Framework based on Apache Spark

scala sql big-data spark etl distributed-computing keep etl-framework etl-pipeline

Updated Sep 1, 2026
Scala

seanharr11 / etlalchemy

Star

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

python sqlalchemy database etl migrations etl-framework

Updated May 23, 2019
Python

usc-isi-i2 / kgtk

Star

Knowledge Graph Toolkit

toolkit rdf graphs wikidata embeddings kg etl-framework knowledge-graphs

Updated Jul 17, 2026
Jupyter Notebook

marciok / gust

Star

A Task-Based Workflow Orchestration Engine

machine-learning airflow elixir elixir-phoenix workflow-automation etl-framework etl-pipeline

Updated Nov 10, 2026
Elixir

quintoandar / butterfree

Star

A tool for building feature stores.

python package data-science etl pyspark data-engineering etl-framework feature-store

Updated Oct 22, 2026
Python

data-dot-all / dataall

Star

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.

aws data-science data aws-s3 redshift etl-framework aws-glue aws-lake-formation lakehouse lakeformation

Updated Jun 21, 2026
Python

velocitybolt / open-extract

Star

Structured Data Extractor for AI Agents. Search your documents or the web for specific data and get it back in JSON or Markdown in a single tool call.

python ai etl openai context-aware etl-framework autogen unstructured-data rag etl-automation llm langchain crewai langgraph agent-tools structured-outputs context-aware-structured-outputs

Updated Jul 14, 2026
Python

Nextdoor / bender

Star

Bender - Serverless ETL Framework

java aws-lambda etl aws-s3 aws-kinesis etl-framework aws-firehose

Updated Dec 19, 2023
Java

wp-labs / warp-parse

Star

Focusing on building industry-leading ETL engines.

events streaming pipeline etl metrics logging telemetry data-collector siem wpl observability etl-framework dataprocessing etl-pipeline streaming-processing oml etl-process wp-parse warp-parse

Updated Nov 8, 2026
Rust

Navigation Menu

etl-framework

Here are 308 public repositories matching this topic...

pathwaycom / pathway

elastic / logstash

cloudquery / cloudquery

noflo / noflo

apache / hamilton

san089 / goodreads_etl_pipeline

singer-io / getting-started

marsupialtail / quokka

apache / seatunnel-web

stitchfix / hamilton

Cinchoo / ChoETL

YotpoLtd / metorikku

seanharr11 / etlalchemy

usc-isi-i2 / kgtk

marciok / gust

quintoandar / butterfree

data-dot-all / dataall

velocitybolt / open-extract

Nextdoor / bender

wp-labs / warp-parse

Add this topic to your repo