Blog
Tutorials, changelogs and engineering notes from the CocoIndex team: incremental processing, the Rust engine, and pipelines in production.
How Tree-sitter Works: Parsing Code as You TypeNew
Grammars, concrete syntax trees, GLR parsing, error recovery and queries, then how CocoIndex splits code into chunks on function and class boundaries.
- Updated
Dry-run previews, per-call dependencies and bounded batchingNew
CocoIndex 1.0.26 and 1.1.0 add a preview that stores no state, per-call logic dependencies, bounded batching and component concurrency.
- Updated
Timeouts, batch retries, and target inspection in CocoIndex
CocoIndex 1.0.17–1.0.25 adds timeouts, batch retries, target inspection, faster target updates, and a Linux hang fix.
- Updated
Persistent state, rate limits, Snowflake and BigQuery
CocoIndex 1.0.8–1.0.16 adds per-component state, LiveMap, a shared rate limiter, batched target writes, and Snowflake, BigQuery and Azure Blob connectors.
- Updated
Codebase Indexing for AI Agents with CocoIndex
Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
- Updated
Memoization keys, scheduled refresh and scoped stats
CocoIndex 1.0.1–1.0.7 adds per-argument memo keys, scheduled live refresh, scoped stats, OCI and Iggy sources, and Neo4j, FalkorDB and Turbopuffer targets.
- Updated
Stream CSV changes to Kafka: publish only changed rows
Watch a folder of CSV files and publish each changed row as JSON to a Kafka topic with CocoIndex, with no producer loop.
CocoIndex V1 is Live!
CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
- Updated
Build a Knowledge Graph from YouTube Podcasts with an LLM
Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
- Updated
A Safe pickle Alternative: Serialize Python with Type Hints
Why CocoIndex replaced pickle with serialization that reads Python type hints to choose msgspec, Pydantic or restricted pickle for each value.
- Updated
Build a Python Daemon for a CLI: Auto-Start, Upgrade, Stop
Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
- Updated
Five new targets and Python 3.14 free-threading in CocoIndex
CocoIndex 0.3.27–0.3.34 adds Apache Doris, ChromaDB, FalkorDB, Ladybug and Pinecone targets, LocalFile change notifications and Python 3.14 free-threading.
CocoIndex joins the GitHub Secure Open Source Fund
CocoIndex joined the GitHub Secure Open Source Fund, hardening the AI data infrastructure developers depend on with threat modeling, CodeQL, and audits.
- Updated
Hybrid search over SEC EDGAR filings with Apache Doris
Index SEC 10-K text and XBRL facts into one Apache Doris table with CocoIndex v1, then rank each chunk by meaning and by keywords, fused with RRF.
- Updated
Generate Codebase Documentation with an LLM, Kept Current
Generate a Markdown wiki page for every Python project in a folder with an LLM and CocoIndex v1. On each run, the LLM re-reads only the files that changed.
- Updated
Turn PDF Slides into Narrated Audio with Gemini and TTS
Build a CocoIndex v1 pipeline that turns PDF slides into vision-generated speaker notes, local Pocket TTS narration, and searchable LanceDB records.
- Updated
Structured errors and Azure OpenAI support in CocoIndex
CocoIndex 0.3.11–0.3.26 adds structured errors with Python stack traces, warnings for slow live updates, Azure OpenAI and OpenRouter embeddings.
- Updated
Extract Patient Intake Forms with DSPy and CocoIndex
Extract Pydantic-typed patient records from PDF intake forms with DSPy and Gemini vision into Postgres with pgvector, and see which edits re-run extraction.
- Updated
Build a Knowledge Graph from Meeting Notes with Neo4j
Build a self-updating Neo4j knowledge graph from meeting notes with CocoIndex v1: LLM-extracted decisions, tasks, and owners, with embedding-based dedupe.
- Updated
Normalize and rank Hacker News topics with an LLM
Normalize free-text topics with an LLM prompt and rank them with a weighted SQL score in Postgres, in an incremental CocoIndex v1 pipeline over Hacker News.
CocoIndex Changelog 0.2.21 - 0.3.10
Featuring batching support for CocoIndex functions, execution robustness, schema & type system improvements, custom source support, and more.
- Updated
Turn a polling API into an incremental source: Hacker News
Turn a polling API into an incremental CocoIndex v1 source in plain Python, using Hacker News: list keys, fetch by key, own rows, survive failed fetches.
- Updated
Extracting Intake Forms with BAML and CocoIndex
Use BAML and CocoIndex v1 to extract typed patient records from PDF intake forms into Postgres with pgvector, and avoid the role("user") PDF gotcha on OpenAI.
- Updated
Adaptive Batching - 5x throughput on your data pipelines
CocoIndex batches concurrent calls to embedders and your own functions, with no timer to tune. Batching made the code_embedding example about 5x faster on a small embedding model.
- Updated
AI-native data pipelines: why we built CocoIndex
Why AI applications need open, AI-native data pipelines that keep context fresh as sources and code change, and the design choices behind CocoIndex.
- Updated
Multimodal PDF Search: Embed Text and Images with CLIP
Split PDFs into text chunks and images, embed text with SentenceTransformers and images with CLIP, and search both in Qdrant with the page number on every hit.
- Updated
Write a Custom Source in Python: Index Any API
Index an internal API with CocoIndex v1: plain async Python, deletes propagate, and a memo on updated_at skips unchanged items.
CocoIndex Changelog 2025-10-19
Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
- Updated
Extract Invoice Data from PDFs into Snowflake with an LLM
Extract supplier invoices from PDFs in Azure Blob Storage into a typed Snowflake table with an LLM and CocoIndex, processing only new or changed files.
- Updated
Thinking in Rust: Ownership, Access, and Memory Safety
A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
- Updated
Trace search results back to source data
Define query handlers in CocoIndex and trace search results back to source data in CocoInsight to close the loop on indexing strategy.
- Updated
Add Semantic Search to a Postgres Table with pgvector
Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.
- Updated
Search PDFs and Images Without OCR Using ColPali and Qdrant
Index PDFs and images into one Qdrant collection with CocoIndex v1 and ColPali multi-vector embeddings. Each page is embedded as an image, so no OCR is needed.
CocoIndex Changelog 2025-08-18
CocoIndex updates: production readiness, scalability, and reliability, plus more customization, native integrations, and multi-modal pipeline features.
- Updated
Control Processing Concurrency in CocoIndex
How CocoIndex v1 caps concurrent processing components with one app-level limit, lends slots to nested mounts, and adds rate and batch bounds.
- Updated
Multi-vector image search with ColPali and Qdrant
Index images with ColPali multi-vector patch embeddings and a Qdrant MaxSim collection using CocoIndex v1: incremental, live, in plain async Python.
- Updated
Multi-Dimensional Vector Support in CocoIndex
CocoIndex v1 describes vector and multi-vector columns with VectorSchema and MultiVectorSchema, so Qdrant gets dense vectors and MaxSim multi-vectors.
- Updated
Write a Custom Target: Export Indexed Data Anywhere
Export CocoIndex data anywhere: implement a TargetHandler with a tracking record and an action sink; the engine handles diffing, syncing, and cleanup.
- Updated
Search your photos by face with CocoIndex
Detect every face in a photo folder, embed each with dlib, and index one Qdrant point per face with CocoIndex v1, so a face finds the photos it appears in.
- Updated
Extract Metadata from Research Paper PDFs with an LLM
Index academic papers with CocoIndex: read the first page of each PDF, extract title, authors, and abstract with an LLM, and embed them for semantic search.
CocoIndex Changelog 2025-07-07
CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.
- Updated
Introducing CocoInsight
CocoInsight, the CocoIndex v0 tool for inspecting pipeline data and lineage step by step. It is not part of CocoIndex v1; v1 apps use cocoindex show.
- Updated
Schema inference for Qdrant collections
CocoIndex v1 creates your Qdrant collection from the code: the vector size comes from the embedding model, and a schema change rebuilds the collection.
CocoIndex + Kuzu: Real-time knowledge graph with Kuzu
Build a real-time knowledge graph with Kuzu as a native CocoIndex target: incremental updates, high-performance graph queries.
CocoIndex Changelog 2025-05-31
CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.
- Updated
Incremental ETL from Amazon S3 to Postgres pgvector
Index Markdown from Amazon S3 into Postgres pgvector with CocoIndex v1. A re-run reads only the objects that changed and embeds only the chunks with new text.
- Updated
Build Text-to-Image Search with CLIP and Qdrant
Build live image search with CocoIndex v1, CLIP, FastAPI, and Qdrant. Embed photos and text in one vector space and search by meaning.
- Updated
What Are Text Embeddings? Build Semantic Search in Python
Text embeddings 101: what an embedding is, how similarity works, why you chunk, and how to build a semantic search index with CocoIndex v1 and pgvector.
Story of CocoIndex, at 1k stars
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
- Updated
Build a Product Recommendation Engine with an LLM and Neo4j
An LLM extracts each product's taxonomy and complementary taxonomy; CocoIndex turns the labels into a Neo4j knowledge graph that answers 'bought this, also need…'.
CocoIndex Changelog 2025-04-30
CocoIndex updates: knowledge graph support, Qdrant and Supabase targets, KTable and LTable data types, additional LLM providers, and more.
- Updated
Build a Knowledge Graph from Documents with an LLM and Neo4j
Build a Neo4j knowledge graph from Markdown docs with an LLM and CocoIndex v1. An edit re-extracts only that file, and unsupported nodes and edges are deleted.
CocoIndex Changelog 2025-04-07
CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.
- Updated
Keep an AI agent's context in sync with live mode
Run a CocoIndex app in live mode so an agent's index follows every edit, add and delete as it happens, and re-embeds only what changed. Updated for v1.
- Updated
What Is Incremental Processing? Keep Indexes Fresh
Incremental processing re-runs only the work a change affects. See how CocoIndex v1 finds changes, reuses memoized results and deletes stale rows.
- Updated
Extract Patient Intake Forms from PDF and DOCX with an LLM
Extract typed Patient records from PDF and DOCX intake forms with an LLM and CocoIndex v1, and compare the plain, BAML, DSPy and local-model approaches.
- Updated
Semantic Search over Google Drive Files with pgvector
Index the documents in a shared Google Drive folder as text embeddings in Postgres with CocoIndex v1, then search them by meaning instead of by filename.
CocoIndex Changelog 2025-03-20
First release of CocoIndex Changelog: LLM support, codebase indexing, custom functions, and assorted core/performance improvements
Build Real-Time Codebase Indexing for AI Code Generation
Indexing codebase for RAG with CocoIndex and Tree-sitter in real-time: chunking, embedding, semantic search, and build vector index for efficient retrieval.
- Updated
Extract Structured Data from PDFs Locally with Ollama
Turn PDFs into typed Postgres rows on your own hardware: docling converts each PDF to Markdown, a local Ollama model fills a nested schema.
CocoIndex is now open source
CocoIndex is now open source: an engine that combines custom transformation logic with incremental processing for data indexing.
- Updated
Customizable Data Indexing Pipelines
What customizable data indexing pipelines are and why custom transformation logic matters, with practical CocoIndex examples.
- Updated
How indexing pipelines differ from other data pipelines
How indexing pipelines differ from ETL, streaming and transactional systems: an index is derived data that must stay correct as its source changes.
- Updated
How CocoIndex handles data and code changes
When a file, a function, a prompt or a table schema changes, CocoIndex re-runs only the work that depends on it. See what each change costs.
- Updated
Processing Large Files in Indexing Pipelines
A USPTO weekly XML file holds thousands of patents. Size CocoIndex processing components so rows land early and restarts resume, measured on Postgres.
Data Consistency in Indexing Pipelines
How a crash or two racing versions leave stale rows in a RAG index, and how CocoIndex v1 keeps its internal store and your vector index consistent.
- Updated
Data Indexing Pipelines for RAG: Common Challenges
What a good RAG indexing pipeline needs, where pipelines break (incremental updates, upgrades, stale rows from LLM extraction), and how CocoIndex handles it.
- Updated
CocoIndex: a data indexing platform for AI applications
CocoIndex keeps AI-ready data in sync with its sources: connectors in, plain Python transforms, and incremental writes to vector, SQL and graph targets.
- Updated
Welcome to CocoIndex
CocoIndex keeps derived data (chunks, embeddings, graphs, summaries) in sync with its sources and code, so AI agents read current context.