Build software better, together

firecrawl / firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping

Updated Nov 10, 2026
TypeScript

ScrapeGraphAI / Scrapegraph-ai

Sponsor

Star

Python scraper based on AI

Updated Oct 6, 2026
Python

oxylabs / ai-crawler-py

Star

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

ai web-crawler ai-agents ai-crawler ai-studio ai-scraping ai-web-crawler crawl-agent

Updated Aug 21, 2026

vakra-dev / reader

Star

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

Updated Oct 6, 2026
TypeScript

BrowserCash / teracrawl

Star

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

html-to-markdown web-scraper web-scraping data-extraction browser-automation ai-agents web-search google-serp ai-search serpapi browser-agent ai-crawler antibot-bypass ai-scraping crawl4ai firecrawl-mcp firecrawl-api firecrawl-alternative

Updated Dec 3, 2025
TypeScript

puneetindersingh / open-seo-crawler

Star

Free SEO crawler, website audit and AI crawler checker. Self-hosted and open source: crawls a site for technical SEO issues and checks that Googlebot, Bingbot and AI crawlers (GPTBot, PerplexityBot, OAI-SearchBot) get the same readable page. CMS-aware, XLSX export, free Screaming Frog alternative.

Updated Nov 10, 2026
Python

lennyerik / crawl4ai-proxy

Star

A simple proxy server to integrate crawl4ai with OpenWebUI

adapter crawler machine-learning ai proxy web-crawler inference gpt llm ai-crawler llm-ui openwebui llm-crawler crawl4ai

Updated Nov 20, 2025
Go

makralabs / makra

Star

Makra is a memory layer between the web and your AI agents. It serves real-time structured data from the open web at the cost of a vector search.

crawler scraper ai web-crawler web-scraper web-scraping ai-agents vector-search vector-database ai-tools web-scraping-python ai-crawler web-scraping-api ai-memory ai-scraper web-memoization web-memoization-engine

Updated Aug 29, 2026
Python

agentmarkup / agentmarkup

Star

Build-time llms.txt, JSON-LD, markdown mirrors, AI crawler controls, and validation for Vite, Astro, Next.js, Nuxt, and any static site.

nextjs seo geo nuxt schema-org astro robots-txt json-ld structured-data machine-readable vite-plugin ai-crawler llms-txt

Updated Nov 10, 2026
TypeScript

webscrape-ai / webscrape-ai

Star

The web scraping API that turns any URL into clean data.

markdown crawler scraper ai html-to-markdown scraping web-scraper web-scraping data-extraction webscraping web-data-extraction llm ai-crawler ai-scraping

Updated Jul 7, 2026
Java

kr3t3n / smolagents-video-script-generator

Star

A sophisticated system that uses multiple AI agents to research, create, and polish video scripts for social media platforms. The system employs specialized agents for research, script writing, polishing, and evaluation to ensure high-quality, engaging content.

ai-agents ai-crawler ai-writer smolagents

Updated Jan 2, 2025
Python

spidra-io / spidra-js

Star

The official Node.js SDK for Spidra.

markdown crawler ai scraping web-scraper web-scraping data-extraction webscraping web-data-extraction scraping-api web-data llm ai-crawler ai-scraping

Updated Oct 25, 2026
TypeScript

Saqierma / miaowageo

Star

免费开源的 GEO 检测工具 | Free open-source GEO (Generative Engine Optimization) checker — 检测网站能否被 ChatGPT、Claude、Perplexity、Gemini 抓取、理解与引用

seo geo robots-txt ai-search perplexity seo-audit ai-crawler gptbot ai-seo llms-txt generative-engine-optimization claudebot

Updated Sep 21, 2026
JavaScript

motiv8-team / go-ua-parser

Star

High-performance, zero-allocation HTTP User-Agent parser for Go — browser, OS, device, bot & AI crawler detection with Client Hints support

go golang user-agent user-agent-parser prometheus browser-detection http-middleware client-hints bot-detection opentelemetry ai-crawler

Updated Nov 3, 2026
Go

JerryZhi / AI-Crawler-Detector

Star

Tool for Fast Detection of Website/Server AI Crawler Blocking Policies（Not robots.txt）

seo geo robots ai-crawler ai-seo-tool

Updated Aug 20, 2025
Python

maango-io / ai-policy.json

Star

Machine-readable AI permissions for websites. A consolidated spec at /.well-known/ai-policy.json for declaring how AI agents may train on, search, or use your content.