Repository navigation
Home
DAD ingests syslog, indexes every word, and gives you a web UI for searching the corpus, alerting on patterns, and running scheduled maintenance against it.
Status (2026-11). Rails 8.1.4.1 on Ruby 3.3.8, PostgreSQL in production, MIT
licensed. The 2026 review-and-hardening pass is complete: the two eval sinks
that let a Job row or a display filter run arbitrary Ruby are gone, SQL is
parameterized throughout, events/events_words are range-partitioned by month
with archive-before-drop retention, the Rails 4.2 → 8.1.4 upgrade landed, and CI
runs the whole suite on PostgreSQL with three blocking security gates, and the
dependency audit is clean. Windows Event Log collection now polls hosts directly
over WinRM rather than requiring OpenWEC; see Windows Events.
A follow-up pass closed four defects that had survived that one, each reproduced
by a failing test first: a blank job Frequency could crash-loop the scheduler so
no job ran, two accounts could share a username (the second could never sign
in), the per-request session lookup scanned the whole sessions table, and every
management index rendered an unbounded table.
A further ingest pass closed two more, again each reproduced by failing tests
first: a production-sized batch built an INSERT over the database's
bind-parameter limit and the flush failed outright, and the importer's word-cache
lifetime was never initialised, so the first cache prune on a long ingest killed
the process. See Architecture.
The same line of work then profiled ingest end to end and found the bottleneck was
not batching but the words table: every token missing from the word cache was
resolved against the database one at a time. Resolving them in bulk per batch cut
a 20,000-line run from 186,089 statements to 360 and made ingest 6.4× faster
on a cold words table, 2.36× faster at steady state — writing exactly the
same index rows, so search is untouched. See Architecture.
The project now also runs as a Docker Compose stack — the application in its own
containers, PostgreSQL in a separate one — and bin/docker-test runs the whole
suite against that PostgreSQL. PostgreSQL is now the only engine the app runs
anywhere: SQLite has been removed from development and test as well, because the
split between engines is what let a query that worked in the suite fail outright
in production. The Dockerfile that shipped before it could not build at all: a
bare gem install line is not a Dockerfile instruction, and the image had no
JavaScript runtime. See Development and Installation.
The most recent pass added a second log source: DAD ingests Windows Event Log events. They are converted into the record format the ingestor already accepts rather than the ingestor being extended to understand Windows events, so word indexing, batch chunking, crash checkpointing and partitioning apply to them unchanged and the importer itself was not touched.
DAD polls the hosts itself over WinRM by default, which needs no extra process
— the poller runs as a Job inside dad-scheduler, and each host's interval
adapts to how many events it is producing. Collecting through
OpenWEC remains available for hosts that
push, and runs as an optional fourth unit, dad-windows. Both feed the same
LogsToProcess queue. See Windows Events.
A final pass went to the browser-facing layer, which had never been tested: no test
rendered the application layout at all. That suite now exists (rake test:system,
40 runs), and it found real defects — the app emitted 209 duplicate id attributes
across 17 pages, and complete: $.ajax() in three places fired a stray request
for the page you were already on every time it ran, including on the dashboard's
60-second poll. Both are fixed and guarded. The stock scaffold stylesheet was merged
into application.css, the 14 copy-pasted form error summaries became one partial,
and the layout gained a viewport tag so the nav bar and alert queue are usable on a
phone. See Interface and Development.
See What Changed Since 2016 before using any older page here.
syslog/UDP:514 ───► syslog_collector (dad-syslog) writes Logs/LogStaging,
rotates it on a 60s boundary
(only when traffic arrives)
│
import_syslog.rb (dad-importer) claims each file, feeds
│ EventIngestor.storeEvent,
│ moves it to ProcessedLogs
PostgreSQL ── events + words + events_words
│
Rails app (Passenger/Apache) search, alerts, dashboards
scheduler.rb (dad-scheduler) runs due Job rows
Three processes, three systemd units, one database — plus an optional fourth,
dad-windows, if you collect Windows events through OpenWEC. Polling Windows hosts
directly adds no process; it runs inside dad-scheduler. Details in
Architecture.
| I want to… | Read |
|---|---|
| see what the UI looks like | Interface |
| understand the system | Architecture |
| ingest Windows events | Windows Events |
| install or upgrade it | Installation |
| run it locally in Docker | Installation or Development |
| write a display filter | Displays |
| understand why a search missed | Search |
| schedule or debug a job | Jobs |
| keep history without filling the disk | Retention and Archives |
| give someone limited access | Access Control |
| work on the code | Development |
Example Jobs collects the alerting recipes written in 2015 — host reachability, sudo activity, event-rate anomalies, disk space, outbound email, and user-activity correlation. They are a recipe archive, not instructions: they were written when a Job held arbitrary Ruby, which DAD no longer accepts. See Jobs for what replaced that design and how to contribute the equivalent as a registered handler.