8000 Development · dhoelzer/DAD Wiki · GitHub
[go: up one dir, main page]
Skip to content

Development

Jump to bottom Edit New page
openhands edited this page Nov 11, 2026 · 14 revisions

Running the suite

bin/docker-test                # all three suites on PostgreSQL, in Docker — preferred
bundle exec rake test          # needs a PostgreSQL cluster and DATABASE_URL
bundle exec cucumber           # needs a PostgreSQL cluster and DATABASE_URL
bundle exec brakeman -q        # static analysis, must report 0 warnings
bundle exec bundle-audit check # dependency CVEs

bin/docker-test needs only Docker. It builds the image, runs minitest and cucumber against the Compose db service, and then repeats the two assertions the CI test job makes: that events and events_words really are partitioned, and that no index was left INVALID by a failed CONCURRENTLY build. Extra arguments replace the container command rather than appending to it, so bin/docker-test bundle exec rails test test/models/event_test.rb runs one file and bin/docker-test bash opens a shell inside it.

PostgreSQL is the only engine, in every environment. SQLite was dropped as the development and test engine: it cannot partition events/events_words, so a SQLite run never exercised the shape production queries, and the adapter-conditional skips it caused are what made a green suite meaningless. The suite now reports 0 skips — 542 unit/controller runs plus 40 system runs, 1883 assertions in total — and a green run means the partitioned path actually ran.

Three suites, not one

command covers
bundle exec rake test models, ingest, controllers
bundle exec rake test:system rendered HTML — the layout, nav bar, partials, asset tags
bundle exec cucumber the ingest and search feature files

rake test excludes test/system by design: railties sets DEFAULT_TEST_EXCLUDE to test/{system,dummy,fixtures}/**/*_test.rb. So the browser-level suite needs its own task, or it never runs — which is how a whole class of view breakage reached master. bin/docker-test and CI both run all three.

The system tests use rack-test, not a real browser, and that is deliberate. A real browser is not viable here: ApplicationController#ssl forces http → https outside development, and there is no matching chromium-driver for the installed Chromium, so Selenium fails at session creation. The https requirement is satisfied the way the controller tests already satisfy it, with HTTPS=on on the request env.

The consequence matters when you write one: rack-test runs no JavaScript. Assert on the static shell the server rendered, never on AJAX-populated content — the dashboard's #events and #stats divs are empty here. Where the interesting behaviour is a script, render the script and assert on its text: test/controllers/dashboard_render_test.rb renders /events.js and pipes it through node --check.

Use the cucumber binary, not rake cucumber:ok — the rake task runs cucumber in-process and minitest's autorun then dies on cucumber's --profile flag.

Two support files carry the machinery the suite needs, and both exist because the simple approach does not work:

  • test/support/scratch_database.rb creates a throwaway database per test for the ingest tests that run the importer in a subprocess. They cannot share the test database: transactional fixtures hold an open transaction, so the child's committed writes are invisible to the parent and vice versa. It opens its own raw PG connection rather than going through ActiveRecord, because CREATE DATABASE fails with PG::ActiveSqlTransaction inside a transaction. The role behind DATABASE_URL therefore needs CREATEDB; bin/docker-test and CI run as the cluster superuser.
  • test/support/query_plan.rb gives tests explain(sql, binds). It sets enable_seqscan = off deliberately: on a test-sized table the planner is entitled to prefer a sequential scan, so asserting its cost choice asserts a guess. Turning scans off asks the question the finding was about — can this index serve this query shape at all?

Running the whole stack in Docker

docker compose up --build     # migrate, then web + importer + scheduler + syslog

Browse to http://localhost:3000 and sign in as admin / dad_dev_password (override with DAD_ADMIN_PASSWORD=... docker compose up). Without a seeded account the stack comes up with zero users: the login form renders fine but can never succeed, because db/seeds.rb raises rather than inventing a well-known account. The migrate service seeds only when the users table is empty — seeding on every start would re-create the maintenance Jobs an operator deleted.

One image, one container per long-lived process — db (Postgres 17), migrate (one-shot), web, importer, scheduler, syslog — mirroring the three systemd units rather than supervising all three in one container, so the container topology is the topology production runs.

docker compose logs -f importer
docker compose exec db psql -U dad dad_development
docker compose down -v        # discard the database too

Four things that will otherwise cost you an afternoon:

  • The image WORKDIR is /var/DAD, not /app. The collector hardcodes DEFAULT_STAGING="/var/DAD/Logs/LogStaging", while import_syslog.rb uses paths relative to its own working directory (Dir.chdir('Logs/LogsToProcess'), then PROCESSED_DIR = '../ProcessedLogs'). The two agree only because every systemd unit sets WorkingDirectory=/var/DAD. Nothing validates that they line up: pick any other working directory and the collector keeps accepting UDP packets and writes into a tree the importer never reads — silently, with no error anywhere.
  • Rotation is traffic-driven. ensure_open is called from the per-message path, so with no inbound traffic the open staging file is never moved into LogsToProcess and the importer sees nothing — a quiet system looks like a broken importer. A single probe packet sits in LogStaging until a second packet arrives after the 60s boundary, which is why the docker job sends two. The collector flushes each record as it writes, so the staging file is readable immediately rather than only after rotation.
  • 5432 is not published by default. The app containers reach db over the Compose network by service name; publishing the port collides with a Postgres already running on your machine and docker compose up fails with "address already in use" before any container starts. For host access use the opt-in override, bound to loopback because the credentials are dad/dad: docker compose -f docker-compose.yml -f docker-compose.db-port.yml up
  • Code is baked into the image, not bind-mounted. The image runs as uid 1000; mounting a checkout owned by a different uid makes Rails' log/ and tmp/ writes fail with EACCES in a way that reads like an app bug. The gem layer is cached, so after an edit docker compose up --build only re-runs COPY . ..

web runs with RAILS_ENV=development, not production: production sets public_file_server.enabled = false and assets.compile = false because Apache serves assets, and there is no Apache in the stack. A production-mode container boots fine and 404s every s 8000 tylesheet and script.

The migrate service uses db:migrate_from_scratch on an empty database and plain db:migrate afterwards. Plain db:migrate on an empty Postgres database loads the committed portable db/schema.rb — which cannot express PARTITION BY — and skips db/migrate entirely, so the partitioned shape is never built. And without a schema at all, the importer crash-loops on relation "statistics" does not exist.

A CI job now builds this image. It did not used to, which is how a Dockerfile.dev with a bare gem install bundler line — not a Dockerfile instruction, so the image could never build — sat on master unnoticed. The docker job in .github/workflows/ci.yml builds it, boots the stack, and then drives it: it asserts the migrate service left a partitioned events/events_words and at least one user, that the login page actually renders, that the app signs in and loads /events, that an asset path the page emitted returns 200, and that a UDP packet sent to syslog:514 ends up as a row in events. That last one is the only automated end-to-end test of the collector through the importer — no Ruby-level job would notice it failing to start. The collector job covers its unit and integration tests, which docker does not run.

The same job also boots the windows profile and drives the Windows path end to end: it copies a fixture in as openwec.log, then asserts a row reaches events joined to the expected systems row. Keying on the fixture's EventRecordID rather than editing it to carry a probe marker keeps the fixture looking like real upstream output. Without this leg, a compose service that could not start or a forwarder that could not find its input would have shipped unnoticed — the same gap the syslog assertion exists for. See Windows Events.

Two details that cost a debugging cycle each:

  • The probe line must carry a syslog timestamp. EventIngestor parses one off the front and counts an unparseable line as a failure rather than storing it, so a bare probe proves only that UDP is open.
  • The wait must poll, not sleep a fixed interval. Rotation needs a packet after the 60s boundary and the importer scans LogsToProcess once a minute, so packet-to-row latency is 0–60 s on top of that.

CI steps run bash -e, so a bare code=$(curl ...) aborts the step if curl fails — the assignment carries its exit status. While Rails is still binding, the first request can be reset rather than refused, and that killed the boot poll on the runner against a perfectly healthy stack. Guard polling commands with || fallback.

PostgreSQL everywhere

config/database.yml uses the PostgreSQL adapter in development, test and production. The old split — SQLite in development and test — is what produced the worst bug in the project's history: Event.insertStatement emitted SQLite ? placeholders through exec_query, Postgres rejected them outright, and production could not insert a single event while every CI job stayed green. pg also sat in the :production Gemfile group, so any job that set BUNDLE_WITHOUT=production silently lacked the driver it needed.

To run without Docker, point DATABASE_URL at a local cluster:

createuser dad && createdb -O dad dad_test
export DATABASE_URL=postgres://dad:dad@127.0.0.1:5432/dad_test
export SECRET_KEY_BASE=not_a_secret   # require_secrets.rb fails the boot without it
bundle exec rake db:migrate_from_scratch
bundle exec rake test
bundle exec cucumber

db:migrate_from_scratch (not db:migrate) is what builds the partitioned shape the tests assert. db:migrate loads the committed db/schema.rb dump into an empty database and skips db/migrate, so the partitioning, backup and index tests then fail in ways that read like application bugs.

db/schema.rb is still committed, as a portable, non-partitioned dump. It is what the subprocess tests load into a scratch database, because those tests want the tables without the partition machinery. It is not the schema production runs, and config/environments/test.rb sets dump_schema_after_migration = false so a test run never rewrites it. maintain_test_schema is unconditionally false for the same reason: Rails purges and reloads schema.rb before a run when a migration is newer than the dump, which would install the non-partitioned shape over what db/migrate built. That setting used to be conditional on DATABASE_URL matching postgres:// — with PostgreSQL now the default, the common case is DATABASE_URL being unset, so the old condition evaluated to true and would have done exactly that damage on every run.

Raw SQL is still written with ? and routed through SqlPlaceholders.query, which rewrites it to $1, $2, …. That is not SQLite support; it is the single seam where the form Rails documents becomes the form PostgreSQL wants, and it is the seam the bug above slipped through.

Cucumber against a remote database needs one extra allowance. DatabaseCleaner treats any host outside localhost / 127.0.0.1 / *.local as remote and refuses to clean it, so the Compose service name db trips the safeguard and every scenario errors before its first step. features/support/env.rb whitelists exactly that URL rather than clearing allow_remote_database_url, so a hand-set DATABASE_URL aimed at a real server still fails loudly. The whitelist must be set before strategy = :transaction: assigning a strategy constructs a cleaner, and the cleaner runs its safeguards in its constructor.

Frontend conventions worth knowing

The app has no build step, no bundler, and no framework. Sprockets concatenates app/assets/javascripts/ and app/assets/stylesheets/ into one JS bundle and one CSS bundle, and production serves them through Apache, not Rails.

Assets. There is exactly one stylesheet, application.css. It used to be two: application.css plus a stock scaffolds.css.scss that had never been edited since the app was generated. Both went into the same bundle and neither referenced a selector the other defined, so they were merged. The scaffold rules still matter — every generated _form.html.erb uses div.field, div.actions and #error_explanation, every show page uses #notice, and .field_with_errors is emitted by Rails itself and appears in no view. Deleting them would restyle every form.

config.assets.compile = false and config.assets.digest = true in production mean precompile writes only digested filenames, and Apache — not Rails — answers the request. So a hardcoded path like src="/assets/spinner.gif" 404s in production even though it works in development, where the pipeline compiles on demand. Use the helpers. script/check_asset_paths.rb runs in CI after a production precompile and fails if any emitted path has no file behind it.

JavaScript. application.js is the manifest plus the five draw_* chart helpers; dad_charts.js holds the Chart.js plugins that supply the bands, rules and gauge Highcharts had built in; events.js and systems.js are per-page bootstrap. The draw_* names and argument lists are kept from the Highcharts era because the .js.erb templates call them positionally.

The app has no .coffee files, but coffee-rails cannot be removed: //= require turbolinks pulls in turbolinks-2.5.4's own turbolinks.js.coffee, and Gemfile.lock shows turbolinks (2.5.4) → coffee-rails. Removing the gem fails the production precompile.

Two traps specific to this layer:

  • A call inside a jQuery options object runs while the object is being built. complete: $.ajax() calls $.ajax() with no options; jQuery then uses ajaxSettings.url, which is the current page, so it fires a stray XHR — and the returned jqXHR is dropped as a callback, because Callbacks.add ignores anything that is not a function. CI greps the built bundle for [A-Za-z_$]+\.ajax\(\s*\).
  • Comments in a .js.erb file are shipped to the browser. Use <%# %> for a note that should stay in source; a // note there is part of the response and can trip an assertion made against that response.

Every id in rendered HTML must be unique. Browsers resolve getElementById and #id CSS to the first match, so every later element sharing an id is silently unaddressable. A scan of rendered HTML (not source) once found 209 duplicate ids across 17 pages — the nav bar's li#active/li#inactive on every page, the alert table's per-row ids, and 23 views re-emitting the layout's <p id="notice">. They are now classes (.criticality-5, .alert-time, li.active) and the layout owns the single #notice. test/system/unique_ids_test.rb fails on any duplicate id in rendered output, so this cannot regress.

Forms. The error summary is one partial, app/views/shared/_error_explanation.html.erb, rendered as render 'shared/error_explanation', record: @model. It used to be the same 11-line block copy-pasted into all 14 _form.html.erb files, with the loop variable already drifted between |msg| and |message|. The heading noun comes from record.model_name.human.downcase, not a hardcoded string.

Responsive. The layout declares width=device-width, initial-scale=1 and lang="en". Below 720px the nav bar tabs wrap and the alert queue stops being a fixed-width table. There is no hamburger menu by design — it would need JavaScript the app does not have. test/system/responsive_test.rb guards the markup these rules key off.

Charting

Charts run on Chart.js 4.4.7 (MIT), vendored at vendor/assets/javascripts/chartjs/chart.umd.js. It replaced a vendored Highcharts 4.0.4 build that was an unlicensed commercial dependency. Chart.js lacks Highcharts' plotBands/plotLines and gauge series, so those live as plugins in app/assets/javascripts/dad_charts.js (dadBands, dadGauge, dad_enable_x_zoom). Uglifier was replaced by terser, because UglifyJS 2.x is ES5-only and aborts on Chart.js's ES2015 output.

Before touching the ingest path

Read the comments in app/models/event_ingestor.rb and lib/log_queue.rb. The checkpoint and the batch flush are coupled, and getting the ordering wrong either loses rows or duplicates them. Ingest tests drain the queue (flush_and_wait) between staging steps so exactly one writer is active — that is what makes them deterministic instead of racing the insert thread opening a second connection while the main thread holds a write lock.

All ingest state must be initialised in resetCaches. It is the single place that class-level state is defined. Two defects came from variables it did not cover: @cachelifetime was never assigned anywhere, so the first prune_words on a long ingest raised TypeError: can't convert NilClass into an exact number and killed the importer mid-file; and the statement-parameter ceiling needed a home alongside it.

Some ingest limits are invisible to an in-process test run. DEFAULT_BULK_INSERT_SIZE is 1 in development and test, so every in-process test flushes one event at a time and no test can reach a production-sized batch. That is how a batch exceeding the adapter's bind-parameter ceiling survived: 2000 events is roughly 16 events_words rows each, about 96,000 parameters against PostgreSQL's ceiling of 65,535, and past the ceiling the adapter raises instead of degrading. Only subprocess tests with RAILS_ENV=development cross it — see test/models/event_ingest_limits_test.rb. Same shape as the SqlPlaceholders failure above: green in the suite, broken in production.

Words are resolved in bulk at flush time, not per token during staging. stage_event defers any token missing from the word cache into @staged_tokens, and performPendingInserts resolves the batch through Word.resolve_ids before building the statements. Three things that shape protects, and that any change here has to keep:

  • @staged_tokens carries the event id and generated value per event, not just per batch. Dropping a token makes an event no search can find; attributing it to the wrong event makes a wrong search hit. Neither raises.
  • Resolution is all-or-nothing. Rows are built locally and appended only once every token has an id, and storeEvent's rollback unwinds @staged_tokens along with the other two staging arrays. The importer's per-line rescue calls flush_and_wait, so a partial append followed by a retry would write some index rows twice and return the same event twice from search.
  • words.text has no unique index, so one text can already have several rows. Word.resolve_ids keeps the lowest id, which is what the old per-token find_by returned. A pluck().to_h keeps the highest instead, silently pointing new index rows at a different word than the existing ones use — and searches for that token then stop matching the older events.

Truncation to the 255-character column lives in Word.truncate_for_column and is applied by both stage_event and resolve_ids. If lookup and storage ever disagree about the form of a long token, every token over 255 characters misses on every pass and inserts a fresh row each time. Word.find_or_add is the single-token form of resolve_ids; do not reimplement it alongside it.

Throughput regressions here are silent — nothing fails, ingest just gets slower. test/models/event_ingest_word_resolution_test.rb asserts the mechanism rather than a timing: that the (event_id, word_id) rows written match the tokens, that each token is attributed to its own event, and that a batch whose tokens are all cached issues no statement against words at all.

The syslog collector (Rust)

syslog_collector/ replaced syslog.pl. It is a zero-dependency crate — std only, so there is no supply chain to audit and no dependency lock to maintain.

Build and test it on its own:

cd syslog_collector && cargo test          # 26 unit + 13 integration
cargo fmt --all -- --check && cargo clippy --all-targets -- -D warnings

The integration tests spawn the real binary on high UDP ports and compare bytes on disk, so they need neither root nor Postgres. The DAD_SYSLOG_* variables exist for exactly that: production uses the defaults, and the tests override the port (above 1024) and the log directories.

What is the contract. The collector appends <source-ip> <message>\n. EventIngestor stores that line whole in events.hunks, and Event#event_fields splits it on spaces — so the source IP is field 1 and the syslog timestamp follows it. The read side depends on the write side's formatting. The parsing is byte-oriented (&[u8], never String) because Perl never decoded: a payload with invalid UTF-8 reaches hunks verbatim, and EventIngestor does the scrubbing.

Three deliberate departures from the Perl collector, all because it lost data. Measured, not assumed:

  • It block-buffered stdio with no signal handler. 40 messages sent, 0 bytes durable after both SIGTERM and SIGKILL. The Rust collector flushes per record; measured at 20 000 datagrams, that costs nothing (~400 k msgs/s).
  • It left the in-flight file stranded in LogStaging on every restart, and nothing reads LogStaging — so up to 60s of logs were orphaned permanently, including on a planned systemctl restart. recover_orphans() claims them at startup, which covers SIGKILL, OOM, and power loss in a way a shutdown handler cannot.
  • recv($buf, 1540) silently truncated anything larger. The cap is now 65 507, the maximum UDP payload; events.hunks is a :text column, so no schema change.

Everything else is byte-identical, verified by running both collectors on UDP/514 against the same 22-packet corpus: the rotated file the importer reads was the same 2116 bytes with the same SHA-256. That equivalence check is not automated — it needed root, port 514, and a 61s rotation wait, and the Perl collector no longer exists to compare against. If you change the parse, re-verify the record format by hand.

Toolchain notes worth knowing

  • Ruby 3.3.8, Bundler 2.6.7, Rails 8.1.4.1. The upgrade path was 4.2 → 5.2 → 6.1 → 7.1 → 7.2 → 8.0 → 8.1.4 (tag rails-8.1.4), one commit per minor step; the lock now sits on the 8.1.4.1 patch release.
  • minitest ~> 5.25 is load-bearing and easy to "clean up" by mistake: minitest 6 narrowed Minitest::Test.run, and railties' TestUnit runner still calls it with three arguments.
  • rspec-rails is required even though there is no spec/ directory: cucumber-rails only mixes RSpec::Matchers into Cucumber::Rails::World when rspec/rails/matchers is loadable, and the step definitions call expect.
  • The app never calls config.load_defaults, so no load_defaults-gated Rails default is ever applied here — every behaviour change must be opted into by hand.
  • Gemfile.lock and db/schema.rb are tracked on purpose; do not re-add them to .gitignore.

AGENTS.md in the repo root carries the full set of upgrade landmines and is the best starting point for anything version-related.

Add a custom footer
Add a custom sidebar

Clone this wiki locally
0