Repository navigation
Replace the Perl syslog collector with a Rust one - #32
Merged
Merged
Conversation
syslog.pl is deleted; syslog_collector/ is a zero-dependency Rust crate (std only) that speaks the same UDP/514 protocol and writes the same staging records. Verified equivalent by running both collectors on UDP/514 against the same 22-packet corpus: the file handed to the importer was the same 2116 bytes with the same SHA-256 (c150534478b21a07). Rewriting was a chance to measure the collector rather than read it, and three measurements turned out to be data loss. Per the user's instruction, Perl's failure modes are deliberately not carried over: - Block-buffered stdio with no signal handler. Measured: 40 messages sent, 0 bytes durable after both SIGTERM and SIGKILL. Fixed by flushing per record; measured at 20k datagrams (~400k msgs/s), the flush costs nothing. - The in-flight file stranded in LogStaging on every restart. Nothing in the repo reads LogStaging, so up to 60s of logs were orphaned permanently, including on a planned `systemctl restart`. recover_orphans() claims them at startup, which covers SIGKILL/OOM/power loss where a handler cannot. - recv($buf, 1540) silently truncated larger datagrams (a 3000-byte packet yielded exactly 1540). Cap is now 65507, the max UDP payload; events.hunks is :text, so no schema change. Preserved because it is the contract: the `<source-ip> <message>` record format (Event#event_fields splits hunks on spaces, so the IP is field 1), byte-oriented parsing so invalid UTF-8 reaches hunks verbatim for EventIngestor to scrub, traffic-driven rotation, and chomp removing one trailing newline only. Wiring: dad-syslog ExecStart -> /usr/local/bin/syslog_collector; Dockerfile is now multi-stage (rust:1.85.1 build stage, and the runtime image drops perl); compose command; provision.sh installs rustc/cargo and builds+installs the binary. A new `collector` CI job runs fmt, clippy -D warnings, and cargo test (26 unit + 13 integration) -- nothing else compiled the Rust code before. Verified end to end in the Compose stack: UDP packet -> staging (durable immediately) -> rotation on the next packet after the boundary -> importer -> events row with the source IP as field 1. After `compose restart syslog` the journal shows both a planted orphan and the file the restart itself stranded being recovered, and both then land in events. Ruby suites unchanged and green: 436 runs / 0 failures on Postgres, 433 / 0 on SQLite. Co-authored-by: openhands <openhands@all-hands.dev>
cargo fmt/clippy had no working-directory, so they ran at the repo root where there is no Cargo.toml and failed in ~11s. The test and release-build steps already set it. Co-authored-by: openhands <openhands@all-hands.dev>
RotateLog/logit were Perl function names that no longer exist. Co-authored-by: openhands <openhands@all-hands.dev>
RotateLog/logit/close(LOG)/print LOG were Perl internals that no longer exist, and the paragraph repeated the claim that a 0-byte staging file 'is not data loss' -- measured, it was. This is the file future agents read first, so a stale description here is the most expensive one in the repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
syslog.plis gone.syslog_collector/is a zero-dependency Rust crate (std only — no crates, so there is no supply chain to audit) that binds UDP/514 and writes the same staging records the importer already reads.The deployment wiring follows:
dad-syslogpoints at/usr/local/bin/syslog_collector, theDockerfileis now multi-stage,docker-compose.ymlruns the binary, andprovision.shinstallsrustc/cargoand builds it.Why it is safe to call this a drop-in
Both collectors were run on UDP/514 with the production
/var/DAD/Logspaths against the same 22-packet corpus (RFC 3164 and ISO 8601 timestamps, missing/non-numeric priority, interior newlines, CRLF, empty datagrams, non-UTF-8 bytes, a datagram exactly at the old cap). The file handed to the importer — the only oneimport_syslog.rbreads — was byte-identical:Preserved deliberately, because the read side depends on it:
<source-ip> <message>\nEvent#event_fieldssplitshunkson spaces — the IP is field 1, the timestamp follows&[u8], neverString)hunksverbatim forEventIngestorto scrubensure_openruns from the per-message path, so a quiet system moves no fileschompremoves one trailing newline onlyThree things Perl did that this does not reproduce
You asked that Perl's data loss not be preserved. Rewriting let me measure the collector instead of reading it, and three measurements turned out to be losing logs:
recover_orphans()runs at startup rather than in a SIGTERM handler on purpose: a handler cannot run on SIGKILL, OOM kill, or power loss, and the orphan is on disk next start either way.End-to-end verification (Compose stack)
Restart recovery, in the container: after
docker compose restart syslog, the journal shows both a planted orphan and the file the restart itself stranded being moved intoLogsToProcess— and both then appear inevents. Non-root (uid 1000) binding 514 works via the existingCAP_NET_BIND_SERVICE;systemd-analyze verifypasses the new unit.Coverage
A new
collectorCI job runscargo fmt --check,cargo clippy --all-targets -- -D warnings, andcargo test(26 unit + 13 integration). Before this, nothing in CI compiled the Rust code — thedockerjob builds it as a side effect and would not have run its tests. Integration tests spawn the real binary on high UDP ports and compare bytes on disk, so they need neither root nor Postgres.Ruby suites unchanged: 436 runs / 0 failures on Postgres, 433 / 0 on SQLite.
Two honest caveats:
clippy::pedanticreports 13 style findings here; the gate uses the default lint set.Docs
README,
AGENTS.md,systemd/README.md, and the wiki updated.RECOMMENDATIONS.mdgains §5.8, and §5.6's claim that the 0-byte staging file was "buffering, not data loss" is corrected — it was data loss. Historical changelog rows that namesyslog.plare left as written; falsifying them would misrepresent what was fixed and when.This PR was created by an AI agent (OpenHands) on behalf of the user.