Experimental research tooling. Deterministic signals ship generic default thresholds and calibrate per install: each Pulse runs its own
pulse corpus export -> select -> agreementgate on its own sessions, and that bar (kappa>=0.6, n>=50 per stratum) is a per-user contract (ADR 0017). Gate numbers published by the project come from a developer reference run (the developer's own machine, single operator) — they validate plumbing, they calibrate nobody else. The LLM judge is provisional until an install's agreement gate passes. Artifact verification inspects, never executes.
Pulse is a harness-neutral session quality engine with native integrations for Hermes Agent and Pi. It analyzes conversations with evidence-based signals attributed to the user, agent, or system, while preserving each harness's native session and state model.
| Command | Description |
|---|---|
/pulse |
Analyze the current/latest session |
/pulse trends |
Show the latest 20 analyses |
/pulse models |
Compare analyzed models |
/pulse useful / /pulse not-useful |
Rate this session's analysis (bound to the session just analyzed, idempotent) |
/pulse yes / /pulse no |
Rate whether this session solved the problem |
The CLI supports --file, --session, and --json. pulse analyze is the versioned stdin/stdout JSON protocol used by adapters. --deep adds opt-in LLM-judge analysis (B2) on top of deterministic signals; see below.
Score an unroll trace file (safe AST load, never executed):
uv run pulse --unroll ~/.hermes/traces/unrolled/<session>.py
uv run pulse --unroll ~/.hermes/traces/unrolled/<session>.py --jsonTrace files carry TIMELINE steps but not full message text, so text-based detectors degrade in this mode while timing/cost/graph signals are authoritative. Three unroll-native detectors run on top of the standard set (all thresholds generic defaults — every install calibrates against its own corpus, ADR 0017):
latency_regression(warning) — any step withduration_ms > 5000cost_anomaly(warning) —cost_usdabove task ceiling (brainstorm $0.50, coding $5.00)skill_deadweight(warning with correction, else info) — skill inACTIVE_SKILLSwith zerotool_callsteps
uv run pulse --unroll <trace.py> --deep [--judge-model gpt-4o-mini] [--json]One combined temperature-0 call per session; four verdicts become normal
Signals: goal_completion, context_retention, correction_quality
(user-targeted), hallucination. Unparseable judge output yields zero
signals — never fabricated. Judge failure is fatal (exit 1), never a
silent fallback. --json gains a deep key (model, signal names,
input/output tokens).
Config (plugin-prefixed first): PULSE_API_KEY → OPENAI_API_KEY →
HERMES_API_KEY → $HERMES_HOME/.env; model override PULSE_JUDGE_MODEL,
base URL PULSE_JUDGE_BASE_URL. Cost envelope ~$0.02–0.05/session
(gpt-4o-mini); opt-in only — deterministic mode stays the default.
Privacy: --deep prints what it sends where (model + endpoint) before
calling. The prompt holds at most 12,000 chars (middle truncated with a
disclosed marker), per-message 500 chars, and secrets (API keys, bearer
tokens, emails, hex secrets) are redacted before construction. Deterministic
scoring is local-only; the judge call is the only network egress in Pulse.
uv run pulse corpus export [--out corpus] [--db PATH] [--limit N] [--json]
uv run pulse corpus list [--corpus DIR] [--all] [--json]
uv run pulse corpus select --corpus DIR [--out manifest.json] [--n-core 50 --n-enriched 50 --pair-floor 10 --seed N]
uv run pulse corpus select --corpus DIR --stats # report-only: pools/floors, never writes a manifestpulse corpus export reads the Hermes session DB read-only and writes one
hashed JSON snapshot per eligible session (filename = sha256(content)[:16]).
Message text is redacted at write time with the same redact_text() as the
prompt boundary; each snapshot stamps redact_version and approx_truncated
(the transcript exceeds build_prompt's 12k budget). Sessions under
extract_signals' too-short floor are skipped with the SAME predicate and the
skip counts are printed, so the corpus can never inject all-negative rows.
Incremental: unchanged content hash → skipped; a revisited session re-exports
and its old snapshot is marked superseded_by in the index.json sidecar
(hash → session_id, model, task_type, exported_at), inspectable via
pulse corpus list. Sidecar deterministic labels are ADVISORY only — the gate
recomputes them at run time from the snapshot messages, using the snapshot's
stamped task_type (skeletons carry no tool_calls, so re-detecting the task
type there would gate detectors off — ADR 0018). Corpus files are never
git-tracked (.gitignore
ships corpus/); selection manifests (hashes + metadata, no transcripts)
are also kept local by default — publishing one requires explicit owner
approval (ADR 0011), done with git add -f, never by a whitelist rule.
pulse corpus select freezes a run's membership: a seeded stratified selection
manifest — random core (generalization read) + signal-enriched half with a
=10 detector-positive floor per comparable pair (guards the base-rate collapse), deduped by session id (newest snapshot wins), stratum stamped per item. The manifest records corpus hashes, seed,
redact_version,PROMPT_VERSION, and judge model/base_url placeholder fields (judge identity is finalized at gate run time; judge-positive floors are re-checked then). Unmet floors fail loudly and write NOTHING — bars are never relaxed silently.pulse corpus select --corpus DIR --statsis the report-only companion: it prints per-pair detector-positive pools (recomputed at run time, ML-B7), stratum capacities, and floor verdicts WITHOUT writing a manifest or relaxing anything — the honest way to answer "could a gate run be selected today?" mid-calibration. Membership is run-frozen: a re-pin is a new manifest, never an edit.
Agreement gate (required before any hosted-judge claims):
uv run pulse agreement --corpus DIR --limit 50 [--cache FILE] [--json]Judge-vs-deterministic Cohen's kappa on the comparable pairs
(correction_quality↔correction_chain, goal_completion↔premature_stop;
hallucination/context-retention report judge-rates + need human
spot-check, never kappa). Verdict cache keyed by trace hash + prompt
version, makes reruns free. PASS needs kappa ≥ 0.6 at n ≥ 50 per stratum
on that install's own corpus — until then that install's deep signals
stay labeled provisional. Any gate numbers this project publishes from its
own machine are labeled developer reference run, single operator
(ADR 0017): they validate the plumbing on one machine's traffic and
calibrate nobody else's defaults. First real-key run
(2026-09-05, openai/gpt-4o-mini via OpenRouter, local 10-trace corpus):
judge returned zero verdicts on all 10 traces — thin textless TIMELINEs give
it nothing to grade — so kappa=0.0/agree=1.0 on both pairs, gate FAIL
(pending) as designed. The gate needs n ≥ 50 rich sessions before any
hosted-judge claim; numbers published as measured, not before.
uv run python scripts/build_corpus.py --out corpus # keep bottom-10 traces + sidecar JSONA weekly cron (pulse-session-gym-weekly, Mondays 09:00) rescores the corpus
plus the week's new traces and reports worst session, total cost, and top
recurring signal.
uv run pulse replay --corpus corpus [--live] [--timeout 300] [--jobs 4] [--json]Replays every *.py trace (dry-run from cache by default; --live
executes real LLM calls) with fan-out, per-trace timeout, and a
PASS/FAIL/TIMEOUT table. Exit 0 when all replay clean, 1 otherwise.
Model-change loop: replay → build_corpus.py --traces corpus --out corpus
(refresh mode rescues same-dir) → pulse leaderboard → pulse_gate.py.
uv run pulse leaderboard # ./corpus, no args needed
uv run pulse leaderboard --corpus DIR [--json] [--task coding] [--top 5]Loads *.score.json sidecars (scores *.py traces live when a sidecar is
missing). Ranks top/bottom N per task type (default 3); session IDs are anonymized
(sha256, first 12 chars); score ties break toward lower cost.
--task filters to one task type (onboarding curriculum: print top-5 per
task in the onboarding doc). Standalone: needs no Hermes install — just
pip install hermes-pulse and a directory of trace files.
uv run pulse compare --a traces_prompt_a/ --b traces_prompt_b/ [--json]Mean/median/p25/p75 per variant, cost delta, timing delta (when duration fields exist), and a plain-English verdict ("A wins on quality (+3.2 pts, cost +0.0100, provisional n=20)"). No significance testing at v1 — effect size + N, labeled provisional.
uv run pulse skills --corpus DIR [--json]One card per skill: loads, deadweight rate, correction rate, mean cost, task-type mix, plus the mean cost of skill-less sessions of the same task types as a baseline. Correlation, not causation — the card shows task mix alongside the numbers so a skill loaded only in hard sessions isn't misread. Feeds the skill-curator workflow (kill or fix with data).
uv run python scripts/pulse_gate.py --baseline corpus/main --candidate corpus/pr [--tolerance 5.0]Exit 0 on pass, exit 1 on fail, with a per-task delta table. Fails when any
task-type mean score drops more than --tolerance points (default 5.0).
Standalone: needs no Hermes install — both inputs are plain directories of
trace files + sidecars. Copy-paste GitHub Actions snippet:
- name: Pulse quality gate
run: uv run python scripts/pulse_gate.py --baseline corpus/main --candidate corpus/prAdversarial prompts live in scripts/redteam/prompts.md (12 prompts, 4
categories: ambiguous reference, contradictory instructions, mid-task scope
creep, missing-context traps). Run each prompt as a live session per model,
capture the traces, then rank:
uv run python scripts/redteam_score.py --traces redteam_traces/ [--json]Per model: mean score plus frustration / correction-chain / reasoning-loop counts per 10 sessions, calmest first. Standalone: needs no Hermes install — input is a plain directory of trace files.
uv run pulse portability # ./corpus, no args needed
uv run pulse portability --corpus DIR [--json]Per skill: deadweight rate per model plus a verdict — portable (low
deadweight everywhere), model_specific (deadweight on some models, clean
on others), dead (deadweight everywhere). Reads active_skills +
signals straight from sidecars, so it never re-parses trace files;
corpora scored before v0.3 lack active_skills — rescore with
scripts/build_corpus.py to fill them in. Standalone: needs no Hermes
install.
uv run pulse export --corpus DIR --out export/ [--format sharegpt|jsonl] [--min-score 90]
uv run pulse export --corpus DIR --review # spot-check mined DPO pairs firstTrace TIMELINE → message list with tool calls; pulse score → quality
filter (score ≥ threshold → SFT candidate); correction_chain evidence →
DPO pairs (pre-correction assistant turn = rejected, post = chosen).
manifest.json carries a per-file redaction receipt (redacted-at-capture
— unroll redacts at capture time). --review dumps pairs for human
spot-check: not every correction is a clean chosen/rejected pair.