Repository navigation
Tags
Tags
Release 1.1.0 (#178) * Release 1.1.0 The version set everywhere it lives: server, Python SDK, TypeScript SDK, web app, root skills, and the uv and npm lock files. The coding-agent plugins keep their own version: nothing they ship changed since their last bump. Signed-off-by: pufit <pufit.dev@gmail.com> * Helm chart: appVersion follows the release The chart's appVersion is the default tag of the server image and, since the extractor service got its own image, of the extractor image. It still read 1.0.0, which the 1.0.1 release did not move, and no 1.0.0 extractor image will ever exist. The release-process table now lists the file with the other places the version lives. Signed-off-by: pufit <pufit.dev@gmail.com> --------- Signed-off-by: pufit <pufit.dev@gmail.com>
Release 1.0.1 (#154) The version set everywhere it lives: server, Python SDK, TypeScript SDK, web app, root skills, and the uv and npm lock files. The coding-agent plugins stay at 1.2.1 (nothing under integrations/ changed since v1.0.0). Signed-off-by: pufit <pufit.dev@gmail.com>
Sign-in through an in-cluster OIDC base with advertised algorithms, a… … /me landing, and platform-admin machine identities (#110) * Coded vector DDL follows the codec settings this user can have on Loom took enable_quantized_codec for available whenever system.settings listed it. ClickHouse Cloud lists it but holds it read-only: its default is 0, and the operator enables it per service with a settings-profile override. Where it is held at 0, the coded CREATE sent SETTINGS enable_quantized_codec = 1 and the server refused it (code 452), so a fresh install failed instead of creating the plain layout. With the codec enabled, allow_suspicious_codecs stays read-only at 0 there, and the entity_vectors CREATE failed on its LZ4 chain (code 36 without the setting, 452 with it). Every path that creates a coded table reads value, readonly, min, max and disallowed_values of the codec settings before it sends any (vector_codec_ddl.CodecSettings). A setting is held when this user may not set it to 1: readonly, or a <max>0</max> or <disallowed>1</disallowed> constraint, which system.settings shows with readonly = 0. state enable_quantized_codec allow_suspicious_codecs settable coded, sent LZ4 chain kept, sent held at 1 coded, not sent LZ4 chain kept, not sent held at 0 plain layout Quantized alone, not sent absent plain layout Quantized alone, not sent - the migration runner's fresh-install layout, and codes_state's codec_available; - the provider rebuild's candidate table: refused before its CREATE where the codec cannot be on (an owned stale candidate is dropped first), its chain dropped where it must be; 26.7 keeps the allow_experimental_codecs gate under the same rule; - scripts/ops/vector_codec_rebuild.py: refuses to run where the codec cannot be on, and refuses --chain where allow_suspicious_codecs cannot be; a held setting goes with no statement, the copy, the deltas and a rollback included; - dev/restore_loom.py: refuses a backup with coded tables before anything is written, --dry-run included, where the codec settings cannot be read or the codec cannot be on (checked again under the restore lock), and drops the chain on an explicit CREATE. Without the chain, entity_vectors.entity_embedding takes about four times the space LZ4 gave it: the operator enables allow_suspicious_codecs and rebuilds that table with --chain LZ4, or accepts the larger column. The server accepts a held setting sent at its current value, and clickhouse-connect drops a readonly one before sending; a held setting is left out so neither is relied on. Servers where both settings are settable, and servers before 26.8, take the same layout and settings as before. The one difference on 26.7: scripts/ops/vector_codec_rebuild.py refuses with the reason instead of failing on UNKNOWN_SETTING. Unit tests cover every state pair and each constraint shape at the migration runner and the provider rebuild, and each tool's refusal and degrade branch. Signed-off-by: pufit <pufit.dev@gmail.com> * feat(oidc): reach the provider at a back-channel base, verify the algorithms it lists LOOM_OIDC_BACKCHANNEL_BASE_URL (default empty) names a base URL (http(s), host, optional port) Loom reaches the identity provider at when the issuer's own host is not reachable from it. Loom's own calls to the issuer's host (discovery at <base>/<issuer path>/.well-known/ openid-configuration, the token and revocation endpoints, the signing keys) go there with the path kept and the issuer's host in Host; the browser still goes to the issuer's authorization endpoint, an endpoint on another host is called as it is, and the discovery document and every ID token must still name LOOM_OIDC_ISSUER. None of these calls follows a redirect. The value is checked at load and never echoed. An ID token is verified with the algorithms the discovery document lists in id_token_signing_alg_values_supported among RS256, RS384, RS512, PS256, ES256, ES384 and ES512 (RS256 alone when it lists none), so a provider that signs with ES256 signs people in; none and HMAC algorithms are never accepted. Signed-off-by: pufit <pufit.dev@gmail.com> * feat(auth): let a machine identity hold platform_admin, and mint it with an expiry A control plane that administers Loom (registers trusts, grants, erasure) had no sanctioned way to hold an expiring platform-admin credential: the role was conferred only at sign-in from LOOM_PLATFORM_ADMINS. - `loom bot create --platform-admin` and "platform_admin": true on /v1/admin/bots.create (platform admins only) register a bot whose every token holds the role: a flat token over every namespace, hour-long as any bot's, ended by `loom bot disable`. Such a bot names no namespace and takes no agent claim (refused at registration). The flag is a new column, loom.machine_identities.platform_admin (migration 072, DEFAULT 0), recorded in the auth.machine_created audit row and shown by `loom bot list` and /v1/admin/bots.list. The check that two trusts of one issuer cannot take one token is unchanged; trusts kept apart by their sub claims register side by side. - `loom mint-token --platform-admin --subject sa:<name> --ttl-days N` mints the role for a service account and refuses a token without an expiry, a person's subject, --user-token and --namespace-role. A build without the column writes 0 when it saves a bot, and mints no token for a platform-admin bot (it names no namespace), so a rolling upgrade never widens a bot. schemas/schema.sql carries the column as the dump lays out an added one. Signed-off-by: pufit <pufit.dev@gmail.com> * fix(auth): a platform-admin bot is registered by a person, with exact claims Review fixes for platform-admin machine identities: - A bot row or dict without platform_admin (an older build's, the integration fixture's) reads as one without the role: machines.save, the token exchange and the listings read the flag with a default instead of raising KeyError. The integration fixture names it. - /v1/admin/bots.create refuses platform_admin (create and replace) to every service account's token, a platform-admin bot's own included, so a leaked hour-long token cannot register itself a lasting successor with a generated key or an issuer of its own; a person signed in as a platform admin, or `loom bot create` on the host, registers one. - A platform-admin trust matches every claim exactly: a prefix (`*`) would hand the role to every token it matches. - `loom mint-token --platform-admin` mints for at most 90 days, never for an agent's subject (sa:<trust>:<agent>), and records each mint as an auth.platform_admin_minted audit row before printing the token; a mint the audit log misses is not handed out. - The docs register the platform-admin trust with an audience of its own. Signed-off-by: pufit <pufit.dev@gmail.com> * docs(schema): state the platform-admin column's rolling-upgrade rule docs/architecture/rolling-upgrades.md gains rule 24 for 072_machine_platform_admin.sql: existing bots read 0; an N-1 replica answers a platform-admin bot's exchange 401 and refuses the field on bots.create (422), so such a bot is registered and used only after the rollout; N-1 writes the flag back to 0 only on an explicit disable or replace. Signed-off-by: pufit <pufit.dev@gmail.com> * fix(oidc): only the sign-in provider's own keys go through the back-channel base oidc.signing_key is shared with the machine-identity path, so a bot trust's https jwks_uri on the issuer's host was fetched over LOOM_OIDC_BACKCHANNEL_BASE_URL, past bot_from's https rule. The route now applies only when verify_id_token asks for the sign-in provider's keys (sign_in=True); a trust's keys are fetched from the address it names, and the client cache keeps the two apart. Signed-off-by: pufit <pufit.dev@gmail.com> * fix(auth): a dashboard sign-in reads revocations before it mints The dashboard sign-in reads revocations from ClickHouse before it mints, as every other mint does: a revoked subject gets "Sign-in denied" (audited as auth.signin_denied) instead of a token refused at its first use, and a failed read answers the sign-in-unavailable page. The SSO callback lands on /me#token=, the page served without the LOOM_DASHBOARD_REDIRECT_URL redirect, so the fragment never follows a redirect to another origin. The break-glass and minted-token instructions and the HTTP route table name /me, and the tests pin the landing. Signed-off-by: pufit <pufit.dev@gmail.com> * fix(auth): a third-party client's session registers no platform-admin bot - /v1/admin/bots.create also refuses platform_admin to a third-party MCP client's session token (a session id, not first-party): its tokens end with the session, so it mints nothing that outlives it, as delegation already rules; a person's own `loom login` or dashboard session still registers one. - The dashboard page says /me is the one page load the aggregator redirect (LOOM_DASHBOARD_REDIRECT_URL) leaves alone. - Tests name their trusts with neutral placeholders. Signed-off-by: pufit <pufit.dev@gmail.com> * Renumber the platform-admin migration to 074: main took 072 Signed-off-by: pufit <pufit.dev@gmail.com> * Short-lived tokens cannot rebind an account's SSO sign-in either bots.create refuses platform_admin to a service account's token and to a third-party session, but sso.link, sso.adopt (apply), sso.unlink and sso.password_link took any platform-admin bearer: a platform-admin bot could link an admin's account to an address it holds, sign in as that admin and register a lasting admin bot. They now refuse those callers the same way. Signed-off-by: pufit <pufit.dev@gmail.com> * The unit and integration jobs fit their suites, and the docker smoke and a replay test wait for what they check (#140) * The docker smoke passes until every fixture has its first index The code smoke seeded its eight fixture repositories with at most five back-to-back ingest passes and went on to the compiler tiers whether or not every git_head had landed. A pass hands a missing mirror's clone to a background job and moves on, and an entry's first index (its git_head) is written by the bootstrap leg of a later pass. When the clones took longer than five quick passes, as on the amd64 Quickstart run of main at 49b3d356 (the five passes and the Python tier ended 0.3 s after the first clone's sandbox probe), scip_py_delta found no git_head and the smoke failed with "code-docker-smoke-python: git_head not seeded". The same tree passed on the pull request's own run and on arm64. The seed now passes until every entry's git_head equals its head, a second apart, and fails naming the entries still without one after 180 s. Two unit tests drive seed() with collaborators whose heads land on the eighth pass, and with none landing. Signed-off-by: pufit <pufit.dev@gmail.com> * The replay test waits for its failure's resolved mark test_a_replay_that_inserted_never_inserts_again_after_a_failed_repair asserted that the org has no active projection failure right after the second retry returned resolved. The retry marks the failure with mark_projection_failures_resolved, an ALTER UPDATE without mutations_sync, so the mark lands in the background; _mutations_done waits only for the triple and edge mutations. On a loaded runner the read came first and listed the failure still active, with the payload of the replay that resolved it (main at 49b3d356 and at 18fe4dd3). The test now waits up to 30 s for the active list to empty, as the 4,000-failure mark test in test_retirement_ch.py does; a mark that never lands still fails it. Signed-off-by: pufit <pufit.dev@gmail.com> * The unit and integration jobs have room for the suites they run Over the last 40 pushes to main the unit job's pytest step took 16.9 to 29.7 minutes, and the job hit its 30-minute limit in 10 of them: on 49b3d356 it was cancelled at 99% with every test so far passing (the same seed passes locally, 23,345 tests). The integration job's first pytest step has taken 42 to 59 minutes since the conflict-resolver tests arrived, before the auth-on server's tests (about 6 minutes) start; it was cancelled at its 60-minute limit on 7fba7f5d and 87879ba6, and the green pull request run of 49b3d356's tree took 56 minutes. Neither job hangs: both are cut off while still passing tests. The unit job now has 45 minutes and the integration job 90. Signed-off-by: pufit <pufit.dev@gmail.com> --------- Signed-off-by: pufit <pufit.dev@gmail.com> * Bump the actions group with 2 updates (#141) Bumps the actions group with 2 updates: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) and [actions/setup-node](https://github.com/actions/setup-node). Updates `astral-sh/setup-uv` from 10.2.0 to 10.3.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/v10.2.0...1c37ad07a6a961277cf70c0d37d6f313000f5884) Updates `actions/setup-node` from 7.0.0 to 7.1.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v7...949feb2413d6458794dcd2491c4babbbce0c15c1) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 10.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: actions/setup-node dependency-version: 7.1.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Model calls name whom they are for, and a gateway refusing one identity holds only that identity (#111) * Model calls name whom they are for in configurable identity headers LOOM_INFERENCE_IDENTITY_HEADERS_JSON maps header names to templates over {org}, {namespace}, {subject}, {user_id}, {agent}, {tenant} and {email}. Every chat, embedding, rerank and classify request carries them. A request names its token's subject; a background job names the writer of the row it works on, else the agent whose own namespace the row is in, and splits a batch by identity. A gateway that bills each agent then sees the agent behind background work as well as requests, instead of the deployment for both. With no templates nothing changes. Signed-off-by: pufit <pufit.dev@gmail.com> * The workload token can be sent to the gateway as the bearer itself LOOM_INFERENCE_WORKLOAD_TOKEN_MODE=bearer sends the content of LOOM_INFERENCE_WORKLOAD_TOKEN_FILE as Authorization: Bearer on every model call, for a gateway that verifies the workload's token itself and has no token exchange. The file is read again at most once a minute and at once after the gateway answers 401, so a rotated token is picked up without a restart. The mode requires the gateway, as the exchange does, so the token goes nowhere else. A keyless compatible embedder now sends the workload token in either mode instead of falling back to the fake embedder. The default mode, exchange, behaves as before. Signed-off-by: pufit <pufit.dev@gmail.com> * A gateway refusing one identity holds that identity's calls, not the provider With identity headers set, a 403 or a 429 whose error names a budget, from a gateway host, is the gateway refusing the identity the call was made for. The model client's response hook raises ModelCallRefused (403, MODEL_CALL_REFUSED, with the gateway's status and code) instead of an HTTP error, so the request answers it, nothing retries it or hands it to the fallback model, /v1/health does not read it as a provider failure, and the org's switch is untouched. It is one of customer.HOLDS: a background job leaves the row as it is, and for LOOM_INFERENCE_REFUSAL_COOLDOWN_SECONDS (600) its calls for that identity are refused before anything is sent, while other identities' work goes on. One over-budget agent no longer reads as the provider being down for the whole deployment. Any other 429 is a rate limit, retried as before. The bearer tests now point each sender at a fresh model client: a sender first imported while another test patched get_model_client kept that patch. Signed-off-by: pufit <pufit.dev@gmail.com> * A job's model call is for the row's writer, the namespace's agent only without one Jobs that never passed the writer fell back to the namespace's owner even when the row recorded its actor, so the owner agent, or nobody for a shared namespace, paid for a member's or another agent's work. The embedding worker, projection repair and the code-link judge now read the writer of the memory they work on; observe-extract takes the newest writer of the session's observations, a profile fold the newest writer of its facts, reflection and the period digest the newest writer of the session's memories, and the conflict sweep passes the write's writer to settle(). The owner is used only for a row that records no principal. A pipeline's own actor (process:extraction-worker, process:cli:<command>, the other provenance.PIPELINE_ACTORS) names no principal: subject_of() returns '' for it, so such a row falls back to its namespace instead of going out with no agent. A test pins every process:<name> literal in the server to the registry. The writer reads run only while an identity header names more than the org and namespace. Signed-off-by: pufit <pufit.dev@gmail.com> * Identity headers name the agent whole, and a call naming no org is refused The headers whose templates name {agent} or {tenant} are now sent together or not at all. An agent whose trust requires no exact tenant, or a trust's own token, used to send the agent header alone; the gateway refuses half an identity with a 400, which read as a provider failure for the whole cell. Now neither goes and the gateway applies its default identity. With identity headers set, a call that names no org is refused and logged (MODEL_CALL_UNATTRIBUTED) unless it is the deployment's own, as it already was with the customer header: an MCP call whose identity stamping failed, or a loom command run without --org, no longer goes out under the gateway's default identity. A job batching rows now groups them by the identity they render, so one agent's rows in several namespaces share a call. Templates are trimmed, and an empty template or a hop-by-hop header name is refused at load. Signed-off-by: pufit <pufit.dev@gmail.com> * Only the answers a deployment lists hold one identity, matched exactly LOOM_INFERENCE_IDENTITY_REFUSALS_JSON lists the gateway answers that are one identity's, not the provider's: {"refused": [...], "wait": [...]}, each entry a status and at least one of the error body's code, type or message, matched exactly. The default lists the metering's 429 budget_exceeded. A refused answer raises ModelCallRefused as before; a wait answer (too many of an agent's requests in progress, no credential for it yet) raises the new ModelCallDeferred, 503 MODEL_CALL_DEFERRED with the gateway's Retry-After. Any other answer is the provider's: another 403 is a provider failure again, still not retried, and no word in a message counts. A job's hold moves from customer.require() to the model client's request hook (gateway.sending) and is keyed by the identity and the path and model the answer was for, so one model's refusal no longer holds an agent's embeddings. A wait holds for its Retry-After (at most 5 minutes), a refusal for the cooldown, and a success at that path and model ends the hold. The identity headers go only to the gateway's hosts: the request hook drops them from any other request, a redirect hop included, and setting them now requires LOOM_INFERENCE_GATEWAY_HOSTS. The reranker and the classifier post through the model client (never redirected, as before), so the gateway check, the identity headers and the refusal handling apply to them, and a refusal raises instead of keeping the input order or the caller's type. Signed-off-by: pufit <pufit.dev@gmail.com> * A job skips an identity the gateway holds instead of stalling on it A job that meets a held identity remembers the work it left until the hold ends (server/workers/holds.py, ModelCallRefused.hold_s) and leaves it out of its next selections, so its batch goes to other identities' work. The profile fold leaves the held identity out of its scan. The embedding worker names the held stubs for find_embedding_stubs to leave out (exclude_keys) and, while any identity is held, reads four times its batch and drops the held identities' stubs before calling, so stubs a held writer filled the oldest slots with no longer stop other writers' stubs from being embedded. What names an identity going unread (IdentityUnread) now holds one group like a switched-off org or a refused identity (customer.GROUP_HOLDS) in the four batch embedders: the memory and code embedding workers, loom reembed and a model rebuild write the vectors the other groups' calls bought instead of stopping the pass or failing the rebuild step as a provider error. Signed-off-by: pufit <pufit.dev@gmail.com> * The extraction queue holds, backs off and spends no attempt on an outage A replay whose model call is held (the org switched off since the claim, the switch unread, the enqueuer's identity refused, asked to wait or unread) is marked `held` with the reason in last_error until the hold ends, instead of staying `running` behind its lease; no attempt is spent. While it is held the claim takes none of that enqueuer's other requests in the namespace and for that user, so the batch goes to other work rather than rows that would only be held again. A failed attempt now waits 15 s, 1, 5, then 30 minutes before the next claim, and LOOM_EXTRACTION_QUEUE_MAX_ATTEMPTS defaults to 8 instead of 3 (recorded in the settings history). A model provider or gateway that cannot be reached (a transport error, a timeout, 502/503/504) spends no attempt and waits the same backoff. A request still waiting on its last day before the table's 30-day TTL is dropped with a memory_drops record (held_past_ttl or waited_past_ttl) and marked failed, never removed unseen. memory.extraction_status takes and reports the `held` state, and its pending count, memory.search's pending_extraction and the pending evidence include held batches. Signed-off-by: pufit <pufit.dev@gmail.com> * code.refresh calls models as its caller; in-database reembed is refused when calls must be named The ingest pass code.refresh runs (the poller's) now makes its model calls for the request's caller, its token's subject, as any request's calls are; the poller's calls stay the namespace's agent's. loom reembed --in-database is refused, before any row is read, while a customer header or identity headers are set or the inference gateway is required: ClickHouse's aiEmbed calls the provider itself, naming no customer or identity and outside the gateway's host check. The CLI prints the reason and exits 2; the Python path is unchanged. Signed-off-by: pufit <pufit.dev@gmail.com> * Identity headers are decoded once per value, not on every read Every model call reads LOOM_INFERENCE_IDENTITY_HEADERS_JSON several times (rendering, grouping, the request hook, the gateway's answer); it is now decoded once per value, as the refusals table is. Signed-off-by: pufit <pufit.dev@gmail.com> * Only a provider that does not answer spares a queued extraction an attempt A request the client cannot make at all (a bad scheme, a local protocol error) is the request's failure and spends an attempt; no connection, a timeout, an answer cut off, no gateway token, or a 502/503/504 spends none. Signed-off-by: pufit <pufit.dev@gmail.com> * The CLI surface snapshot pins the in-database refusal in reembed's help Signed-off-by: pufit <pufit.dev@gmail.com> * The embedding worker holds writer groups, not stub keys, and binds lists that fit Holding a writer's stubs by their keys put up to 2000 eight-string tuples in one parameter, and the 4x read put 1600 candidate keys in the next two: both past ClickHouse's 128 KiB form-field limit (http_max_field_value_size), so once enough stubs were held every pass failed with "Field value too long". The worker now keeps the held identities (each stub's writer's) and the (org, namespace, agent, user_id) groups their stubs are in. The selection reads the held groups' stubs after every other stub (a handful of tuples, cut to one field), and the held identities' rows among what it reads are left uncalled, so other writers' stubs, a held writer's neighbours under its label included when the batch has room, are embedded meanwhile. The read is the batch again, not four times it. Every list a stub read or writer lookup binds goes in chunks that each fit one field (server/core/ch_params.py), measured as the driver renders them. session_writers read loom.memory without FINAL and by created_at, so it could name the writer of a deleted memory or of a replaced version (an update keeps created_at). It reads FINAL now and picks by version, as the profile fold does. Signed-off-by: pufit <pufit.dev@gmail.com> * The extraction queue's enqueuer gets its own column, so the claim reads no payloads The claim left out every enqueuer with a request held now by reading JSONExtractString(payload) in both its outer NOT IN and the subquery, so each 15-second pass read every payload in the table twice (3000 finished rows of 20 KB: 120 MB a pass, against 165 KB before), growing for the queue's 30 days. Migration 063 adds memory_extraction_queue.enqueued_by (String DEFAULT ''), the principal the payload already names. enqueue writes it, every mirrored version carries it, a hold fills it for a row queued before the column, and the claim's held-enqueuer check reads it with the other narrow columns, so the payload is read for the claimed rows alone. A row queued before the column reads '' and is claimed and held at once if its enqueuer is held. The queue's column list is pinned to the table's in the schema test. Signed-off-by: pufit <pufit.dev@gmail.com> * The profile fold reads four times its slots and gives a held identity's to the next The fold runs every six hours and a hold lasts ten minutes, so an identity held in one pass was selected again by the next, and ten held identities at the top of the ranking took every slot, pass after pass. The scan now reads four times max_identities_per_pass, and a fold held in the pass takes no slot: the next identity read is folded in its place, and the held one stays out of the selection while its hold lasts. A switch gone unread ends the pass. Signed-off-by: pufit <pufit.dev@gmail.com> * Rendered identity header values are trimmed, and left out when only space remains A template such as "{email} via loom" rendered " via loom" for an agent, which has no address, and the HTTP layer refuses a header value with a leading space. Each value is now trimmed after rendering; one with nothing left is not sent. Signed-off-by: pufit <pufit.dev@gmail.com> * A short wait on a request's model call is waited out once before it is deferred A "wait" answer made a request fail at once with 503 MODEL_CALL_DEFERRED, where Loom used to retry a busy provider in process; the gateway's per-agent request limit asks for one second. A request's call (never a job's) asked to wait two seconds or less now waits it out once and lets its sender's own retry ask again; a second such answer for the same call, or a longer wait, defers it. A job's call is still held at once. Signed-off-by: pufit <pufit.dev@gmail.com> * The built-in refusals table matches the gateway's answers by code The inference proxy now writes a stable error.code in every body, so the default LOOM_INFERENCE_IDENTITY_REFUSALS_JSON matches its per-a 8000 gent answers by code: agent_denied (403), identity_incomplete and identity_malformed (400) refused, agent_request_limit (429) a wait. The metering's budget refusal is matched twice: by code (budget_exceeded) on OpenAI routes, and by type and whole message (rate_limit_error, "budget exceeded") in the Anthropic envelope of /v1/messages, which carries no code. token_unavailable and identity_missing stay out of the table and the docs say never to list them: the first is also what a credential-service outage answers, the second a gateway with no default agent, and either would vanish from health. The deployment-level answers (caller_*, scope_not_granted, route_not_found, upstream_*, misconfigured, no_identity) stay provider failures. The docs' examples match by code, no longer by message text. Signed-off-by: pufit <pufit.dev@gmail.com> * A loom command names its namespace, and one naming none is refused while the headers name the agent The CLI named only --org around a command's model calls, so an in-process import's calls rendered no agent and went out as the gateway's default agent (not refused, since an org was named). The dispatcher now names the command's --namespace too, so its calls are for the agent whose namespace it is, and while the identity headers name the agent, a command's call with --org and no namespace or subject is refused (MODEL_CALL_UNATTRIBUTED, logged) rather than billed to the default agent. Commands that walk namespaces name each row's own. Signed-off-by: pufit <pufit.dev@gmail.com> * A deferred conflict judge scheduled inside a job is for the job's writer schedule_settle took the request's caller, which a worker has none of, so a write an extraction replay made for its enqueuer sent its deferred judge out for the namespace's agent. It now takes whom the surrounding calls are for (a request's caller, or the writer a job named for its row), else the write's own principal from its provenance; a pipeline's own write still names no one. Signed-off-by: pufit <pufit.dev@gmail.com> * The SDK references say where a deferred model call's wait is; a stale test citation is fixed MODEL_CALL_DEFERRED reaches both SDKs as BackendUnavailable, which carries no retry_after_s: the references say the wait is in detail.retry_after_s and the Retry-After header (the Python transport's retries of an idempotent op honour it), and the error's docstring says so too. The provenance registry's comment named a test file that does not exist. Signed-off-by: pufit <pufit.dev@gmail.com> * The analyzer corpus drives the new statements, and its shape list is current Regenerates tests/analyzer/shapes.txt for the claim, pending_count and the schema dump, and adds cases so CI checks the new statements against an engine: the queue's TTL drop, the session-writer read, the profile fold's scan with and without a held identity, and the stub selection through its FINAL read with and without a held group ordered last. Signed-off-by: pufit <pufit.dev@gmail.com> * The architecture page describes the holds as they now work The embedding worker orders a held writer's groups last instead of reading four times its batch, the profile fold over-fetches instead, and the queue's held-enqueuer check reads its own column. Signed-off-by: pufit <pufit.dev@gmail.com> * The carry-columns test reads the stub selection's FINAL read where it now is find_embedding_stubs reads its stubs in _stub_rows (one read per bound chunk of keys); the test that pins the columns the RMT carry needs reads that function and checks find_embedding_stubs calls it. Signed-off-by: pufit <pufit.dev@gmail.com> * Deployment calls inside a loom command are exempt, the refusal log no longer leaks across tests, and list parameters are measured in bytes The CLI's no-namespace refusal had no exemption for the deployment's own calls, so `loom recalibrate`, or a model change's calibration run inside a command, was refused while identity headers were on. A call inside customer.system() is exempt from it, as from the no-org refusal. The test suite now clears the once-a-minute limiter of the MODEL_CALL_UNATTRIBUTED log line after every test (tests/unit/conftest.py), which made the CLI refusal test depend on the order tests ran in (seed 1357); the test asserts the refusal itself. server/core/ch_params.py measured a parameter's rendered length in characters, but ClickHouse limits the field in bytes: 300-character CJK keys made a 65,359-character chunk of 148,759 bytes. It measures UTF-8 bytes now, tested with multi-byte keys. Signed-off-by: pufit <pufit.dev@gmail.com> * The structured profile build calls for its user's namespace, user and writer The structured profile worker from #93 named only the org, so its model calls went out as the gateway's default agent. build_profile now names the namespace, the user and the newest principal that wrote the facts it reads (user_facts returns each fact's actor and version), as the profile fold does; a pipeline's or a worker's own write names no principal, and the call is then for the agent whose namespace it is. A build held for its identity (customer.GROUP_HOLDS: the org switched off, the identity refused, asked to wait or unread) leaves that user out of the namespace's passes until the hold ends (hold_seconds, as the fold's), through the worker's own per-user wait: no failure is counted, and a failure backoff's next wait stays as it was. Other holds still only skip the user. Tests pin the identity the build sends (the newest writer; the namespace's agent when no principal wrote the facts) and that a held user waits out its hold while the pass and the next one take the namespace's other users. Signed-off-by: pufit <pufit.dev@gmail.com> * Keep model attribution and held work correct across retries Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Attribute filtered model inputs and count held extraction work Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Model unmerged rows and worker inputs accurately in tests Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Keep unchanged analyzer descriptions as generated on main Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Record held enqueuer claim SQL in the analyzer corpus Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Collections menu focus rings fit inside their cells Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Projection repair walks past held failures A keyset cursor lets each bounded pass reach later failures while held identities remain active. Timestamp ties follow ClickHouse's native UUID order. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Exercise current knowledge hits and resolver attribution in unit tests Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * jsdom supplies scrolling for delayed transcript navigation Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Accessibility audits wait for the page entry fade Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Assert default user and session in sensitivity canary responses Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Record namespace guide statements in the analyzer corpus Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Focused rows stay visible during page entry The entry translation suppresses scroll anchoring while offscreen rows replace their size estimates. Focus removes the translation while the opacity fade continues, so keyboard navigation keeps the row clear of the sticky toolbar. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Back keeps its saved scroll offset after focus Restoration checks the offset one frame after scrolling or focusing. Deferred rows can resize after focus and change the offset, so restoration continues until the requested position remains in place. The existing timeout bounds missing content. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Drain buffered events before erasure counts their rows A delayed telemetry batch can arrive after the event deletion step finds no rows and fail the residue census. Serialize flushes and drain this replica before counting, with a bounded wait that leaves erasure retryable. Coalesce threshold flushes so slow inserts do not accumulate waiting tasks. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Row estimates use content height Skeletons and estimates share a row height. Containment subtracts the existing padding and borders because its size applies to content. The browser regression measures a fresh skipped box instead of comparing different box models. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Namespace keyboard checks wait for search focus Popover visibility precedes the lazy panel focus handoff. Arrow-key traversal starts after the search owns focus, preserving the scroll and fade assertions. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Configure the event drain deadline used by erasure Expose the five-second event drain deadline as LOOM_ERASURE_EVENT_DRAIN_TIMEOUT_S. Reject non-positive and non-finite values so erasure always bounds its wait before counting event rows. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Scope model-call selector fixtures to their own namespace Global bounded selectors can fill their batches from unrelated orgs before reaching the fixture. Keep both fixture orgs in one fresh namespace and use the existing namespace filters while retaining the limits and switch assertions. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Extraction tests expect ignored learning arguments to be removed Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> --------- Signed-off-by: pufit <pufit.dev@gmail.com> Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com> Co-authored-by: Lareb Zafar <lareb.zafar@clickhouse.com> * Inserts into the tables written on every request share ClickHouse parts by default and still return once written (#129) * Inserts into the tables written on every request batch server-side in ClickHouse audit_log, conversations, conversation_embeddings, agent_events, the webhook outbox and attempts, entity_canonical_map, memory_evidence, memory_belief_sources and memory_chunks are written with async_insert = 1, wait_for_async_insert = 1: concurrent inserts share a part, and each returns once that part is written. Two live settings bound the wait. bench/write_rate.py measures latency and parts per table at a fixed write rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: pin the new defaults, state the parts bound per node and shape, test batched read-back and dedup, harness takes a token Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: parts bound reads both ways, carry acks use the bounded wait, bench counts unstored turns as errors Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: the remaining async-wait inserts use the bounded helper, the shared-parts test stops merges, the docs name each batched writer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: name the policy witness's exposures rows, fix the chunk insert comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: keep code spans whole in the doc, say what a failed shared flush means, read back every field Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: label the interval writers in the batched list, shorten the flush comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: sign-in attempts, projection-failure records and graph Undo markers use the bounded helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: the attempts comment sits on its insert, the doc lists every batched site, the failure-row test says what it checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Address review: entity_canonical_map stays committed, a flush makes one part per partition, the usage drain test expects the batched settings entity_canonical_map is partitioned by org, so a shared flush writes one part per org and can exceed max_partitions_per_insert_block across tenants; it goes back to a committed insert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Tests check that the tables written on every request use batched inserts A parametrised AST check asserts that the audit_log, agent_events, conversation_embeddings, webhook outbox and attempts, memory_evidence and memory_belief_sources inserts pass `batched_insert_settings()`, and that no insert into them omits `settings=`. The audit test stubs require `settings`, and the emit_many test asserts both audit inserts are batched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Batched-insert tests pin every insert site, and entity_vectors and write marks batch The read-path test records an insert without settings as the session defaults and any unresolved settings value as a failing entry. The batched test covers memory_chunks, entity_vectors, namespace_write_marks and carry_acks. entity_vectors and every namespace_write_marks insert are sent batched. write_rate.py reports the async_insert value each conversation insert carried, from system.query_log. The architecture page calls inserts that wait "waited", and "committed" means async_insert = 0 only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * conversation.append and set_from_messages budgets cover their batched waits conversation.append makes two batched inserts in sequence and memory.set_from_messages about six, so their p99 budgets rise to 130 and 2800 ms. The memory.set budget, the restricted-access docstring, the batched-insert comments and the failure-and-recovery page describe batched inserts as waited, not committed, and the batched-insert rule as waited inserts that need no fence or identity token. The runtime-override test fakes go back to their optional settings argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Batched inserts are described by the rows that use them, and write_rate.py cleans up its client rows The setting, client and helper text name the inserts sent with batched_insert_settings() instead of defining them by what they lack, the async-insert dedup rule names both kinds, the restricted-access audit docs say the row is a batched insert, the failure page lists the two wait settings in the client table, and write_rate.py runs under a per-run org and deletes its in-process conversation rows at the end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Async inserts are described as deduplicated by token, and every batched insert site is pinned normalize_insert_settings and the clickhouse.md bullet say dedup_settings turns async deduplication back on with the per-batch token. The conversation append, the graph Undo markers, auth_attempts, memory_projection_failures and memory_usage_events are pinned to batched_insert_settings(); the usage-event fallback insert is batched like its flush. The restricted_audit_inline comment names the awaited insert's whole latency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Async inserts are deduplicated by token without a separate normalising step, and write_rate.py sends its baseline directly normalize_insert_settings set async_insert_deduplicate = 0 only for dedup_settings to set it back to 1, so it is removed. The batched-insert wait settings describe the writes they affect in two lines each. The write-rate bench passes committed or batched settings to its inserts instead of reassigning a store attribute, and deletes its client rows in a finally block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Rows with a Quantized-coded vector stay committed, and the session dedup comment says what each insert sends A mis-sized vector fails the shared part of a batched flush and with it every other insert in that flush, so conversation_embeddings, entity_vectors and memory_chunks keep committed inserts. The conversation.append and set_from_messages budgets drop to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * The policy witness writes its rows, and write_rate.py reads an unset async_insert as the server default The witness sends batched_insert_settings(), so its helper, docstrings, log lines and test say the row is written, not committed. write_rate.py's inserts_sent read an insert sent at the server's default async_insert (1 on recent releases, which the query log leaves out) as 0; it now reads the default from system.settings. Two long lines are re-wrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * A waited entity-graph insert uses the batched-insert settings, and batched inserts have their own section graph_insert_settings(wait=True) returns batched_insert_settings(), so the live ch_batched_insert_wait_* bounds govern memory.set's waited entity_triples inserts too. The batched-insert convention moves into a "Batched inserts" subsection of the ClickHouse page, with the tables that stay committed listed one group per line, and the failure-and-recovery Inserts row links to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * write_rate.py deletes only its own run's rows, and batched inserts send the adaptive busy timeout The client-target cleanup scopes its delete to the run's thread ids, so an org named with --org keeps its own transcripts. batched_insert_settings sends async_insert_use_adaptive_busy_timeout = 1, so the min and max wait bounds hold whatever the server profile sets. The duplicate design-principles bullet is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Pin the tables that stay committed, and give entity_canonical_map's reason A block-level error in a shared flush fails every insert waiting on it, so the Quantized-coded vector tables and the org-partitioned canonical map stay committed; a test now pins each of their insert sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Batch the conversation embedding row when its width cannot fail the flush A conversation_embeddings row is batched when the column is plain, or coded at the served model's embedding_dim with a vector of that width; it stays committed without a reading or while a model change or rebuild is pending, and a batched row whose flush another width failed is sent again committed. The usage-event recorder loses its unreachable direct-insert branch, and the p99 budgets of the ops that await a batched insert (memory.delete, memory.delete_version, memory.pin, memory.get, memory.get_by_id, conversation.append) cover its wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Describe memory.pin's batched inserts, share the committed insert settings, split the usage-buffer lock from the flusher start memory.pin's budget comment names the same inserts as memory.set. The committed insert settings in the conversations store are one module constant the identity-token append and the bench reuse. The usage store's buffer lock comes from _buffer_lock(), so _ensure_usage_flusher_started only starts the flush task and the analyzer can stub it with a no-op. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Give memory.update, update_by_id and restore the outbox wait allowance, and name the batched wait as an OVERLOADED cause memory.update, memory.update_by_id and memory.restore await the same batched outbox insert as memory.delete, so their p99 budgets gain the same 50 ms allowance. The troubleshooting entry for OVERLOADED names batched inserts holding their connection and the settings that bound it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Check every committed-table insert under server/, give memory.pin memory.set's budget, reuse the committed insert settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Batch inserts by default behind a live switch, give background inserts a longer wait ch_batched_inserts (live, on) sends every batched insert committed when off. Inserts no request waits on (the usage-event flusher, the write-mark flush and process-start mark, webhook delivery attempts) wait up to ch_batched_insert_background_wait_max_ms (200). memory.bulk_get and memory.history get the batched audit row's allowance, COMMITTED_INSERT_SETTINGS moves to ch_writes, and the docs state ClickHouse's own target and when to turn batching off. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe what the batching switch leaves async, size the pool by mechanism, test batched settings by value - ch_batched_inserts' description and clickhouse.md: off sends the inserts that would be batched as committed inserts; fire-and-forget inserts are unchanged. - clickhouse.md states the pool cost as a formula instead of one run's figures. - insert_embedding tells a batched insert by its async_insert value, and the settings helpers hand out a copy of the committed settings. - memory.set's budget comment names all three batched inserts it awaits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Switching batching off sends each insert as before, pin each table's wait window With ch_batched_inserts off, the sites that were waited async inserts before batching (auth attempts, carry acks, the write-mark and usage flushes, the policy witness, projection failures, graph Undo markers and waited graph inserts) pass async_when_off and stay async on the profile's timeouts; the rest are committed. The batched-tables test pins each table's wait window and checks both switch states. COMMITTED_INSERT_SETTINGS is exported, and the unreachable no-loop branch of the usage flusher start is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Describe the batching switch by what it does off, pass the shared insert settings as they are Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Test each switch-off insert form, hand out copies of the shared insert settings With ch_batched_inserts off, the policy witness, the Undo markers and the waited graph insert are tested as waited async inserts and a plain conversation append as committed. batched_insert_settings and embedding_insert_settings return a new dict on every path. Rewrap the restricted_audit_inline description, its handler's docstring and the batched-inserts section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Pin each usage-event flush's wait window, hand out committed settings from a function, size the pool per window The flush a full buffer forces inside record_usage_event is tested to take the short window, and the flush loop's tick and the shutdown drain the background one. committed_insert_settings() returns a new dict per call, as batched_insert_settings() does, so no caller shares a mutable module dict. The pool-sizing rule counts both wait windows, and the OVERLOADED fix names the background wait and the batching switch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Send async_insert_max_query_number with a batched insert, describe the switch by what it does off, note why a batched table takes no row that can fail its flush Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Keep the ledger flushes on the batched bounds with the switch off, budget bulk_set's two waits, describe the switch by what it does The sites that were waited async inserts already (carry acks, auth attempts, policy witness rows, projection failures, graph Undo markers, waited entity triples, usage and write-mark flushes) now call async_insert_settings, which ignores ch_batched_inserts; the switch governs only the sites it commits. memory.bulk_set's budget covers its audit and outbox waits. The docs scope the committed resend to conversation_embeddings, and the usage docstring says the call that fills the buffer waits for its flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Mark the async-insert buffer bound on the line above it, pin the usage flushes for both switch values The `# not a setting:` marker of _ASYNC_INSERT_MAX_QUERY_NUMBER now sits directly above the assignment, where the tunable-constants ratchet looks. The usage-flush tests run with ch_batched_inserts on and off, since those flushes ignore it, and the live-ClickHouse test names its settings dicts by what they hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Name the live path for the insert waits under OVERLOADED, describe the wait bounds by both their uses, mark the buffer bound in one line Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Test a mis-sized vector in a shared flush on the coded table, budget save_message's waits, state the pool formula as a lower bound Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Ship batched inserts off by default, call the store from the write-rate bench, pin the write marks through a recording client ch_batched_inserts now defaults to false: a waited async insert holds its pooled connection for the window and the flush, and one process at 200 two-insert writes a second refused writes at ch_pool_maxsize = 64 where committed inserts refused none. The docs name the pool-sizing step before turning it on and list every insert the switch covers. The write-rate client arm calls conversations.append and insert_embedding, and --committed switches ch_batched_inserts off. The live-ClickHouse tests run with batching on and a 1 s window so their inserts share a flush. The write-mark test drives bump, flush_now and mark_process_start with a recording client under both switch values. The broad except in insert_embedding states why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Ship batched inserts on by default, state the pool cost under the operator steps ch_batched_inserts defaults to true again; LOOM_CH_BATCHED_INSERTS=false sends the inserts main committed as committed inserts. clickhouse.md and troubleshooting state that a waited batched insert holds its pooled connection for the wait, that the measured 200 writes/s single-process run once ended with 212 OVERLOADED at ch_pool_maxsize = 64, and that a process at that rate raises ch_pool_maxsize or turns the switch off. The op budget comments say their waits apply with ch_batched_inserts on. The wait-bound rows in failure-and-recovery.md cover the waited async inserts too. The write-rate docs name the batched default arm and the committed arm. The width test compares each read-back row with the row it was built from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Size the pool for batched inserts before upgrading, budget memory.evidence's audit wait, read every embedding column back The upgrade steps, the ch_pool_maxsize and ch_batched_inserts comments and the tuning and failure tables now say a batched insert holds a pooled connection for its wait, so the pool grows with the write rate. memory.evidence gets the 50 ms allowance for its high-sensitivity audit row, as memory.get does. The shared-flush test reads every written column with a count per row instead of DISTINCT, so a row that lands twice fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Define the pool-sizing terms for batched inserts and the check after rollout The upgrade step defines R_request and R_background, says the measured need was more than 3 times the bound, and names the pool gauges to watch. The architecture page's example is a reachable request-path rate, the tuning guide lists the minimum wait, and the width-error catch keeps the re-raise comment form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Shorten the batched-insert and restricted-audit setting comments, name compaction's summary turn among batched inserts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Send a follow plan's successor inserts on the batched bounds, scope the switch's description The follow plan's three successor inserts now use async_insert_settings(), so ch_batched_insert_wait_max_ms bounds them; a test pins it with the switch on and off. The ch_batched_inserts description names the request-path sites it covers and that ledger flushes stay waited async inserts either way. The op budgets state the batched-insert wait once and name each op's waits in one line, and the usage-event docstring says when a failed flush drops its batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Cite the pool exhaustion the committed write-rate harness reproduces At 200 writes a second in one process, bench/write_rate.py with batching on refuses 3,367 of 12,000 writes with OVERLOADED at a pool of 20, the formula's bound, and none at 40 or 64. The upgrade, troubleshooting and batched-insert pages cite that run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Keep the write marks' erasure bump and process-start mark committed, state the pool step as a floor Both write one row per erasure or per boot, so batching them saved no parts and only added a wait. The upgrade step now names the default pool of 64 and when to raise it, and the batched-insert and pool setting comments fit two lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Committed inserts use committed_insert_settings, the usage flush takes its lock through _buffer_lock The three inserts that stay committed (entity_vectors, memory_chunks, entity_canonical_map) call the helper instead of repeating its dict. The ch_pool_maxsize comment names the base sizing again, and the troubleshooting pool sentence is split in two. Co 523F -Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * The ch_batched_inserts description names the inserts it batches and the ledger flushes it leaves waited Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * The ch_batched_inserts description names the inserts the switch leaves waited, the pool step uses one x3 rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * The pool step sizes from its own rule, the batched tables are listed once, a non-width batched failure is not resent Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * memory.move's budget covers its redirect's outbox-row wait, the pool sizing step reads cleanly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Pool sizing counts a memory.set at six waited inserts, and the batching switch's comment names what it batches The upgrade step, the batched-inserts section and the PR's operator steps count the audit, outbox, two entity-edge and two entity-triple inserts (plus carry_acks) per memory.set, and say the graph inserts stay waited with ch_batched_inserts off. The memory.set budget comment names the edge-projection wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Op budgets state when a batched-insert wait raises one, and the pool step names its margin once Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Batched-insert waits are capped well under the write deadline, and the pool sizing rule lives on one page The three wait settings take an upper bound (1000, 5000 and 1000 ms), the ch_batched_inserts description names the tables it batches and the pool sizing it needs, and upgrades.md and troubleshooting.md link the sizing rule in clickhouse.md instead of restating it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Batched-insert setting comments take two lines and point to the architecture page The table list and the wait bounds the fields already declare live on docs/architecture/clickhouse.md#batched-inserts, not in the comments the reference prints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * The batched-insert switch names the inserts it covers, and memory.move keeps its 400 ms budget ch_batched_inserts names the audit, transcript, agent_events, outbox and evidence inserts it switches, and says the waited async inserts keep their windows. The clickhouse page lists a follow plan's successor inserts and the carry-ack withdraw among those. memory.move stays at 400 ms, as the budget rule says, and memory.restore's comment names the Undo's audit wait that every namespace pays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * The pool-sizing paragraph reads in order, and the batching switch's comment points to its list Move the lower-bound sentence next to the formula it qualifies and say a process above the default's rate acts. Cut the ch_batched_inserts comment to two lines and regenerate the settings reference, schema and example. Keep the literal committed settings at the sites batching does not touch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Committed inserts in the write-marks and webhook stores use committed_insert_settings() The session-settings comment states the rule and points to the batched-inserts list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Put the pool-sizing rule in the settings comments, name fire-and-forget inserts in the session-settings comment, give memory.set's budget its current reason Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Point the pool comment at the full sizing rule, keep the upgrade procedure release-neutral, name committed_insert_settings among the entry points Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Keep webhook delivery attempts committed, pin that each resent embedding row went batched first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Drop batched_insert_settings' unused background flag, order the wait bounds min first, name the wait setting rather than its default in budget comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Name the inserts that take the background window, order the wait rows min first, give each raised budget a one-line reason Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Name the wait settings by field, not position; make each usage flush say its window; match table constants within their own module Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> * Give each batched table one window in its test, use committed_insert_settings for the thread i…