Repository navigation
The bundled ClickHouse bounds how long query_log and text_log keep rows #436
Description
Component
Deployment (bundled ClickHouse config, Helm dev ClickHouse)
Use case
A refused insights.query statement is logged with its ClickHouse error, and that error can quote a value it failed on: SELECT toUInt64(title) FROM loom.documents refused on a title quotes the title. The value is one of the caller's own allowlisted metadata columns (a memory key, a title, a user id), never a memory value or document text. It lands in system.query_log.exception and in the server log (system.text_log). Erasure reaches neither.
ClickHouse's default configuration, which the bundled ClickHouse uses, sets no TTL on query_log or text_log, so those rows stay until an operator bounds them. docs/architecture/assistant.md ("Stored data and erasure") describes this. The server log already kept these errors before analyst statements were logged to query_log.
Describe the solution you'd like
Ship a retention for system.query_log and system.text_log in the bundled ClickHouse config (deploy/clickhouse-config/) and the Helm chart's dev ClickHouse, for example 30 days, and name the setting in the deployment docs for self-managed servers.
Changing a system log's definition makes ClickHouse rename the existing table (query_log_0) and keep it, so the change needs an upgrade note on what happens to the old table.
Describe alternatives you've considered
- Turning query logging off for the analyst: ClickHouse Cloud pins
log_queriesandlog_queries_min_typefor its users, so Loom writes one profile shape for every install. - Documenting the operator step alone, which the architecture page does today.
Where the need comes from
Found while making insights.query work on ClickHouse Cloud.
Activity