Repository navigation
Background worker (valkey/redis requirement) #6442
Description
Is your feature request related to a problem? Please describe.
Currently, several background tasks pile up that need to be handled by either NestJS's cron hooks or other means of regular scheduling.
These tasks run in the same process and thread as the main app, therefore sitting in between the running event loop of Node. This can create performance issues with the core functionality while such a background task runs.
Another problem is that several actions depend on up-to-date data in the database while this data is still in memory instead of being persisted to the database. For example, updating the title of a note and then quickly jumping back to the explore page still shows the old title on the explore page, since the change is still in memory (the persistence of opened realtime documents runs later).
Describe the solution you'd like
We introduce a background worker that runs in another process, decoupled from the main app. Actually even HedgeDoc 1.x even has some sort of parallel worker running.
After some comparison in the Node ecosystem, I came to the conclusion that BullMQ is providing everything we need, is well-established and seems in good maintenance. It has an official NestJS integration making the integration into our backend easier.
BullMQ comes with a cost however, it requires Valkey or Redis (or any other compatible software) as a deployment-dependency. Adding a new deployment-dependency to the stack is not an easy step, but on the other side we can benefit from Valkey/Redis in other parts of HedgeDoc as well.
The idea is to add a new config option HD_ENABLE_INTEGRATED_WORKER or similar. When this is set to true (the default), the backend spawns a new worker process using child_process.spawn. This worker runs the BullMQ worker which subscribes to the Valkey/Redis to get tasks in realtime and then execute them.
If the config option is disabled, it's the instance owner's responsibility to run (at least) one worker process next to the main HedgeDoc process. This could be done for example by setting command: "worker" or similar for the docker container.
The other benefit is that we could use another virtual database of the same Valkey/Redis instance to store temporary note contents and metadata until a new revision is created. This temporary data could then be used by API methods that need to show near realtime data (like the note list on the explore page).
Of course, there are other benefits like being able to cache some data which is heavier to compute, like permissions for a given note and user for example. However, this is just a nice side effect and should not be considered a strong argument for this.
A list of background tasks (there could be more that I've forgotten right now):
- revision saving for active notes (as long as the realtime data is dumped into Valkey/Redis)
- regular updates of the OIDC provider configs
- regular clean-up of expired API access tokens from the database
- regular clean-up of expired sessions from the database
- regular clean-up of old/unused guest users from the database
- regular clean-up of old note revisions (if configured)
- regular clean-up of old guest notes (if configured, see Automatically delete guest notes after 30 days #2967)
- for the future: creating backup archives
Describe alternatives you've considered
There are several alternatives:
- The methods invoked by NestJS's cron could spawn a new thread using Node's worker_threads. The Node documentation however recommends doing this only for CPU-heavy tasks, not for IO-heavy tasks. Most of our clean-up tasks are database-bou 8AE2 nd and therefore IO-heavy.
- The queue for the background worker could be implemented using a table in the database. This has the drawback of needing to pull the database for updates to the queue in regular intervals. Pulling creates unnecessary resource strains and adds artificial delay between task scheduling and execution, which might become relevant in case of revision saving.
- Content from the currently opened notes could be dumped very frequently from the in-memory Y.Doc into a temporary database table until it is saved as a proper revision. This has the drawback of having very increased write-loads to the database, especially when many notes are actively edited at the same time.
- Create revisions much more frequently. Currently we aim to reproduce HedgeDoc 1.x's revision system of saving a revision about every 10 minutes. The interval is already configurable. Setting a more frequent default increases write-load on the database, increases storage requirements for the database and degrades the user's UX when they browse through the revisions to catch up what changed in a note.
Additional context
No response
Activity