Repository navigation
feat(sandbox): share the office desktop with people through /pi-desktop - #178
Merged
Merged
Conversation
Deploying mikan with
|
| Latest commit: |
77d5321
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://2c0f8671.mikan-1wv.pages.dev |
| Branch Preview URL: | https://feat-office-desktop.mikan-1wv.pages.dev |
…he composition root
… hand-off to direct messages
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Members of a conversation can open its office desktop from a private
/pi-desktoplink: a headless Wayland desktop (Sway + wayvnc) in the managedocisandbox, shown with noVNC behind mikan's web server, where the browserjev_browserdrives appears as a normal window. Browser control follows ego-browser: a person can take control (the agent's browser calls stop) and hand it back, and the agent can hand a step such as signing in to the person withjev_browser'shandOff. Design and measurements: ADR 0025 anddocs/research/browser-sharing-2026-10.md.New dependency:
@novnc/novnc(MPL-2.0, 0.6 MB, no dependencies), approved; only itscore/andvendor/modules are served.Architecture
Components
flowchart LR subgraph People chat["Slack / Telegram / Discord"] viewer["Person's browser<br/>/desktop page + noVNC"] end subgraph Host["mikan host process"] cmd["/pi-desktop<br/>DesktopCommandHandler"] resolver["ActorExecutionResolver.prepare<br/>(provision office container)"] tokens["InMemoryDesktopTokenStore<br/>person + office, 24 h"] web["web server<br/>GET /desktop · /desktop/novnc/*<br/>GET/POST /desktop/control<br/>WS /desktop/ws"] od["OfficeDesktop<br/>start · status · control · connect"] prov["DockerContainerManager<br/>retain() blocks idle stop"] jev["jev_browser<br/>status check · handOff"] end subgraph Container["office container (oci sandbox image)"] script["mikan-desktop script"] sway["Sway (headless, pixman)"] vnc["wayvnc<br/>127.0.0.1:6080 WebSocket"] chrome["Chromium via agent-browser<br/>headed on Wayland when desktop runs"] term["foot terminal"] ctl["/dev/shm/mikan-browser-control<br/>agent | user"] end chat -- "/pi-desktop" --> cmd cmd --> resolver --> prov cmd --> od cmd --> tokens cmd -- "private link" --> chat viewer -- "page, control (token)" --> web viewer <-- "WebSocket (token)" --> web web --> tokens web --> od od -- "engine exec / exec -i" --> script od --> prov script --> sway --> vnc sway --- chrome sway --- term script --- ctl jev -- "pi-env exec: mikan-desktop status / control user" --> script jev -- "pi-env exec: agent-browser (+ Wayland env)" --> chrome jev -- "handOff: step + link via responder" --> chat<engine> exec -i <container> mikan-desktop connect, like the pi-env connection.<engine> execfrom the host, not pi-env, because pi-env kills the process groups it started when its connection ends./dev/shm, cleared on container restart), sojev_browserreads it with the same exec it already uses, and no host state links the web server to a run.Flows
sequenceDiagram autonumber actor P as Person participant S as Slack participant M as mikan host participant C as office container participant A as Agent run (jev_browser) P->>S: /pi-desktop S->>M: command M->>C: provision (replace a stopped outdated container) M->>C: exec mikan-desktop start alt image has the desktop M-->>P: private link /desktop?token=… else container still runs an older image M-->>P: "available after the sandbox idles out" end P->>M: open link, WebSocket /desktop/ws M->>C: exec -i mikan-desktop connect (retain container) C-->>P: desktop frames (wayvnc) A->>C: mikan-desktop status C-->>A: display wayland-1, control agent A->>C: agent-browser … (headed on Wayland) P->>M: POST /desktop/control owner=user M->>C: mikan-desktop control user A->>C: mikan-desktop status C-->>A: control user A-->>A: returns person-has-control, no browser command A->>C: handOff: mikan-desktop control user A-->>S: step + desktop link, turn ends P->>M: POST /desktop/control owner=agent P->>S: "done" S->>A: next turn continues in the same browserHow I tested
npm run lintnpm test(full gate: lint, fmt, knip, build, typecheck, tests)npm run build/pi-desktopin a DM/desktop?token=linkRFB 003.008handOffE14 ran with a 1.4.0-era image without the desktop as the "before" image, because the local 1.4.0 image could not be re-pulled during the run.
2034d519. Earlier runs lost S-021 (the model mistyped a long attachment path) and, once, S-011, S-018, and S-019 to a second daemon connected to the same Slack app.Review follow-up (second commit)
A code review found three blockers and several smaller issues; all fixed with tests:
The WebSocket upgrade socket had no error listener, so a client reset during refusal or desktop start crashed the daemon; it now gets one first, and URL parsing moved inside the handled chain.
/desktopin a shared conversation on a platform without private replies (Telegram groups) would have posted the link publicly; it now asks for a direct message.handOffon a browser started headless handed over an invisible browser; it now starts the desktop and reopens the current page on it (without the headless browser's cookies).An open connection closes when its link expires; control returns to the agent when the last viewer leaves; a failed hand-off post returns control.
mikan-desktop startis serialized withflockand restarts only wayvnc while Sway lives; hostexeccalls time out after 20 s.Without the desktop, the tab limit counts every tab again; with it, only tabs the tool opened. Status is checked only in
ocimode, andhandOffmust be used alone.WAYLAND_DISPLAYis passed as an absolute path instead of settingXDG_RUNTIME_DIR, because agent-browser keeps its session sockets underXDG_RUNTIME_DIRand other calls would not find the session./pi-desktopandhandOffopen the desktop through oneOpenDesktopservice (src/harness/desktop.ts) that the composition root builds and hands to the command and to runners, so the command no longer builds a harness resolver; inocimode the resolver's key is the office key that tokens carry.ADR 0025 states that desktop links are bearer tokens like
/sessionand/adminlinks; binding links to a signed-in person is left to a later change of the portal token model.The acceptance eval was rerun after these fixes; every row still matches. In the eval, E7 now calls
handOffalone and E8 opens the desktop first, following the tool contract.Second review follow-up
handOffdecides whether to reopen the page from agent-browser's lifecycle (launched/relaunchedBrowser), not from the tool's own flags, so a page opened with raw commands or a browser relaunched headless after an idle stop is moved onto the desktop. The page reopens under a new session name, because agent-browser's per-session daemon keeps the environment it started with and a headless daemon cannot launch Chromium on Wayland; verified in a real container (headless page, thenhandOff: one Chromium window with the page on the desktop, controluser). With no page open,handOfffails and asks the model to open one withurl.handOffworks only in direct messages, where the link can be sent privately; in shared conversations it fails without touching control.OfficeDesktopcounts viewers and gives control back to the agent 10 s after the last one leaves, so a reload keeps control; the web adapter no longer keeps its own count.Notes for reviewers
deploy/docker/mikan-desktopscript; the host runs it with<engine> exec, never through pi-env, whose daemon kills the process groups it starts when its connection ends.jev_browserrunsmikan-desktop statusbefore each call (one extra exec); a failed call means no desktop, as in host mode or an older image./pi-desktopadded to their manifest.