A user opens a web page, signs in with a Microsoft account, picks a notebook, and downloads it as a .zip of Markdown. No MSAL, no Graph API — the automation drives the real OneNote web UI through Playwright, because Microsoft does not offer an API for this.
The hard part is not the export. It is that a user's Microsoft password must reach a browser-automation process and nothing else, on infrastructure the user does not control. Almost every decision in this codebase follows from that one sentence.
The password is held in browser memory, sent cross-origin to exactly one build-time-configured origin, streamed byte-for-byte through two proxies, and handed to a Playwright driver inside a throwaway container.
It is never parsed, never JSON-encoded, never turned into a string, never logged, never retried.
The capability to hold a password, the capability to reach the internet, and the capability to create containers on the host are three different processes on three different networks.
No single compromise reaches two of them.
The backend's public surface is deliberately small and its authority surface is a closed set of verbs, because a browser-facing process is the one most likely to be found by a scanner.
And it makes almost no decisions about authorisation itself.
There are two ways to use this system, and they are not two codebases — they are the same three libraries wrapped in two different shells. The libraries are the engine; everything else is packaging, and none of it is ever forked.
auth.json —
a Playwright storageState blob, {cookies, origins}.
login() writes it; listNotebooks() and
runExport() both feed it straight into
browser.newContext({storageState}). Nothing else crosses between them.
ms-onenote-exporter login → list →
export. One process, one Chromium, your terminal, your machine.
The umbrella CLI is a pure translation layer: rename options, call one function,
map the result to an exit code.
| Step | Calls | Reads the result |
|---|---|---|
| login | login() | the boolean |
| check | checkAuth(url, file) | the boolean (positional, legacy 2-arg form) |
| list | listNotebooks() | the returned array — an empty list is success |
| export | runExport() | the returned stats via exitCodeForStats() — deliberately not process.exitCode |
The CLI reads the return value rather than the process exit code because
runExport() sets process.exitCode as a side
effect. Reading the side effect would make the library's contract unobservable and would
leak state between callers.
The same three libraries, but invoked from inside a disposable container that a Go
orchestrator creates per session. The browser talks to a Node API; the API talks to the
orchestrator; the orchestrator creates the runner; the runner is the only process that
touches the password and the only one that may import an @msout/
package at runtime.
| Component | Language | The one thing it is |
|---|---|---|
| frontend | React 18 + Vite | static files. no state, no secret, no authorisation logic |
| caddy | Caddy 2.10 | TLS termination and a sendfile file server |
| api | Fastify 5 + node:sqlite | the authority — session state, the SSE hub, the CSRF wall |
| orchestrator | Go 1.25, zero deps | the pool. the only holder of docker.sock |
| runner | Fastify 5 + Playwright | the throwaway container holding the credential and Chromium |
@msout/microsoft-webauth, @msout/microsoft-onenote-list-notebooks,
@msout/microsoft-onenote-export-notebook, archiver,
fastify — and fails the build otherwise.
This is the diagram everything else hangs off. Click a network row on the right to trace its
members; click a container to read its brief. The single most consequential word in the whole
compose file is internal: true — it is what makes
"the api cannot reach the internet" a property of the topology rather than a rule somebody has
to remember.
Each box above is one process with exactly one job. Hover to highlight, click to read its brief — including what it is structurally prevented from doing.
vacant means "no container and no create in flight", and it is deliberately distinct from starting. Conflating the two made every release shrink the pool permanently.
A claim creates a bound container rather than relabelling an idle one — an idle
runner holds a tmpfs at /data, not a vault. Holding the empty
artifact directory is not a capability; the vault bind is exactly what makes a container
credential-bearing.
| On the host | Contains | Who may touch it |
|---|---|---|
| /srv/msout/vault/<guid>/ | auth.json, notebook cache, out/, logs | orchestrator rw · runner rw |
| /srv/msout/artifacts/<id>/ | vault.zip or vault.partial.zip | orchestrator rw · runner rw · caddy read-only |
| /srv/msout/artifacts/.staging/ | in-progress archives | never servable, by construction |
| /srv/msout/data/api.db | SQLite in WAL mode | api only · named volume |
auth.json — not because it is told not to,
but because the path is not mounted into it.
The table on the left lists container-side names. On the host they resolve differently for each service, and only the orchestrator sees the identity that table used to claim for all of them.
| Service | Host source | In-container |
|---|---|---|
| orchestrator | /opt/msout/data/vault | the same path |
| caddy | /opt/msout/data/artifacts | /srv/msout/artifacts :ro |
| api | named volume msout_api-data | /srv/msout/data |
VAULT_HOST_DIR and ARTIFACT_HOST_DIR must be
absolute host paths the api has also mounted. As named volumes they resolved to
/var/lib/docker/volumes/…/_data, which the orchestrator did not know
about — so the runner mounted an empty, root-owned directory and every session
failed with EACCES: mkdir '/data/<guid>' while
/healthz cheerfully said ok.
Press play and watch a packet move across the topology above. Nine hops, four processes, three of which never learn what the packet says. The narration on the right names the mechanism at each hop — and the reason it is written that way.
Every row below is load-bearing, and every one of them has shipped as a real bug at
least once. The password is bytes from the browser's
<input> to the Playwright driver — never a value in any
process along the way.
| Rule | What it prevents |
|---|---|
| no JSON encoding of the body | hunter2 becoming "hunter2" — a password containing a quote or backslash arriving as a different password |
| no "equivalent" copy | a byte-identical copy that is still a value living in a process |
| account travels as a header | a delimiter in the body — and a delimiter is another truncation surface |
| framing checked twice | one refactor away from a 4 KB → 256 KB hole |
| cap created on the refusal path too | a caller who understates Content-Length and then sends 4 MB |
| no retry on this route | replaying a password defeats the point of a single submission |
| no trimming | a trailing space is genuinely part of the password |
| the api never logs a body | the credential route's body is the password |
The GUID identifies a session. The 43-character secret authorises it. Both are minted in the browser, which sounds wrong until you notice what it buys: the frontend never has to be trusted, because the server validates the shape mechanically and stores only the hash.
| Fact | Value | Note |
|---|---|---|
| session id | GUID | generated client-side, validated as a UUID before it is ever joined to a host path |
| session secret | 256-bit base64url = 43 chars | stored only as sha256(secret), compared with timingSafeEqual |
| artifact id | 32 random bytes, base64url, 43 chars | minted by the api, not the runner — an id derived from the GUID would put it in every download path and every Referer |
| cookie | __Host-msout | HttpOnly · Secure · SameSite=None · Path=/ · no Domain allowed by the prefix |
| secrets on the host | files, mode 0400 | environment variables carry paths, never values — an env var is visible in docker inspect |
This is what the runner does once it holds the credential. It calls
runExport() — and that single call is a recursive descent through OneNote's
own DOM, clicking through a real web application one navigation item at a time.
Expand any node to see what happens inside it.
The old implementation slept a fixed 3 seconds after every click and hoped. The exporter now proves the page it wants is the page on the canvas, with a settled image count, using a signature that must be identical to the previous observation.
[previous] → [previous+previous] →
[] → [wanted+wanted] →
[wanted]
| Observation | Verdict |
|---|---|
| 0 outlines | never settled |
| 0 titles | settled — an untitled page is legitimate |
| > 1 title | still mid-switch, showing two pages at once |
| images > imagesReady | content still decoding — not settled |
| title matches, normalised | settled |
aria-expanded is read
before clicking. The original code clicked an already-expanded group and
collapsed it — measured, it lost 8 pages in one notebook.
notebookNotFound rather than reporting a cheerful zero-page
export. The summary then never prints "Export complete!".
<name> [passProtected] is still created, so
the omission is visible in the tree rather than silent.
exitCodeForStats(stats) returns 3 when anything was
lost — groups, sections or pages. A missed asset never changes it, because a
linked-but-undownloaded file is recoverable on a re-run and is called out in the note body
instead.
export-progress once per section — deliberately,
because a per-page firehose would drown the log. The hosted edition therefore re-publishes
the last known counts every 15 s as a heartbeat, which it states plainly: it cannot invent
counts inside a section.
One page in, one Markdown file out. Here is the actual pipeline, with a real DOM snippet on the left and the bytes it becomes on the right. Hover any element to isolate its contribution.
| DOM source | Becomes |
|---|---|
| .OutlineContainer | a block, in visual order |
| .WACEFContainer | an attachment — a real downloaded file |
| .WACImageContainer | an image, saved locally |
| .TitleOutline | the file name |
| .TitleDateTimeOutline | the first line of the body |
| iframe[src*=youtube|vimeo] | an embed link |
| a[href^=onenote:] | an internal link — resolved in a second pass |
| Markdown | Why |
|---|---|
![[assets/x.png|alt]] | Obsidian wikilink, not a relative path — the output is meant to be opened as a vault |
[[assets/report.pdf]] | an attachment becomes a link to the file beside it |
<!-- onenote-link:id --> | a placeholder — the target path is not knowable until the whole walk finishes |
[[Real/Path|Text]] | written in a second pass over every emitted file, once all paths are settled |
> ⚠️ n assets… | an honest failure note — rebuilt every run, so it disappears when a re-run succeeds |
| Strategy | How | Cost |
|---|---|---|
| direct | sharepoint / OneDrive only — opens the cloud page, clicks through the Office viewer to Download a Copy | ≈72 s |
| ui-click | double-clicks the chip in the OneNote canvas, races the download against a popup | seconds |
| fallback | a plain request for the source URL, rejecting any text/html | fast |
Duplicate Title.md and
Duplicate Title_1.md — the used-name sets are per-run and never
consult the filesystem. That is what stops report.pdf_1.pdf chains
from a previous bug.
The @msout packages report what they saw on screen. The
api reports what is true of a session. Those are different claims, and neither
side can see the other's vocabulary — which is exactly why the runner is a deliberate seam rather
than a thin pipe.
Click any row to see the reasoning.
EventSource with
withCredentials: true. The api writes named frames, so
the transport registers a listener per contract event name and keeps
onmessage only as a fallback for an unnamed one.
EventSource cannot send
custom headers — which is precisely why the CSRF header check on mutating routes
is a security control rather than hygiene. And a transport that only listens on
onmessage will silently receive nothing from a server
writing named frames.
login-started on acceptance. A pump attached after the
response misses events already emitted — and the symptom is a login that works, with no
challenge shown and no failure reported.
The package states its own contract plainly: "what this package owes the caller is the
truth… labelling is the caller's job." It emits export-done the
moment the walk ends and returns its statistics
separately.
| After the call returns | api outcome |
|---|---|
| the walk threw | failed · nothing staged |
| notebookNotFound | failed · nothing staged |
| items lost, no pages written | failed · nothing staged |
| items lost, some pages written | partial · staged and labelled |
| otherwise | done · staged |
export-done before it returns, so the counts do not exist
when the event fires; and the api dedupes terminals by artifact id, so whichever arrives
first is the only one acted on. Without this the api is told only
pages: 0 and answers done — a 130-byte empty zip
presented as a successful download. Which is exactly what happened on the deployed host.
A boundary nobody can violate is worth ten boundaries everybody must remember. Each invariant below names the mechanism that enforces it and, where one exists, the assertion that catches its removal. Expand a card to read it.
Two facts about the running system exist in two different places, and each has exactly one reconciler. Neither trusts the other's view alone.
| Fact | Authority | Reconciler |
|---|---|---|
| session state · expiry · notebook list · export state · artifact id | SQLite | the api's sweeper — every 30 s |
| container existence · slot binding · artifact presence on disk | Docker labels + the filesystem | the orchestrator's pool — adopts on boot, never deletes |
409,
recoverable only by restarting the orchestrator.
JSON.stringify on a password body;
an event pump left pointing at a runner the session had been rebound away from — every signal
green, and the export simply lost.
| Service | Value |
|---|---|
| caddy | :80, :443 — the only published ports in the stack |
| api | 0.0.0.0:3000, unpublished |
| orchestrator | :9100, unpublished |
| runner | 0.0.0.0:3100, unpublished, addressed by container alias |
| protocol version | 3 |
| signing replay window | 60 s |
| State | Deadline |
|---|---|
| unclaimed | 10 min |
| login in progress | 15 min |
| authenticated & idle | 30 min |
| absolute | 12 h |
| exporting | no idle deadline |
| erasing | never touched |
| Knob | Value |
|---|---|
| recycle budget | 5 min |
| slot idle | 30 min |
| ready timeout | 45 s · 3 s probe |
| pool size | compose default 2 · built-in 4 |
| body quota | 8 GB · 2 GB min free |
| export timeout | 1 h · login 5 min |
| credential cap | 4 KB |
vacant is not starting| State | Means | Transition |
|---|---|---|
| vacant | no container and no create in flight | → starting |
| starting | a create is on the wire | → idle when healthy |
| idle | a warm container holding a tmpfs, not a vault | → bound |
| bound | claimed by exactly one session, with the vault mounted | → draining |
| draining | releasing | → idle or dead |
| dead | gone | → vacant |
vacant into starting made every
release shrink the pool permanently. And a claim creates a bound container
rather than relabelling an idle one — because an idle runner holds a tmpfs at
/data, not a vault. An idle pool one claim away from
auth.json is the wrong shape: holding the empty artifact directory is
not a capability, but the vault bind is exactly what makes a container
credential-bearing.
--profile runner flag is mandatory — without it,
build runner "succeeds" without building and
up -d prunes the network every login needs.
And a .env containing $(…) is a silent total
outage: the pool cannot create a runner, so every login 409s while
every other signal looks healthy.
| Question | Answer lives in |
|---|---|
| how does anything talk to anything? | the compose file — its header comment explains the hostname |
| what is the design, and why? | the backend's planning document |
| what is broken or unfinished? | GitHub issues — the one consolidated list, so a document and reality cannot disagree |
| what is each service for? | the four per-service READMEs, read in order: edge → api → orchestrator → runner |
| why is the topology like that? | the infrastructure README |
| what does the browser send? | frontend/src/lib/api.ts |
| what does the user see? | the frontend's App.tsx and pages/ |