Skip to content

0025: Blob wire frames, the sync endpoint and the client module boundary

  • Status: proposed
  • Date: 2026-09-25
  • Amended by: 0046 (policy admission on the chain tip with a conditional slot write; see "Policy chain admission" below) and 0052 (a wrap is stored only under its calling link's directory; see "Wrap link segment" below)

The number was assigned at merge of the add-cloud-service proposal's task group 10 into staging (this file was XXXX-blob-wire-and-client-module.md until then).

Context

Task group 4 (the hosted sync module) shipped Enforcer, Pusher, AuditLog/MemAuditLog, NewAuditedStore and Exporter, but left the blob wire frames after negotiation and the /sync/ HTTP handler out of scope, deferring them, revocation-list polling in cmd/vaultfold-cloud and direct tokens.Verifier.Recheck tests to a follow-up group, and leaving open review finding N10 (an ADR for group 4's own choices), N2 (a cross-namespace export must not be possible) and N6/N7 (unmediated policy and key uploads; refolding the policy chain per event). Task group 10 closes all of that. It does not fix the 1.1 frame shapes and limits, how a blob connection authenticates, the outcome of a put item that cannot yet be decided, how a namespace is created, how a put item's class is decided, who may delete or export, where blob quota usage lives, or how revocation propagates to open connections. The test author chose the frames and the /sync/ test harness; the implementer chose the handler's internal admission model and the binary's wiring. This records both.

Decision

  • Blob 1.1, not 2.0. The frames after negotiation are added as minor version 1.1 of docs/protocol/blob-v1.md, implemented by cloud/client/blobwire. Nothing was defined after negotiation in 1.0, so 1.1 only adds frames; a 1.0 offer is still accepted and answered accept{1.0}, but the connection gets no frame after ready (any frame from a 1.0 connection is refused unexpected-frame).

  • Frame encoding and limits. Every frame is a canonical CBOR map with a text "type" key, encoded and decoded by cloud/client/blobwire, the service's only implementation of this codec (the server imports it directly; there is no second copy). A control frame (every type except put, put-result, get-result and audit-export-result) is at most 4096 bytes, the same limit negotiation uses. A put item's data is at most 1 MiB; a put batch is at most 256 items and 4 MiB total; a list page is at most 1000 paths; an auth token is at most 4096 bytes. Decode refuses input over the largest of these (4194304 + 256*(512+32) + 4096 bytes, the put limit) before parsing at all, and separately enforces each type's own limit once it knows the type; Encode enforces the same limits so that Decode(Encode(f)) round-trips for every f Encode accepts.

  • Device authentication on blob connections. After accept the service sends a fresh 32-byte challenge; the client answers auth with its raw Ed25519 device key, a signature and (when required) a capability token. The signed message is AuthContext ("cloud-blob/v1/auth"), one zero byte, the nonce and the device key, the relay's own scheme (ADR 0023) under a different context, so an auth frame captured on one service is useless on the other. A device key encoding a point of small order is refused auth-failed regardless of its signature, exactly as the relay refuses one (duplicated in cloud/sync, since cloud/sync cannot import cloud/relay/wire from the small cloud/client module and this package is not in it anyway). The token, when the deployment requires one, is verified and re-checked (tokens.Verifier.Recheck) before every operation; an expired or revoked token closes the connection with error seq 0, never a per-operation refusal. The audit log and the admin checks need the authenticated device key; a token alone does not name one (ADR 0023, "Identifying a connection by the token alone. Rejected").

  • One namespace per connection, created by its genesis record. open always succeeds (ack): a namespace with no policy chain does not reject open itself. Its first put batch must open with a genesis policy record, which fixes the namespace's vault id; any other first batch is refused whole, unknown-namespace. The vault id is never read from a frame.

  • Put items are opaque on the wire, classified on the server. A put item is {path, data}; cloud/sync classifies it by path prefix (events/, policy/, keys/, epochs/, heads/) and admits, holds or refuses each independently within one put-result (group 4 finding N6): events through Enforcer.Admit; policy records structurally (below); key records only when their own content hashes to the KeyID their path names; epoch wraps only when their path names a device key and epoch this Handler already knows about; heads/ records only for the connection's own resolved key; anything else, or a path id.ValidateName itself rejects, is refused not-supported.

  • Held is a third put outcome, distinct from refused. A put item whose dependency (a policy record's parent, or an event's policy anchor or rights-reduction event heads) is not yet present is held with dependency-missing and the paths it waits for, never rejected: the client may push it again once the dependency arrives. Every other admission failure is a refusal, most often unauthorised-author.

  • Policy record admission is a real fold, derived from the backend and cached per namespace, keyed on its "policy/" listing. (Superseding this ADR's original "lightweight, not a full re-fold" decision: the review of this group found the lightweight version forgeable -- a record's Author field was trusted as given, with no check that the connecting device actually held that key, findings B1 and B2 on ticket 01M3DG4PC50ENF3709EVCE02N8 -- and the orchestrator's ruling on that ticket replaced it with this.) cloud/sync.namespaceState holds the last policy/ listing a fold was built from, sorted, and the *policy.State (and pubkey-to-KeyID index) that fold produced; refreshPolicy relists policy/ on every operation and only re-reads and re-folds (through foldPolicy, the package's one counted call site of core/policy.Fold, TestPolicyFoldOnlyThroughSeam) when that listing has actually changed (group 4 finding N7, group 10 finding B4: never refolded per event). A put batch's own policy/ and keys/ items are admitted together in one step (admitPolicyAndKeyGroup): a policy record (including a namespace's own genesis) is accepted only when core/policy.Fold, run once over the namespace's existing stored chain plus every policy/ item staged from the same batch, accepts it structurally, and its Author then resolves -- through the accompanying or already-stored key record's raw Signing bytes (identity.DeviceKeyRecord.Signing) -- to this connection's own authenticated device key (closing B1: an Author field alone is never trusted). A namespace's own genesis must arrive in the same batch as its creator's key record: core/policy.Fold treats a genesis whose own key file cannot be read as the hard, whole-fold ErrInvalidGenesis (unlike every other record kind's key dependency, which it reports as a soft, per-record rejection), so admitPolicyAndKeyGroup checks that key is resolvable itself, before ever folding, and holds the genesis dependency-missing on the creator's key path when it is not (a first put with only a genesis record, or with the wrong key record, is held, never refused, and creates nothing). A namespace that already has a chain refuses a second genesis outright, before it ever reaches Fold (closing B2's second half: forging a second genesis could otherwise hard-fail every future fold of the chain with ErrMultipleGenesis for good). Full semantic validation (quorum, now chain position (ADR 0046), roles, key hashes, revocation) is exactly what core/policy.Fold itself decides, the same fold Enforcer.AdmitWithState (below) and Exporter.Export read; there is no separate, weaker check anywhere in this package.

  • The policy cache is keyed on the backend state a fold actually reads, and never keeps a fold a read failure could have skewed (review finding R1 on ticket 01M3DG4PC50ENF3709EVCE02N8, round 2 of this group). namespaceState's cache key is the "policy/" listing and the "keys/" listing, both sorted: a key file that arrives after the record referencing it, with "policy/" itself unchanged, still invalidates the cache. Within a fold, a KeySource read that fails for any reason other than the key genuinely not existing yet (store.ErrNotFound) is a failed refresh, not a cached key-file-missing rejection: refreshPolicy and admitPolicyAndKeyGroup's own joint fold both return that error instead of caching or deciding anything, so a transient backend fault self-heals on the very next read (this package's own admission calls policySnapshot several times over one operation, so a single-shot fault is often already gone by the time a later step needs the same state) rather than pinning a wrong verdict until "policy/" next changes. A put item whose outcome could not be decided this way is held, waiting on itself, never refused: refusing it would be final (the client never retries a refusal), and a transient fault must not be able to make a legitimate write, or a revocation, disappear for good (cloud spec, "Transient read error does not stick"). Writing a policy record's own referenced key record is also reordered: the key is Put before the record it belongs to (previously the other way around), so no reader of the backend, including this same connection's own next fold, can ever observe the record without it.

  • An epoch wrap is admitted only for a device the folded policy authorizes to rotate that collection (review finding N3 on the same ticket, made blocking in round 2). A wrap is put-if-absent, so whoever puts first owns that path for good; without a check on the connecting device (as opposed to the device the wrap names), any device that can open the namespace could squat another device's wrap slot for a minted epoch, denying it that epoch permanently. admitWrap now also requires policy.State.Role(connectingDeviceKeyID, collection) >= policy.RoleRotator (core/policy's own ActionRotate), refusing unauthorised-author otherwise.

  • Admin resolution is backend-derived and survives a restart or a second instance. (Superseding this ADR's original "per-Handler- instance" decision, closing review finding B2's first half.) A connection's raw Ed25519 device key resolves to an id.KeyID by comparing it against every folded device's own keys/<keyid>.key record's Signing bytes (namespaceState.pubkeys, rebuilt whenever the policy chain is refolded); the admin role is policy.State.Role(kid, pol.RootCollection()) == policy.RoleAdmin on that same fold. Nothing is held only in memory that is not also derivable from the namespace's own stored blobs: a service restart, or a second Handler instance over the same backend (a load balancer fronting more than one process), folds the same policy/ listing to the same *policy.State and agrees on the same admin set without any device re-authenticating or re-pushing anything (TestHandlerStateSurvivesRestart, TestHandlerTwoInstancesAgreeOnAdmin).

  • Delete and audit-export are admin-only (DECISION 2, carried from design.md): both are refused unauthorised-author for a device key that does not resolve to the admin role in the namespace (above), and audit-export's from/to fields carry no namespace, so it structurally covers only the connection's own namespace (closing group 4 finding N2).

  • AuditExportResult.Data is the full, signed export, ready to verify offline (orchestrator's ruling on this group's open point). sync.ComplianceExport.Encode() returns the canonical CBOR of every field SignedBytes covers, plus Signature; audit-export-result's data is exactly that. A recipient calls sync.DecodeComplianceExport and then sync.VerifyComplianceExport against the service's published compliance export public key, with no further round trip. Documented in docs/protocol/blob-v1.md.

  • Blob quota, and (round 2 of this group) the last applied revocation list, live in the relay datastore, never the billing one. cloud/relaydb's schema version 2 added a usage table (token id and byte count only), backing sync.UsageStore; Claims.AdmitBlobWrite is checked, and usage recorded, per stored byte, before every write a token-mode connection admits (design.md, "Two datastores, two credentials"). Schema version 3 adds revocation_list (one row: the last signed list document SaveRevocationList stored), for server.Run to fall back to across a restart when it cannot fetch a fresh one.

  • Configuration and revocation polling in the binary. New environment variables (cloud/config) configure the sync module's blob store (a directory or s3://<bucket>), an optional compliance export signing key, audit retention and the revocation list's location and poll interval; setting the revocation URL without a token issuer is a configuration error, since there would be no key to verify the list against. cmd/vaultfold-cloud's server.Run opens one relaydb.DB backing the relay's store, the sync module's audit log, its usage table and (round 2 of this group) the last applied revocation list, builds one tokens.Revocations shared by the relay's and the sync module's tokens.Verifiers, and after every poll tick (successful or not) sweeps both relay.Server.Sweep and the new sync.Handler.Sweep, so a revoked token's open connections on either module close within about one poll interval, faster than relying on the relay's own independent once-a-minute internal sweep loop (which keeps running unchanged, as a coarser second net). The poll waits for each tick before fetching, never fetching once immediately first, so server.Options's NewTicker and Fetcher seams (alongside Now, already used by the relay and sync modules' own clocks) let a test drive every round, including the first, deterministically (review finding B3 on ticket 01M3DG4PC50ENF3709EVCE02N8). The poll and sweep run whenever a token issuer is configured, defaulting to a one-minute interval when no revocation URL is set (review finding M5): there is then nothing to fetch, but an expired token's open connections still need to close on their own schedule. The default Fetcher uses an http.Client with a bounded timeout and refuses a plaintext http:// URL unless its host is loopback, the same rule config.Load's normaliseOrigin applies to the service's own origin (review finding M4).

    • Fetched once before the listener opens, not only on the first tick (review finding R2, a round 2 regression this group's own review caught: waiting for the first tick left a whole interval, five minutes by default, during which a token revoked before a restart was accepted). Run fetches, applies and persists the list before calling Listen, and every later fetch does the same (fetchApplyPersist, the poll loop's own per-round step too). When the startup fetch fails, Run falls back to the last list relaydb.DB.LoadRevocationList returns (the one a previous run's SaveRevocationList persisted after a fetch it verified and applied): a token that list names is refused from the very first connection even though the source is unreachable right now. The poll loop's own first wait then uses a short backoff (startupRetryBackoff, capped to the configured interval) instead of that interval, so a service started while its revocation source was briefly down keeps retrying promptly rather than for a full interval; a successful fetch (at startup or any later round) returns the loop to the configured interval.
    • Never logs the revocation location's credentials or query string, including in a failed fetch's own error text (review finding N1, the round 1 fix's residual gap): a fetch failure is logged with the location redacted to scheme and host (redactRevocationURL, as before) and with the error's own text sanitized (sanitizeFetchError), since net/http's *url.Error formats as "<op> <url>: <err>" and so puts the full request URL, presigned query string and all, into Error() itself; only the wrapped Err's own text is logged.
  • The cloud/client module boundary (DECISION 3, carried from design.md): cloud/client requires only core and, from group 11, github.com/coder/websocket, guarded by TestClientModuleGraph; cloud/protocol moved into it as cloud/client/protocol, and cloud/client/blobwire lives beside it, so cmd/vaultfold's hosted backend client (group 11) never pulls in cloud's server-only dependencies (aws-sdk-go-v2, modernc.org/sqlite).

  • Neither core nor cloud/client is a require line in cmd/vaultfold/go.mod, matching cloud/client/go.mod's own treatment of core. Both are unpublished workspace modules resolved through go.work: a module named in the workspace's use directive is visible to every other workspace module without a require line, which is how cmd/vaultfold already used core before this group and how it now reaches cloud/client/blob too. cmd/vaultfold/modulegraph_test.go's TestModuleGraphHasNoServerDependencies checks this module-graph absence with GOWORK=off go list -m all (unaffected either way, since neither module is required) and separately checks, in workspace mode, that no server-only package is ever linked and that cloud/client/blob is: "the package-level check runs in workspace mode, which is how every real build of the binary resolves imports" (make crossbuild never sets GOWORK=off). This resolved an apparent conflict flagged on ticket 01M3DG4PCR8AH611SD65BQWD0E between this line's original wording ("cmd/vaultfold/go.mod requires cloud/client only") and the test author's own handoff note that it should not: both a bare require cloud/client line and no line at all satisfy the guard test, and omitting it is the existing, hand-documented convention.

  • Local mirror and push queue (group 11). A hosted vault's files live in an fsstore mirror under Paths.VaultCloneDir (cmd/vaultfold/internal/hosted.Backend), so every command works offline exactly as for a filesystem vault. The push queue is not a separate list: it is every mirror file that is neither settled (stored by the service, or listed by it at a Pull) nor refused for good, so a write through the mirror's store.Store is queued by construction; the settled and refused sets are a small JSON state file kept outside the mirror (Paths.HostedStateFile, written atomically) so a lost or stale state file re-derives safely (Pull re-marks every listed path settled, so a second genesis is never re-sent even from an empty state). Queue orders what it returns causally to minimize server-side holds: the genesis first (it must open the first batch and travel with its creator's key record), then key records, then other policy records (parents first), then epoch wraps, then data events (parents first), then head records; ties within a class break ascending by path. vault.json is never queued: the service refuses it as not content-addressed, and Pull rebuilds it locally from the genesis record's VaultJSONHash, RootCollection and HLC wall time, checked against the hash.

  • Token storage (group 11). vault add --token-file and init --hosted --token-file never keep the token in memory beyond the dial and the one copy: profile.WriteToken copies it into a 0600 file under the profile's data directory (profile.TokenFileRel(name), tokens/<name>.token), named by the registry entry's new token key (VaultEntry.Token, a Paths.Data-relative path; AddVault refuses an absolute path, one with a .. segment, or a Token on any backend but BackendHosted). profile.ReadToken refuses, before ever reading the content, a file that is missing, empty, over profile.MaxTokenFileSize or (outside Windows) readable or writable by anyone but its owner; no error, log line or --json field ever contains the token.

  • Per-item outcomes reach the sync report and the cache (group 11, closing task 4.6's client half). A push's Refused items (other than quota-exceeded) are appended to profile.Cache.ServiceRejected (first refusal kept per path, ascending by path) and never queued again; Held items replace profile.Cache.ServiceHeld outright each sync (they stay queued and are retried automatically); a quota-exceeded refusal is neither, and stays queued. doctor surfaces both from the cache alone (CheckRejected's service-rejected findings, CheckHeld's held-by-service findings), so it never needs to dial the service. A token refusal (token-expired, token-revoked, token-invalid, token-required) or quota-exceeded is a typed *session.ServiceError / *cli.BackendError whose --json code is the reason itself and whose message never contains the token; for quota-exceeded the rest of the sync's report (what was sent, received and rejected) is still returned and printed, since only the over-quota items were left behind.

  • Policy chain admission (OpenSpec change policy-linear-chain, group 3). Superseding the "real fold" and "epoch wrap ... rotate" bullets above for policy records and wraps: the namespace's policy records are one linear chain, replayed with core/policy.Replay from the slot files stored under policy/ (still through the counted seam foldPolicy, and still cached per namespace on the policy/ and keys/ listings). admitPolicyAndKeyGroup admits a policy record only as the next link: its author's key record carries the connection's raw signing key, its parent is the tip, its seqno is the tip's plus one, its path is that slot, and the replay of the stored chain plus the batch's records makes it a link. It is stored with store.Put (put-if-absent, on S3 a conditional If-None-Match: * write), which is the per-namespace compare-and-swap: no lock is shared between service instances. A batch's records are checked in order before any is stored, and stored all or nothing with their key records and wraps; a record's own key record is written just before it, after checking that its slot is still free. A Handler whose conditional write loses discards its cached namespace state and the Enforcer's chain-prefix cache for it. An epoch wrap is stored only when the stored link that newly gives the recipient that epoch (found by binary search, since a key never loses an epoch) was authored by the putting device, which must be active; a key supersession's wraps may also come from its old key (the author, superseded by the link) or its new key, active. A key record is stored only when its bytes equal the key record a staged or stored link carries for that KeyID: the KeyID covers only the public keys, so a variant with another member, authoriser or signature would otherwise squat the path (review round F3); other bytes already at the path refuse the key content-mismatch and the record is still stored. Enforcer.AdmitWithState memoises, per *policy.State, the nearest link of each anchor (core/fold's rule for an anchor on a void record), link positions, AuthorizedAt, RightsReductions and minted epochs, so admitting an event against a cached state does not cost more as the chain grows. The orchestrator's rulings on the test author's design calls (DECISION on ticket 01M66KDMNR1XTZJSW2CWNXMFT9) are part of this decision: a byte-identical resend of a stored link is answered stored and the rest of its batch is checked on top of it; a delete of a policy slot is refused unauthorised-author; the key records and wraps of a refused batch are not stored, with no separate reason. Reasons a policy record, its key record or a wrap is refused with:

    Outcome Reason When
    refused stale-policy-head The record's parent is not the tip, its seqno is not the tip's plus one, or its slot is taken (another record won it, or the conditional write lost).
    refused unauthorised-author The author is not the connecting device, the record names another vault, a genesis other than the stored one, or the replay voids the record; a wrap no stored link by the putting (active) device calls for; a key record no staged or stored link carries; a delete of a policy slot, by any device.
    refused content-mismatch The record does not decode or its path is not its own slot; a key record whose path is not the KeyID it derives, whose bytes differ from the key record the registering link carries, or whose path already holds other bytes (the record is still stored).
    refused not-supported A policy record of a kind this version does not implement.
    refused the first refusal's reason Every other policy record, key record and epoch wrap of a batch in which one policy record is refused.
    stored - A byte-identical copy of a link already stored at its slot (the genesis included), put before any new record of the batch.
    held dependency-missing A genesis without its creator's key record (and the batch's other records, on the genesis); any item whose check met a transient backend error. A non-genesis record is never held.
  • Wrap link segment (OpenSpec change wrap-paths-by-mint-link, group 1; ADR 0052). A wrap's path is epochs/<collection>/<epoch>/<link>/<keyid>.wrap, parsed with id.ParseWrapPath; any other shape, the earlier layout with no <link> segment included, is refused not-supported. On top of the putter rule of the bullet above, admitWrap stores a wrap only when the path's <link> segment equals the first 16 hex characters of the id of the stored link that newly gives the recipient that epoch (the link callingLink already finds). A wrap under any other directory (the prefix of no stored link, of another stored link such as the mint when a later grant is the calling link, or of a record that lost its slot) is refused unauthorised-author and stored nowhere, with no new reason. So a namespace never holds a wrap under a directory other than its calling link's. The check reads the chain as stored and the path only, never the wrap's content.

Consequences

  • docs/protocol/blob-v1.md is normative for the 1.1 frames, and cloud/client/blobwire is their one reference implementation on both sides of the wire.
  • A client must publish every device's key record to the namespace alongside the policy record that names it, in the same put batch for a genesis: there is no longer a lightweight fallback that admits a policy record without it. Group 11's hosted-backend client is expected to do so as part of its normal push order (policy records and key records before the events that depend on them, per design.md "Local mirror and push queue"); an honest client that does this sees no difference from before, and one that does not gets its record held dependency-missing rather than silently degraded validation.
  • Every put batch containing a policy/ or keys/ item pays a fold of the namespace's whole policy chain when its policy/ listing has changed since the cache was last built; a batch of events, wraps or heads records with no policy or key item pays no fold at all (the cached *policy.State is reused). A namespace with an unusually large policy chain (many members, devices or grants) pays this cost on every membership change, not on every event.
  • server.Run's revocation sweep cadence follows VAULTFOLD_CLOUD_REVOCATION_INTERVAL, so a shorter interval trades more frequent polling for faster revocation; the relay's own once-a-minute sweep is unaffected and keeps providing a floor even if polling is disabled.

Alternatives considered

  • Trusting a policy record's Author field and admitting records independently of their own key file (this ADR's original decision). Rejected on review (findings B1 and B2 on ticket 01M3DG4PC50ENF3709EVCE02N8): any device could forge a record naming another key as Author, and the in-memory admin index did not survive a restart or a second instance, so replaying a forged genesis against a fresh instance was a working attack. Superseded by "Policy record admission is a real fold..." above.
  • Requiring a genesis's creator key file to already exist on the backend before the genesis itself can be admitted (this ADR's original reason for the lightweight design). Rejected in its own right (it never worked: nothing can exist before the first put succeeds) in favor of requiring the key record in the same batch as the genesis and folding them together, which admits both or holds the genesis, never a contradiction.
  • Identifying a connection's admin role by the presented capability token alone. Rejected for the same reason ADR 0023 rejects it for the relay: a token names a plan and its caveats, never a member's role in one specific vault's policy chain.
  • Persisting a pubkey-to-KeyID or admin index in cloud/relaydb, alongside usage. Unnecessary once the policy state itself is derived fresh from the backend's own policy/ and keys/ blobs (cached, not persisted separately): there is nothing left that a restart or a second instance would need recovered from a side table.