0025: Blob wire frames, the sync endpoint and the client module boundary
- Status: proposed
- Date: 2026-09-25
- Amended by: 0046 (policy admission on the chain tip with a conditional slot write; see "Policy chain admission" below) and 0052 (a wrap is stored only under its calling link's directory; see "Wrap link segment" below)
The number was assigned at merge of the add-cloud-service proposal's
task group 10 into staging (this file was
XXXX-blob-wire-and-client-module.md until then).
Context
Task group 4 (the hosted sync module) shipped Enforcer, Pusher,
AuditLog/MemAuditLog, NewAuditedStore and Exporter, but left the
blob wire frames after negotiation and the /sync/ HTTP handler out of
scope, deferring them, revocation-list polling in cmd/vaultfold-cloud
and direct tokens.Verifier.Recheck tests to a follow-up group, and
leaving open review finding N10 (an ADR for group 4's own choices), N2
(a cross-namespace export must not be possible) and N6/N7 (unmediated
policy and key uploads; refolding the policy chain per event). Task
group 10 closes all of that. It does not fix the 1.1 frame shapes and
limits, how a blob connection authenticates, the outcome of a put item
that cannot yet be decided, how a namespace is created, how a put item's
class is decided, who may delete or export, where blob quota usage
lives, or how revocation propagates to open connections. The test author
chose the frames and the /sync/ test harness; the implementer chose
the handler's internal admission model and the binary's wiring. This
records both.
Decision
-
Blob 1.1, not 2.0. The frames after negotiation are added as minor version 1.1 of docs/protocol/blob-v1.md, implemented by
cloud/client/blobwire. Nothing was defined after negotiation in 1.0, so 1.1 only adds frames; a 1.0 offer is still accepted and answeredaccept{1.0}, but the connection gets no frame afterready(any frame from a 1.0 connection is refusedunexpected-frame). -
Frame encoding and limits. Every frame is a canonical CBOR map with a text
"type"key, encoded and decoded bycloud/client/blobwire, the service's only implementation of this codec (the server imports it directly; there is no second copy). A control frame (every type exceptput,put-result,get-resultandaudit-export-result) is at most 4096 bytes, the same limit negotiation uses. Aputitem's data is at most 1 MiB; aputbatch is at most 256 items and 4 MiB total; alistpage is at most 1000 paths; anauthtoken is at most 4096 bytes.Decoderefuses input over the largest of these (4194304 + 256*(512+32) + 4096 bytes, theputlimit) before parsing at all, and separately enforces each type's own limit once it knows the type;Encodeenforces the same limits so thatDecode(Encode(f))round-trips for everyfEncodeaccepts. -
Device authentication on blob connections. After
acceptthe service sends a fresh 32-bytechallenge; the client answersauthwith its raw Ed25519 device key, a signature and (when required) a capability token. The signed message isAuthContext ("cloud-blob/v1/auth"), one zero byte, the nonce and the device key, the relay's own scheme (ADR 0023) under a different context, so anauthframe captured on one service is useless on the other. A device key encoding a point of small order is refusedauth-failedregardless of its signature, exactly as the relay refuses one (duplicated incloud/sync, sincecloud/synccannot importcloud/relay/wirefrom the smallcloud/clientmodule and this package is not in it anyway). The token, when the deployment requires one, is verified and re-checked (tokens.Verifier.Recheck) before every operation; an expired or revoked token closes the connection witherrorseq 0, never a per-operation refusal. The audit log and the admin checks need the authenticated device key; a token alone does not name one (ADR 0023, "Identifying a connection by the token alone. Rejected"). -
One namespace per connection, created by its genesis record.
openalways succeeds (ack): a namespace with no policy chain does not rejectopenitself. Its firstputbatch must open with a genesis policy record, which fixes the namespace's vault id; any other first batch is refused whole,unknown-namespace. The vault id is never read from a frame. -
Put items are opaque on the wire, classified on the server. A put item is
{path, data};cloud/syncclassifies it by path prefix (events/,policy/,keys/,epochs/,heads/) and admits, holds or refuses each independently within oneput-result(group 4 finding N6): events throughEnforcer.Admit; policy records structurally (below); key records only when their own content hashes to the KeyID their path names; epoch wraps only when their path names a device key and epoch this Handler already knows about;heads/records only for the connection's own resolved key; anything else, or a pathid.ValidateNameitself rejects, is refusednot-supported. -
Held is a third put outcome, distinct from refused. A put item whose dependency (a policy record's parent, or an event's policy anchor or rights-reduction event heads) is not yet present is held with
dependency-missingand the paths it waits for, never rejected: the client may push it again once the dependency arrives. Every other admission failure is a refusal, most oftenunauthorised-author. -
Policy record admission is a real fold, derived from the backend and cached per namespace, keyed on its "policy/" listing. (Superseding this ADR's original "lightweight, not a full re-fold" decision: the review of this group found the lightweight version forgeable -- a record's Author field was trusted as given, with no check that the connecting device actually held that key, findings B1 and B2 on ticket 01M3DG4PC50ENF3709EVCE02N8 -- and the orchestrator's ruling on that ticket replaced it with this.)
cloud/sync.namespaceStateholds the lastpolicy/listing a fold was built from, sorted, and the*policy.State(and pubkey-to-KeyID index) that fold produced;refreshPolicyrelistspolicy/on every operation and only re-reads and re-folds (throughfoldPolicy, the package's one counted call site ofcore/policy.Fold,TestPolicyFoldOnlyThroughSeam) when that listing has actually changed (group 4 finding N7, group 10 finding B4: never refolded per event). Aputbatch's ownpolicy/andkeys/items are admitted together in one step (admitPolicyAndKeyGroup): a policy record (including a namespace's own genesis) is accepted only whencore/policy.Fold, run once over the namespace's existing stored chain plus everypolicy/item staged from the same batch, accepts it structurally, and its Author then resolves -- through the accompanying or already-stored key record's rawSigningbytes (identity.DeviceKeyRecord.Signing) -- to this connection's own authenticated device key (closing B1: an Author field alone is never trusted). A namespace's own genesis must arrive in the same batch as its creator's key record:core/policy.Foldtreats a genesis whose own key file cannot be read as the hard, whole-foldErrInvalidGenesis(unlike every other record kind's key dependency, which it reports as a soft, per-record rejection), soadmitPolicyAndKeyGroupchecks that key is resolvable itself, before ever folding, and holds the genesisdependency-missingon the creator's key path when it is not (a firstputwith only a genesis record, or with the wrong key record, is held, never refused, and creates nothing). A namespace that already has a chain refuses a second genesis outright, before it ever reachesFold(closing B2's second half: forging a second genesis could otherwise hard-fail every future fold of the chain withErrMultipleGenesisfor good). Full semantic validation (quorum, now chain position (ADR 0046), roles, key hashes, revocation) is exactly whatcore/policy.Folditself decides, the same foldEnforcer.AdmitWithState(below) andExporter.Exportread; there is no separate, weaker check anywhere in this package. -
The policy cache is keyed on the backend state a fold actually reads, and never keeps a fold a read failure could have skewed (review finding R1 on ticket 01M3DG4PC50ENF3709EVCE02N8, round 2 of this group).
namespaceState's cache key is the "policy/" listing and the "keys/" listing, both sorted: a key file that arrives after the record referencing it, with "policy/" itself unchanged, still invalidates the cache. Within a fold, aKeySourceread that fails for any reason other than the key genuinely not existing yet (store.ErrNotFound) is a failed refresh, not a cachedkey-file-missingrejection:refreshPolicyandadmitPolicyAndKeyGroup's own joint fold both return that error instead of caching or deciding anything, so a transient backend fault self-heals on the very next read (this package's own admission callspolicySnapshotseveral times over one operation, so a single-shot fault is often already gone by the time a later step needs the same state) rather than pinning a wrong verdict until "policy/" next changes. Aputitem whose outcome could not be decided this way is held, waiting on itself, never refused: refusing it would be final (the client never retries a refusal), and a transient fault must not be able to make a legitimate write, or a revocation, disappear for good (cloud spec, "Transient read error does not stick"). Writing a policy record's own referenced key record is also reordered: the key isPutbefore the record it belongs to (previously the other way around), so no reader of the backend, including this same connection's own next fold, can ever observe the record without it. -
An epoch wrap is admitted only for a device the folded policy authorizes to rotate that collection (review finding N3 on the same ticket, made blocking in round 2). A wrap is put-if-absent, so whoever puts first owns that path for good; without a check on the connecting device (as opposed to the device the wrap names), any device that can open the namespace could squat another device's wrap slot for a minted epoch, denying it that epoch permanently.
admitWrapnow also requirespolicy.State.Role(connectingDeviceKeyID, collection) >= policy.RoleRotator(core/policy's ownActionRotate), refusingunauthorised-authorotherwise. -
Admin resolution is backend-derived and survives a restart or a second instance. (Superseding this ADR's original "per-Handler- instance" decision, closing review finding B2's first half.) A connection's raw Ed25519 device key resolves to an
id.KeyIDby comparing it against every folded device's ownkeys/<keyid>.keyrecord'sSigningbytes (namespaceState.pubkeys, rebuilt whenever the policy chain is refolded); the admin role ispolicy.State.Role(kid, pol.RootCollection()) == policy.RoleAdminon that same fold. Nothing is held only in memory that is not also derivable from the namespace's own stored blobs: a service restart, or a secondHandlerinstance over the same backend (a load balancer fronting more than one process), folds the samepolicy/listing to the same*policy.Stateand agrees on the same admin set without any device re-authenticating or re-pushing anything (TestHandlerStateSurvivesRestart,TestHandlerTwoInstancesAgreeOnAdmin). -
Delete and audit-export are admin-only (DECISION 2, carried from design.md): both are refused
unauthorised-authorfor a device key that does not resolve to the admin role in the namespace (above), andaudit-export'sfrom/tofields carry no namespace, so it structurally covers only the connection's own namespace (closing group 4 finding N2). -
AuditExportResult.Data is the full, signed export, ready to verify offline (orchestrator's ruling on this group's open point).
sync.ComplianceExport.Encode()returns the canonical CBOR of every fieldSignedBytescovers, plusSignature;audit-export-result'sdatais exactly that. A recipient callssync.DecodeComplianceExportand thensync.VerifyComplianceExportagainst the service's published compliance export public key, with no further round trip. Documented in docs/protocol/blob-v1.md. -
Blob quota, and (round 2 of this group) the last applied revocation list, live in the relay datastore, never the billing one.
cloud/relaydb's schema version 2 added ausagetable (token id and byte count only), backingsync.UsageStore;Claims.AdmitBlobWriteis checked, and usage recorded, per stored byte, before every write a token-mode connection admits (design.md, "Two datastores, two credentials"). Schema version 3 addsrevocation_list(one row: the last signed list documentSaveRevocationListstored), forserver.Runto fall back to across a restart when it cannot fetch a fresh one. -
Configuration and revocation polling in the binary. New environment variables (
cloud/config) configure the sync module's blob store (a directory ors3://<bucket>), an optional compliance export signing key, audit retention and the revocation list's location and poll interval; setting the revocation URL without a token issuer is a configuration error, since there would be no key to verify the list against.cmd/vaultfold-cloud'sserver.Runopens onerelaydb.DBbacking the relay's store, the sync module's audit log, its usage table and (round 2 of this group) the last applied revocation list, builds onetokens.Revocationsshared by the relay's and the sync module'stokens.Verifiers, and after every poll tick (successful or not) sweeps bothrelay.Server.Sweepand the newsync.Handler.Sweep, so a revoked token's open connections on either module close within about one poll interval, faster than relying on the relay's own independent once-a-minute internal sweep loop (which keeps running unchanged, as a coarser second net). The poll waits for each tick before fetching, never fetching once immediately first, soserver.Options'sNewTickerandFetcherseams (alongsideNow, already used by the relay and sync modules' own clocks) let a test drive every round, including the first, deterministically (review finding B3 on ticket 01M3DG4PC50ENF3709EVCE02N8). The poll and sweep run whenever a token issuer is configured, defaulting to a one-minute interval when no revocation URL is set (review finding M5): there is then nothing to fetch, but an expired token's open connections still need to close on their own schedule. The defaultFetcheruses anhttp.Clientwith a bounded timeout and refuses a plaintexthttp://URL unless its host is loopback, the same ruleconfig.Load'snormaliseOriginapplies to the service's own origin (review finding M4).- Fetched once before the listener opens, not only on the first tick
(review finding R2, a round 2 regression this group's own review
caught: waiting for the first tick left a whole interval, five
minutes by default, during which a token revoked before a restart was
accepted).
Runfetches, applies and persists the list before callingListen, and every later fetch does the same (fetchApplyPersist, the poll loop's own per-round step too). When the startup fetch fails,Runfalls back to the last listrelaydb.DB.LoadRevocationListreturns (the one a previous run'sSaveRevocationListpersisted after a fetch it verified and applied): a token that list names is refused from the very first connection even though the source is unreachable right now. The poll loop's own first wait then uses a short backoff (startupRetryBackoff, capped to the configured interval) instead of that interval, so a service started while its revocation source was briefly down keeps retrying promptly rather than for a full interval; a successful fetch (at startup or any later round) returns the loop to the configured interval. - Never logs the revocation location's credentials or query string,
including in a failed fetch's own error text (review finding N1,
the round 1 fix's residual gap): a fetch failure is logged with the
location redacted to scheme and host (
redactRevocationURL, as before) and with the error's own text sanitized (sanitizeFetchError), sincenet/http's*url.Errorformats as"<op> <url>: <err>"and so puts the full request URL, presigned query string and all, intoError()itself; only the wrappedErr's own text is logged.
- Fetched once before the listener opens, not only on the first tick
(review finding R2, a round 2 regression this group's own review
caught: waiting for the first tick left a whole interval, five
minutes by default, during which a token revoked before a restart was
accepted).
-
The
cloud/clientmodule boundary (DECISION 3, carried from design.md):cloud/clientrequires onlycoreand, from group 11,github.com/coder/websocket, guarded byTestClientModuleGraph;cloud/protocolmoved into it ascloud/client/protocol, andcloud/client/blobwirelives beside it, socmd/vaultfold's hosted backend client (group 11) never pulls incloud's server-only dependencies (aws-sdk-go-v2, modernc.org/sqlite). -
Neither
corenorcloud/clientis arequireline incmd/vaultfold/go.mod, matchingcloud/client/go.mod's own treatment ofcore. Both are unpublished workspace modules resolved throughgo.work: a module named in the workspace'susedirective is visible to every other workspace module without arequireline, which is howcmd/vaultfoldalready usedcorebefore this group and how it now reachescloud/client/blobtoo.cmd/vaultfold/modulegraph_test.go'sTestModuleGraphHasNoServerDependencieschecks this module-graph absence withGOWORK=off go list -m all(unaffected either way, since neither module is required) and separately checks, in workspace mode, that no server-only package is ever linked and thatcloud/client/blobis: "the package-level check runs in workspace mode, which is how every real build of the binary resolves imports" (make crossbuildnever setsGOWORK=off). This resolved an apparent conflict flagged on ticket 01M3DG4PCR8AH611SD65BQWD0E between this line's original wording ("cmd/vaultfold/go.modrequirescloud/clientonly") and the test author's own handoff note that it should not: both a barerequire cloud/clientline and no line at all satisfy the guard test, and omitting it is the existing, hand-documented convention. -
Local mirror and push queue (group 11). A hosted vault's files live in an
fsstoremirror underPaths.VaultCloneDir(cmd/vaultfold/internal/hosted.Backend), so every command works offline exactly as for a filesystem vault. The push queue is not a separate list: it is every mirror file that is neither settled (stored by the service, or listed by it at aPull) nor refused for good, so a write through the mirror'sstore.Storeis queued by construction; the settled and refused sets are a small JSON state file kept outside the mirror (Paths.HostedStateFile, written atomically) so a lost or stale state file re-derives safely (Pullre-marks every listed path settled, so a second genesis is never re-sent even from an empty state).Queueorders what it returns causally to minimize server-side holds: the genesis first (it must open the first batch and travel with its creator's key record), then key records, then other policy records (parents first), then epoch wraps, then data events (parents first), then head records; ties within a class break ascending by path.vault.jsonis never queued: the service refuses it as not content-addressed, andPullrebuilds it locally from the genesis record'sVaultJSONHash,RootCollectionand HLC wall time, checked against the hash. -
Token storage (group 11).
vault add --token-fileandinit --hosted --token-filenever keep the token in memory beyond the dial and the one copy:profile.WriteTokencopies it into a 0600 file under the profile's data directory (profile.TokenFileRel(name),tokens/<name>.token), named by the registry entry's newtokenkey (VaultEntry.Token, aPaths.Data-relative path;AddVaultrefuses an absolute path, one with a..segment, or aTokenon any backend butBackendHosted).profile.ReadTokenrefuses, before ever reading the content, a file that is missing, empty, overprofile.MaxTokenFileSizeor (outside Windows) readable or writable by anyone but its owner; no error, log line or--jsonfield ever contains the token. -
Per-item outcomes reach the sync report and the cache (group 11, closing task 4.6's client half). A push's
Refuseditems (other thanquota-exceeded) are appended toprofile.Cache.ServiceRejected(first refusal kept per path, ascending by path) and never queued again;Helditems replaceprofile.Cache.ServiceHeldoutright each sync (they stay queued and are retried automatically); aquota-exceededrefusal is neither, and stays queued.doctorsurfaces both from the cache alone (CheckRejected'sservice-rejectedfindings,CheckHeld'sheld-by-servicefindings), so it never needs to dial the service. A token refusal (token-expired,token-revoked,token-invalid,token-required) orquota-exceededis a typed*session.ServiceError/*cli.BackendErrorwhose--jsoncode is the reason itself and whose message never contains the token; forquota-exceededthe rest of the sync's report (what was sent, received and rejected) is still returned and printed, since only the over-quota items were left behind. -
Policy chain admission (OpenSpec change policy-linear-chain, group 3). Superseding the "real fold" and "epoch wrap ... rotate" bullets above for policy records and wraps: the namespace's policy records are one linear chain, replayed with
core/policy.Replayfrom the slot files stored underpolicy/(still through the counted seamfoldPolicy, and still cached per namespace on thepolicy/andkeys/listings).admitPolicyAndKeyGroupadmits a policy record only as the next link: its author's key record carries the connection's raw signing key, itsparentis the tip, itsseqnois the tip's plus one, its path is that slot, and the replay of the stored chain plus the batch's records makes it a link. It is stored withstore.Put(put-if-absent, on S3 a conditionalIf-None-Match: *write), which is the per-namespace compare-and-swap: no lock is shared between service instances. A batch's records are checked in order before any is stored, and stored all or nothing with their key records and wraps; a record's own key record is written just before it, after checking that its slot is still free. A Handler whose conditional write loses discards its cached namespace state and the Enforcer's chain-prefix cache for it. An epoch wrap is stored only when the stored link that newly gives the recipient that epoch (found by binary search, since a key never loses an epoch) was authored by the putting device, which must be active; a key supersession's wraps may also come from its old key (the author, superseded by the link) or its new key, active. A key record is stored only when its bytes equal the key record a staged or stored link carries for that KeyID: the KeyID covers only the public keys, so a variant with another member, authoriser or signature would otherwise squat the path (review round F3); other bytes already at the path refuse the keycontent-mismatchand the record is still stored.Enforcer.AdmitWithStatememoises, per*policy.State, the nearest link of each anchor (core/fold's rule for an anchor on a void record), link positions,AuthorizedAt,RightsReductionsand minted epochs, so admitting an event against a cached state does not cost more as the chain grows. The orchestrator's rulings on the test author's design calls (DECISION on ticket 01M66KDMNR1XTZJSW2CWNXMFT9) are part of this decision: a byte-identical resend of a stored link is answered stored and the rest of its batch is checked on top of it; a delete of a policy slot is refusedunauthorised-author; the key records and wraps of a refused batch are not stored, with no separate reason. Reasons a policy record, its key record or a wrap is refused with:Outcome Reason When refused stale-policy-headThe record's parent is not the tip, its seqno is not the tip's plus one, or its slot is taken (another record won it, or the conditional write lost). refused unauthorised-authorThe author is not the connecting device, the record names another vault, a genesis other than the stored one, or the replay voids the record; a wrap no stored link by the putting (active) device calls for; a key record no staged or stored link carries; a delete of a policy slot, by any device. refused content-mismatchThe record does not decode or its path is not its own slot; a key record whose path is not the KeyID it derives, whose bytes differ from the key record the registering link carries, or whose path already holds other bytes (the record is still stored). refused not-supportedA policy record of a kind this version does not implement. refused the first refusal's reason Every other policy record, key record and epoch wrap of a batch in which one policy record is refused. stored - A byte-identical copy of a link already stored at its slot (the genesis included), put before any new record of the batch. held dependency-missingA genesis without its creator's key record (and the batch's other records, on the genesis); any item whose check met a transient backend error. A non-genesis record is never held. -
Wrap link segment (OpenSpec change wrap-paths-by-mint-link, group 1; ADR 0052). A wrap's path is
epochs/<collection>/<epoch>/<link>/<keyid>.wrap, parsed withid.ParseWrapPath; any other shape, the earlier layout with no<link>segment included, is refusednot-supported. On top of the putter rule of the bullet above,admitWrapstores a wrap only when the path's<link>segment equals the first 16 hex characters of the id of the stored link that newly gives the recipient that epoch (the linkcallingLinkalready finds). A wrap under any other directory (the prefix of no stored link, of another stored link such as the mint when a later grant is the calling link, or of a record that lost its slot) is refusedunauthorised-authorand stored nowhere, with no new reason. So a namespace never holds a wrap under a directory other than its calling link's. The check reads the chain as stored and the path only, never the wrap's content.
Consequences
docs/protocol/blob-v1.mdis normative for the 1.1 frames, andcloud/client/blobwireis their one reference implementation on both sides of the wire.- A client must publish every device's key record to the namespace
alongside the policy record that names it, in the same
putbatch for a genesis: there is no longer a lightweight fallback that admits a policy record without it. Group 11's hosted-backend client is expected to do so as part of its normal push order (policy records and key records before the events that depend on them, per design.md "Local mirror and push queue"); an honest client that does this sees no difference from before, and one that does not gets its record helddependency-missingrather than silently degraded validation. - Every
putbatch containing apolicy/orkeys/item pays a fold of the namespace's whole policy chain when itspolicy/listing has changed since the cache was last built; a batch of events, wraps or heads records with no policy or key item pays no fold at all (the cached*policy.Stateis reused). A namespace with an unusually large policy chain (many members, devices or grants) pays this cost on every membership change, not on every event. server.Run's revocation sweep cadence followsVAULTFOLD_CLOUD_REVOCATION_INTERVAL, so a shorter interval trades more frequent polling for faster revocation; the relay's own once-a-minute sweep is unaffected and keeps providing a floor even if polling is disabled.
Alternatives considered
- Trusting a policy record's Author field and admitting records independently of their own key file (this ADR's original decision). Rejected on review (findings B1 and B2 on ticket 01M3DG4PC50ENF3709EVCE02N8): any device could forge a record naming another key as Author, and the in-memory admin index did not survive a restart or a second instance, so replaying a forged genesis against a fresh instance was a working attack. Superseded by "Policy record admission is a real fold..." above.
- Requiring a genesis's creator key file to already exist on the
backend before the genesis itself can be admitted (this ADR's
original reason for the lightweight design). Rejected in its own
right (it never worked: nothing can exist before the first
putsucceeds) in favor of requiring the key record in the same batch as the genesis and folding them together, which admits both or holds the genesis, never a contradiction. - Identifying a connection's admin role by the presented capability token alone. Rejected for the same reason ADR 0023 rejects it for the relay: a token names a plan and its caveats, never a member's role in one specific vault's policy chain.
- Persisting a pubkey-to-KeyID or admin index in
cloud/relaydb, alongside usage. Unnecessary once the policy state itself is derived fresh from the backend's ownpolicy/andkeys/blobs (cached, not persisted separately): there is nothing left that a restart or a second instance would need recovered from a side table.