Skip to content

0026: Operations and self-hosting

  • Status: accepted
  • Date: 2026-09-26

Context

add-cloud-service task group 9 gives cloud/cmd/vaultfold-cloud the operational surface a self-hoster or the vaultfold-operated instance needs: health, readiness and metrics endpoints, the transport security policy ADR 0015 requires, and a container image with a Compose example. Two review rounds on ticket 01M3A20PWH4B5RWMR1QT6W7HQW found real gaps between the first pass and the cloud spec's "Health, readiness and metrics" and "Transport security and headers" requirements; this ADR records the rulings that closed them, as design.md's "Group 9 round 2" note said it would.

Decisions

Metrics: fixed labels, aggregate-only quota. A metric label is never derived from request data. MetricHTTPRequestsTotal and MetricHTTPRequestDurationSeconds label by HTTP method and a path template, but the method itself is mapped to a fixed set (GET, HEAD, POST, PUT, DELETE, OPTIONS, else OTHER) before it ever reaches a label: Go's HTTP server accepts any RFC 7230 token as a method, so recording r.Method verbatim let a client choose the label value (an address- or token-shaped string) or mint one new series per request, an unbounded-memory vector as well as a potential address leak (review round 2, finding B1). The path template is a caller-chosen constant (/relay, /sync, PathHealthz, and so on), never the request's actual path, so a sync namespace or a share id never reaches a label either.

Quota usage (the cloud spec's "Metrics content" scenario) is reported only as one unlabelled aggregate gauge, vaultfold_cloud_quota_charged_bytes ("bytes charged against quotas"), never broken out per token or per namespace (review round 2, finding B6). It sums cloud/sync.UsageStore's per-token byte counts via an optional AggregateUsage interface (TotalStoredBytes), which MemUsageStore and cloud/relaydb.DB both implement; a UsageStore that does not implement it (or no token-mode usage store at all) simply reports 0.

Round 2 also shipped a total-blob-count gauge and a namespaces-over-quota gauge, both hard-wired to a constant 0, since neither per-item counting nor centralized quota-limit tracking exists yet. Reviewer round 2 (finding R2-1) flagged this as actively misleading: a gauge that can never be anything but 0 is indistinguishable from a genuine "nothing over quota" reading, so an alert built on it could never fire -- worse than not exposing the metric at all. The orchestrator's amended ruling (round 3) removed both outright rather than leave them at a placeholder; they are deferred in the cloud spec and here until the tracking that would make them meaningful exists: a per-item counter (likely a new column alongside cloud/relaydb's usage table) for the blob count, and somewhere to persist a quota's limit centrally (today a quota is only a caveat carried on the presented token itself, compared against usage at request time, never stored) for namespaces-over-quota.

Separately, reviewer round 2 also nitted the surviving gauge's original name, vaultfold_cloud_quota_stored_bytes_total: a counter-style _total suffix on a value that is a gauge (it can go down), and "stored bytes" overstated what it measures -- the same per-token accounting Claims.AdmitBlobWrite checks against a quota caveat, not an independent count of bytes actually resident in the blob store. Renamed to vaultfold_cloud_quota_charged_bytes (round 3); the Go method (AggregateUsage.TotalStoredBytes) keeps its name, since renaming it would have required editing test files that already reference it, which the implementer role does not do.

Transport security: an explicit trusted-proxy list. The first pass honored X-Forwarded-Proto from any peer and treated any loopback peer as exempt unconditionally. Both were exploitable (review round 2, finding B2, reproduced against a directly exposed instance): a plaintext request carrying X-Forwarded-Proto: https reached a machine endpoint from a non-loopback address, and a same-host proxy's own X-Forwarded-Proto: http was silently overridden by the loopback exemption.

The fix is VAULTFOLD_CLOUD_TRUSTED_PROXIES, a comma-separated CIDR list, empty by default (config.go's EnvTrustedProxies, Config.TrustedProxies):

  • A forwarded protocol header is honored only when the immediate peer (r.RemoteAddr) falls in that list. From any other peer it is ignored entirely; the connection's own scheme (always plaintext for this service, which never terminates TLS itself) decides.
  • The loopback exemption applies only when all three hold: the peer is loopback, no forwarded protocol header is present at all, and loopback is not itself listed as a trusted proxy. Declaring loopback a trusted proxy trades the automatic dev-convenience exemption for that peer's own header deciding instead, like any other trusted proxy.
  • A trusted proxy that reports http is refused exactly like an untrusted plaintext request: being trusted lets a peer's header override the connection's own scheme in either direction, never guarantee it is secure.

The health, readiness and metrics endpoints (/healthz, /readyz, /metrics) are exempt from this whole policy, checked first and unconditionally: they carry no secret, and health/readiness probing and metrics scraping are conventionally plain HTTP, internal-network traffic from something that never negotiates TLS and never follows a redirect (an orchestrator's liveness probe, a Prometheus scrape, the Compose example's own container-to-container health check). This was confirmed the hard way in round 1: observability_test.go's tests and the original TestPlaintextRedirectedForSafePaths pinned mutually exclusive behavior for these three paths before the test author narrowed the redirect test to the paths that actually need it.

A request under a disabled module's prefix is also exempt from refusal or redirection: there is nothing behind a disabled module but a static not-enabled response, so plaintext there carries nothing to protect (TestHandlerNotEnabled, TestHandlerPrefixIsExact, both pre-existing tests this policy must not break).

Every other machine endpoint prefix -- /relay, /sync, /share, /accounts -- is refused outright over plaintext (400, never proxied to the module): a redirect would still receive the WebSocket upgrade or the account request in the clear before the client ever saw it. Any other path (none exist on this binary today; a future browser-facing page would land here) redirects to HTTPS instead.

Container image: distroless static, non-root, pinned toolchain. cloud/cmd/vaultfold-cloud/Dockerfile builds with golang:1.27.1-bookworm (the same toolchain go.work and Dockerfile.dev pin, so make check-in-docker and the image build the same binary) and CGO_ENABLED=0 (every dependency, including modernc.org/sqlite, is already cgo-free), then copies the static binary into gcr.io/distroless/static-debian12:nonroot: no shell, no package manager, uid/gid 65532. The build stage also creates an empty /data directory owned by that uid before the final COPY --chown, because a Compose named volume seeds its ownership from what is already at its mount point on first creation; without this, the volume comes up root-owned and the non-root process cannot open or create the SQLite file (relaydb: read schema version: unable to open database file) -- caught by TestContainerHealthWithoutDevtun while building the image for the first time.

docker-compose.yml publishes no host port and joins an external traefik network with Traefik HTTPS-routing labels, per the cloud spec's "Self-hosting" requirement and the project's development convention (docker-compose.override.yml at the repository root does the equivalent for devtun in local development). It also sets VAULTFOLD_CLOUD_TRUSTED_PROXIES, defaulting to empty with a comment telling the operator how to find the Traefik network's real subnet (docker network inspect traefik --format '{{(index .IPAM.Config 0).Subnet}}'): the subnet of an externally-created network cannot be known at compose-authoring time, so hardcoding one would be actively wrong rather than merely incomplete.

Consequences

  • A self-hoster who fronts vaultfold-cloud with any reverse proxy other than one they list in VAULTFOLD_CLOUD_TRUSTED_PROXIES will find every machine endpoint refused or redirected: this is intentional (the service must not guess which peers to trust) but is a new step in bring-up, documented in docs/self-hosting.md.
  • The blob-count and namespaces-over-quota metrics do not exist yet (round 3 removed them rather than expose a permanent 0): a dashboard or alert cannot be built against them until the tracking above lands. This is a deliberate absence, called out in both docs pages and here, not an oversight.
  • AggregateUsage is a new, optional interface on top of UsageStore rather than a new method on UsageStore itself, specifically so it does not force every existing UsageStore implementation (including test doubles) to grow a method they do not need.

Alternatives considered

  • Trusting X-Forwarded-Proto from any loopback-reachable peer, treating "runs on the same host" as sufficient trust. Rejected: this is exactly the "mirror problem" round 2 found, where a same-host proxy's own plaintext report was silently ignored.
  • Computing the blob count and namespaces-over-quota by scanning the blob backend or the token's caveats at scrape time. Rejected for now: no existing index supports either query cheaply, and the schema and behavior changes needed are large enough to warrant their own review round rather than riding along with this one.
  • Keeping the two gauges at a constant 0 until that future work lands. Rejected (round 3, reviewer finding R2-1): a gauge indistinguishable from a genuine "all clear" reading is worse than no gauge, since it can silently suppress an alert. Removed outright instead, to be reintroduced once they report something real.