0026: Operations and self-hosting
- Status: accepted
- Date: 2026-09-26
Context
add-cloud-service task group 9 gives cloud/cmd/vaultfold-cloud the
operational surface a self-hoster or the vaultfold-operated instance
needs: health, readiness and metrics endpoints, the transport security
policy ADR 0015 requires, and a container image with a Compose example.
Two review rounds on ticket 01M3A20PWH4B5RWMR1QT6W7HQW found real gaps
between the first pass and the cloud spec's "Health, readiness and
metrics" and "Transport security and headers" requirements; this ADR
records the rulings that closed them, as design.md's "Group 9 round 2"
note said it would.
Decisions
Metrics: fixed labels, aggregate-only quota. A metric label is never
derived from request data. MetricHTTPRequestsTotal and
MetricHTTPRequestDurationSeconds label by HTTP method and a path
template, but the method itself is mapped to a fixed set (GET, HEAD,
POST, PUT, DELETE, OPTIONS, else OTHER) before it ever reaches a label:
Go's HTTP server accepts any RFC 7230 token as a method, so recording
r.Method verbatim let a client choose the label value (an address- or
token-shaped string) or mint one new series per request, an
unbounded-memory vector as well as a potential address leak (review
round 2, finding B1). The path template is a caller-chosen constant
(/relay, /sync, PathHealthz, and so on), never the request's actual
path, so a sync namespace or a share id never reaches a label either.
Quota usage (the cloud spec's "Metrics content" scenario) is reported
only as one unlabelled aggregate gauge, vaultfold_cloud_quota_charged_bytes
("bytes charged against quotas"), never broken out per token or per
namespace (review round 2, finding B6). It sums cloud/sync.UsageStore's
per-token byte counts via an optional AggregateUsage interface
(TotalStoredBytes), which MemUsageStore and cloud/relaydb.DB both
implement; a UsageStore that does not implement it (or no token-mode
usage store at all) simply reports 0.
Round 2 also shipped a total-blob-count gauge and a
namespaces-over-quota gauge, both hard-wired to a constant 0, since
neither per-item counting nor centralized quota-limit tracking exists yet.
Reviewer round 2 (finding R2-1) flagged this as actively misleading: a
gauge that can never be anything but 0 is indistinguishable from a
genuine "nothing over quota" reading, so an alert built on it could never
fire -- worse than not exposing the metric at all. The orchestrator's
amended ruling (round 3) removed both outright rather than leave them at
a placeholder; they are deferred in the cloud spec and here until the
tracking that would make them meaningful exists: a per-item counter
(likely a new column alongside cloud/relaydb's usage table) for the
blob count, and somewhere to persist a quota's limit centrally (today a
quota is only a caveat carried on the presented token itself, compared
against usage at request time, never stored) for namespaces-over-quota.
Separately, reviewer round 2 also nitted the surviving gauge's original
name, vaultfold_cloud_quota_stored_bytes_total: a counter-style
_total suffix on a value that is a gauge (it can go down), and "stored
bytes" overstated what it measures -- the same per-token accounting
Claims.AdmitBlobWrite checks against a quota caveat, not an independent
count of bytes actually resident in the blob store. Renamed to
vaultfold_cloud_quota_charged_bytes (round 3); the Go method
(AggregateUsage.TotalStoredBytes) keeps its name, since renaming it
would have required editing test files that already reference it, which
the implementer role does not do.
Transport security: an explicit trusted-proxy list. The first pass
honored X-Forwarded-Proto from any peer and treated any loopback peer
as exempt unconditionally. Both were exploitable (review round 2, finding
B2, reproduced against a directly exposed instance): a plaintext request
carrying X-Forwarded-Proto: https reached a machine endpoint from a
non-loopback address, and a same-host proxy's own X-Forwarded-Proto: http was silently overridden by the loopback exemption.
The fix is VAULTFOLD_CLOUD_TRUSTED_PROXIES, a comma-separated CIDR
list, empty by default (config.go's EnvTrustedProxies,
Config.TrustedProxies):
- A forwarded protocol header is honored only when the immediate peer
(
r.RemoteAddr) falls in that list. From any other peer it is ignored entirely; the connection's own scheme (always plaintext for this service, which never terminates TLS itself) decides. - The loopback exemption applies only when all three hold: the peer is loopback, no forwarded protocol header is present at all, and loopback is not itself listed as a trusted proxy. Declaring loopback a trusted proxy trades the automatic dev-convenience exemption for that peer's own header deciding instead, like any other trusted proxy.
- A trusted proxy that reports
httpis refused exactly like an untrusted plaintext request: being trusted lets a peer's header override the connection's own scheme in either direction, never guarantee it is secure.
The health, readiness and metrics endpoints (/healthz, /readyz,
/metrics) are exempt from this whole policy, checked first and
unconditionally: they carry no secret, and health/readiness probing and
metrics scraping are conventionally plain HTTP, internal-network traffic
from something that never negotiates TLS and never follows a redirect (an
orchestrator's liveness probe, a Prometheus scrape, the Compose example's
own container-to-container health check). This was confirmed the hard
way in round 1: observability_test.go's tests and the original
TestPlaintextRedirectedForSafePaths pinned mutually exclusive behavior
for these three paths before the test author narrowed the redirect test
to the paths that actually need it.
A request under a disabled module's prefix is also exempt from refusal
or redirection: there is nothing behind a disabled module but a static
not-enabled response, so plaintext there carries nothing to protect
(TestHandlerNotEnabled, TestHandlerPrefixIsExact, both pre-existing
tests this policy must not break).
Every other machine endpoint prefix -- /relay, /sync, /share,
/accounts -- is refused outright over plaintext (400, never proxied to
the module): a redirect would still receive the WebSocket upgrade or the
account request in the clear before the client ever saw it. Any other
path (none exist on this binary today; a future browser-facing page would
land here) redirects to HTTPS instead.
Container image: distroless static, non-root, pinned toolchain.
cloud/cmd/vaultfold-cloud/Dockerfile builds with golang:1.27.1-bookworm
(the same toolchain go.work and Dockerfile.dev pin, so
make check-in-docker and the image build the same binary) and
CGO_ENABLED=0 (every dependency, including modernc.org/sqlite, is
already cgo-free), then copies the static binary into
gcr.io/distroless/static-debian12:nonroot: no shell, no package
manager, uid/gid 65532. The build stage also creates an empty /data
directory owned by that uid before the final COPY --chown, because a
Compose named volume seeds its ownership from what is already at its
mount point on first creation; without this, the volume comes up
root-owned and the non-root process cannot open or create the SQLite
file (relaydb: read schema version: unable to open database file) --
caught by TestContainerHealthWithoutDevtun while building the image for
the first time.
docker-compose.yml publishes no host port and joins an external
traefik network with Traefik HTTPS-routing labels, per the cloud spec's
"Self-hosting" requirement and the project's development convention
(docker-compose.override.yml at the repository root does the
equivalent for devtun in local development). It also sets
VAULTFOLD_CLOUD_TRUSTED_PROXIES, defaulting to empty with a comment
telling the operator how to find the Traefik network's real subnet
(docker network inspect traefik --format '{{(index .IPAM.Config 0).Subnet}}'): the subnet of an externally-created network cannot be
known at compose-authoring time, so hardcoding one would be actively
wrong rather than merely incomplete.
Consequences
- A self-hoster who fronts vaultfold-cloud with any reverse proxy other
than one they list in
VAULTFOLD_CLOUD_TRUSTED_PROXIESwill find every machine endpoint refused or redirected: this is intentional (the service must not guess which peers to trust) but is a new step in bring-up, documented in docs/self-hosting.md. - The blob-count and namespaces-over-quota metrics do not exist yet (round 3 removed them rather than expose a permanent 0): a dashboard or alert cannot be built against them until the tracking above lands. This is a deliberate absence, called out in both docs pages and here, not an oversight.
AggregateUsageis a new, optional interface on top ofUsageStorerather than a new method onUsageStoreitself, specifically so it does not force every existingUsageStoreimplementation (including test doubles) to grow a method they do not need.
Alternatives considered
- Trusting
X-Forwarded-Protofrom any loopback-reachable peer, treating "runs on the same host" as sufficient trust. Rejected: this is exactly the "mirror problem" round 2 found, where a same-host proxy's own plaintext report was silently ignored. - Computing the blob count and namespaces-over-quota by scanning the blob backend or the token's caveats at scrape time. Rejected for now: no existing index supports either query cheaply, and the schema and behavior changes needed are large enough to warrant their own review round rather than riding along with this one.
- Keeping the two gauges at a constant 0 until that future work lands. Rejected (round 3, reviewer finding R2-1): a gauge indistinguishable from a genuine "all clear" reading is worse than no gauge, since it can silently suppress an alert. Removed outright instead, to be reintroduced once they report something real.