Service-to-Service Auth

VeloGit’s backend services (rituals-service, rituals-dispatcher, agent-service) call the data-plane’s /internal/... endpoints to record pipeline runs, fetch secrets, post fragments, and so on. These calls are authenticated as service identities, not as human users.

Two credential classes are recognised:

Zitadel service-user bearer token — a short-lived JWT minted by each persistent service (agent-service, rituals-dispatcher) from its own private key. Recognised when the token’s introspected identity appears in the data-plane’s VELO_SERVICE_USER_NAMES env var.
Per-run scoped token (velosys_<hex>) — minted by the dispatcher when it spawns a runner machine, bound to a specific pipeline run, org, and repo. Used by rituals-service for every internal API call and the git clone Basic Auth password.

The legacy X-Internal-Api-Key shared secret was removed in Phase 3.7 — no path in the data-plane accepts it anymore.

Provisioning a service user

For each backend service:

Create the machine user in the Zitadel admin console (Users → Service Users → New). Give it a recognisable username such as velogit-rituals. Set Authentication type to JWT, not Bearer — Bearer hands you a static long-lived token (defeats the purpose of moving off INTERNAL_API_KEY); JWT issues short-lived tokens minted on demand.
Add a key to the service user (Keys tab → New → JSON). Download the generated key.json immediately — Zitadel does not store the private key. If you miss it, delete the key and generate a new one.
Grant the user access to your project. This step’s UI label varies by Zitadel version: it might be a “Grants” tab on the project, an “Authorizations” tab, or a “Memberships” section on the user. The underlying action is the same — give the service user a role on the project so introspection can resolve its identity. Any role works for now; scoped roles come in Phase 3.
Set the key and project ID as Fly secrets on the service that needs it:
Terminal window
```
flyctl secrets set \
  VELO_SERVICE_KEY_JSON="$(cat key.json)" \
  ZITADEL_PROJECT_ID=<numeric project id> \
  -a velogit-agent-service
```
ZITADEL_PROJECT_ID is the project that owns the data-plane’s application — same project the data-plane’s own ZITADEL_KEY_JSON belongs to. Without it, minted tokens have no audience claim and introspection at the data-plane returns 401. The numeric ID is visible in the Zitadel admin URL when viewing the project.
Tell the data-plane that the identity is a service. You can list service users by Zitadel username (preferred) or by numeric user ID — the data-plane accepts either:
Terminal window
```
flyctl secrets set VELO_SERVICE_USER_NAMES=velogit-agent,velogit-rituals \
  -a velogit-data-plane
```
The serviceauth client requests the profile scope automatically so introspection returns the friendly username. If your tokens come back with only a numeric sub (e.g. 375566056376297924), set that ID in VELO_SERVICE_USER_NAMES instead.

Service-specific notes

rituals-service is bundled with data-plane

The rituals-service binary is embedded in the data-plane Docker image (Dockerfile.data-plane compiles rituals-runner and copies it into the final stage). Spawned ritual machines download the binary from the data-plane’s /internal/runner-binary endpoint at boot.

This has one significant consequence: to ship rituals-service changes you must redeploy the data-plane, not the rituals-dispatcher. Redeploying just the dispatcher only updates the orchestrator binary, not the runner binary served to spawned machines.

./scripts/deploy-fly.sh data-plane   # ships rituals-service changes
./scripts/deploy-fly.sh rituals      # only updates the dispatcher

rituals-service authenticates per-run, not per-service

Unlike the other backend services, rituals-service does not carry a Zitadel service identity. The dispatcher mints a per-run scoped token (velosys_<hex>) before spawning the machine and injects it as VELO_RUN_TOKEN. The runner uses that token for every internal API call.

The token is bound to the specific pipeline run, org, and repo. The data-plane rejects any request whose URL targets a different org or run. A compromised runner machine therefore has no credential capable of acting outside the run it was spawned for — even within the same org.

Operational consequences:

No VELO_SERVICE_KEY_JSON on velogit-rituals-dispatcher is needed for the runner’s account. The dispatcher’s own identity is used to mint tokens; the runner inherits nothing.
The velogit-rituals Zitadel service user remains in VELO_SERVICE_USER_NAMES because rituals-dispatcher still uses it for its own internal calls (creating pipeline runs, minting tokens, updating provider job IDs).
Tokens are revoked automatically when the run reaches a terminal state (passed / failed / rejected / canceled / stale-reaped). A leaked token becomes invalid the moment the run finishes.

webhook-dispatcher needs no migration

webhook-dispatcher talks directly to Postgres for webhook configs and delivery records, and the only HTTP traffic it makes is outbound to customer webhook URLs — not to the data-plane. Setting VELO_SERVICE_KEY_JSON on it has no effect; the variable will sit unused.

What changes for handler code

Existing handlers gated on isInternalRequest(c) keep working unchanged. After authentication, the middleware sets:

Context key	Value
`is_internal`	`true` for both X-Internal-Api-Key and service-token auth
`auth_method`	`internal_api_key`, `service_token`, `pat`, or `user_token`
`service_name`	The matched username or user ID when authenticated via token

Use auth_method and service_name for structured logging when auditing internal calls.

Troubleshooting

Symptom in data-plane logs	Likely cause	Fix
`401 unauthorized` on `/internal/...`	Token missing project audience or service user not granted on project	Verify `ZITADEL_PROJECT_ID` on the calling service matches the data-plane’s project; grant the service user a role on that project
`403 internal only` with WARN `not service` log	Introspected identity not in `VELO_SERVICE_USER_NAMES`	The WARN log surfaces the exact `introspected_username` / `introspected_user_id` — add whichever matches to the env var
No `🔐` startup log on the service	Env var not set or service not redeployed with new code	Confirm `VELO_SERVICE_KEY_JSON` is set on the right Fly app; redeploy
rituals-service still on legacy path	Data-plane not redeployed with the new bundled runner binary	`./scripts/deploy-fly.sh data-plane`

Operational rollout completed

As of Phase 3.7 the shared INTERNAL_API_KEY is no longer accepted by the data-plane on any path. The bootstrap binary download from spawned machines uses the per-run scoped token via Authorization: Bearer $VELO_RUN_TOKEN, and the git clone Basic Auth password is the same scoped token. You can unset INTERNAL_API_KEY across all Fly apps after deploying the data-plane:

flyctl secrets unset INTERNAL_API_KEY -a velogit-data-plane
flyctl secrets unset INTERNAL_API_KEY -a velogit-agent-service
flyctl secrets unset INTERNAL_API_KEY -a velogit-rituals-dispatcher
flyctl secrets unset INTERNAL_API_KEY -a velogit-webhook-dispatcher

Why this matters

Per-service identity in logs and audit trails — you can finally tell whether a /internal/pipeline-runs write came from rituals-service or someone replaying a key.
Short-lived tokens — leaked tokens expire in ~12h vs. a static secret that lives forever in environment configuration.
Revocation without redeploy — disable a service in Zitadel and its next token refresh fails; no machine restart needed.
Foundation for scoped permissions — once roles are wired in (Phase 3), per-route enforcement (rituals-service can write pipeline runs, but not agent fragments) becomes a one-line check.