Every surface on this platform sits behind Keycloak and oauth2-proxy. A personal access token authenticates the API; the browser path needs a session, which is a different credential entirely — so for a long time nobody, and no automation, could check that a shipped change actually behaved on the live authenticated view. Work shipped on image digests and stub runs instead (#2132).
There is now a dedicated verification identity and a Playwright harness that uses it to sign in to production non-interactively and drive the real thing. CI runs it and an agent runs it, through the same script, so “verified on production” means one thing whoever says it.
SQL connectivity → covers the other direction: connecting a client to the data planes.
One entry point, from a checkout of the repository, on a machine with cluster access. It reads the credential out of the secret store itself — you never see it, type it, or store it.
scripts/verify-live.sh # production
scripts/verify-live.sh --target sandbox # sandbox
scripts/verify-live.sh --spec catalog-sample-width-2126 # one specIn CI it is the verify-live workflow: dispatchable on demand, and run automatically after a release rolls to production. The workflow calls the same script with the same arguments — deliberately not a reimplementation of it, because the moment CI has its own private recipe the two claims diverge.
If the credential cannot be read, the run fails. It never falls back to an unauthenticated session: a green report over a logged-out view is worse than no report.
The harness performs the actual login. There is no bypass anywhere in the path, which is what makes the result worth anything:
vforce-flow) — a real authorization code flow./oauth2/callback; oauth2-proxy mints its redis-backed session and the app loads signed in.Every layer runs: PKCE, the JWKS check the API performs on the forwarded access token, the tenant isolation gate, and the ReBAC decision on each read. The session is indistinguishable from a person’s except in what it is permitted to do.
Fresh per run, held between none. Each test starts from a cold browser context and signs in again; no storageState is written, so there is no session artifact on disk to go stale or to leak.
The subject is svc-verify@edgentllc.com — a service identity, not a person, and unmistakable as one in any trail it appears in.
| It can | It cannot |
|---|---|
| Sign in to the hosts under test through the normal login, as any staff address does. | Hold any Keycloak realm role, client role or group beyond the realm default. |
Read the catalogs whose surfaces are under test — orchestration (the Catalog Sample Data grids, #2126) and cedms (the Review queue, #2129). | Read anything else. Not datasets, events, forensics, isd, main, rcsd or scratch; not another tenant. |
| Observe. That is the entire job. | Write anything, ever. Twice over: it holds only viewer relations, so ReBAC denies writes on its own — and independently of that, the API refuses every non-idempotent request from a declared verification subject before the router sees it, and records the attempt. |
| Prove its own ceiling, live, on every run. | Be a platform administrator or a tenant administrator. The reference spec asserts both against /api/v1/me each run rather than trusting a one-time setup. |
| Be recognized in the audit chain as verification traffic. | Read the audit chain. A credential that could check it was audited would be a credential holding audit-read authority. |
Adding a surface to the verification scope means adding a grant, which is a reviewed pull request: verificationGrants in the chart, pinned by a test that fails if the relation is anything but viewer, if the object is outside the declared scope, or if the subject is not a declared verification identity.
In the platform secret store, and nowhere else: Kubernetes Secret vforce-verification-identity in namespace vforce-lakehouse-dev, keys username and password.
Not in this repository, not in a values file, not in a memory note, not in a chat message, and deliberately not as a GitHub Actions secret — a second copy in a second store is a second rotation story and a second leak surface. One store, one copy.
This is a rule with a history. The previous end-to-end credential leaked because its plaintext was written into a notes file, and it was a personal-account full-access token. This one is neither personal nor writable, and its value is read only at the moment a run needs it.
Rotation: reset the password in Keycloak and update the Secret. Nothing in git changes, no deploy is required, and no session is disturbed — the harness holds none.
A verification session is a real session and emits like any other. A test identity that leaves no trace is a hole, not a convenience.
LOGIN event, swept into the tenant’s tamper-evident chain by the platform’s Keycloak audit sweeper exactly like a person’s.verification: true in its detail, so an operator can separate harness traffic from human traffic without keeping a list of robot addresses in their head.access_denied with reason verification_readonly_refused — a harness that tried to write is precisely the event worth having.The marker is stamped from the identity the API verified, never from a header a caller can type, so nobody can launder their traffic as verification traffic — or strip the mark off their own.
Copy frontend/e2e-live/verification-identity.spec.ts. Every live spec is three lines of setup and then ordinary Playwright:
import { expect, test } from '@playwright/test';
import { assertSignedIn, openAs } from './session';
test('the thing you actually want to know', async ({ page }) => {
await openAs(page, '/catalog?sel=orchestration.events.project_events&tab=sample');
await assertSignedIn(page);
// ...assert a behavior on the real surface
});assertSignedIn is not ceremony. The way a live suite silently becomes worthless is that the browser lands on an anonymous or logged-out view, the surface renders its empty state, nothing contradicts the assertions, and the run reports green having verified nothing. Asserting whose session it is, before measuring anything, is what makes a green run mean something.
Two rules the existing specs follow and yours should: assert geometry and behavior, not the presence of a CSS rule — a grid can paint plausibly while its layout is wrong — and fail loudly on a surface with nothing to measure rather than passing over an empty table.
Two things this mechanism deliberately does not do, both named in #2132: