Credential proxy cutover and recovery
This is a breaking migration from the separate per-owner proxy and user-held unlock key to the Factorize backend proxy. The release removes the old proxy, unlock, pairing and run-bearer endpoints. Guest access requires a fresh vendor signature bound to the exact run VM; there is no Factorize VM token fallback. Credentials are encrypted with the server’sCREDENTIAL_ENCRYPTION_KEY. The
backend can migrate its own managed credentials, but cannot recover secrets
inside the old user encryption. Those records require owner re-entry.
Do this during a maintenance window. Migration 0052_retire_credential_proxy
refuses nonterminal runs or existing credential records without operator
preparation. This document describes deployment work; repository tests do not
mean the production cutover has already happened.
Before the cutover
- Announce a maintenance window and pause schedules, webhook events, and other dispatch sources for affected triggers. Prevent new runs from starting while the inventory is being prepared.
- Inventory each tenant’s triggers, selected portable integration IDs, model account IDs, and the owner of each credential. Keep this inventory private; it must contain identifiers and owners, never secret values.
- Let active runs finish. For runs that cannot finish, stop them through the Factorize run controls and verify that each is terminal before proceeding. Do not stop a shared execution app or VM to drain one tenant’s runs.
-
Pause credential writes as well as all dispatch. With the pre-cutover
schema (through migration 0051), run the read-only preparation plan:
Supply
DATABASE_URLandCREDENTIAL_ENCRYPTION_KEYthrough the operator’s secret environment. The report contains identifiers and trigger selections, not secret values. Confirm every recordedproxy_identityagainst vendor inventory; distinguish proxy-only apps from execution resources. -
Review the plan, then run the same command with
--apply, preserving its report privately. This disables triggers selecting legacy user ciphertext, leaves their credential IDs and execution configuration intact, tombstones those unreadable credentials, clears the retired change feed and revokes old proxy control/enrollment state. Managed active credentials, GitHub and Linear connections, Fly connections and execution configurations survive. - Keep dispatch and writes paused. Stop and verify deletion of only the inventoried old proxy Machines/VMs, shared proxy integrations and proxy-only apps through documented vendor controls. Record resource identifiers and absence checks. The preparation script does not delete vendor resources.
- Apply migration 0052 and deploy the backend and static application together following the package migration release gates. Do not deploy this release over an undrained database. The migration checks for credential writes after preparation and drops the obsolete proxy, enrollment, unlock and generation tables. Remove obsolete proxy image/build secrets and model browser capability signing keys from deployment configuration.
- Verify live canary runs on exe.dev and Fly: exact run identity, selected and unselected credential behavior, empty scopes, cancellation, artifacts and cleanup. exe.dev requires unique per-run WIF attachment and inventory access; Fly requires organization OIDC discovery/JWKS and guest Unix-socket tokens. Fix failed canaries before resuming dispatch.
Reconnect credentials and repair triggers
- Keep existing GitHub and Linear connections; their upstream permissions do not expand. Owners reconnect unavailable model accounts using an OpenAI-compatible provider URL and optional access key, and re-enter unreadable custom bearer credentials. New IDs may need explicit trigger repair. Existing server-managed records retain their IDs and do not require unnecessary reconnection.
- Compare the new connection metadata with the private inventory. Update each affected trigger to select the corresponding new credential ID. The cutover must not select every tenant credential automatically. A trigger that cannot be mapped confidently stays disabled until its owner repairs it.
- Verify each saved selection belongs to that trigger’s tenant and that the selected account is connected. Run one canary per repaired trigger and confirm its outbound requests use only those selections. Confirm Factorize API scopes match the trigger’s configured scopes.
- Keep triggers with no portable selections unchanged. Do not add a credential to a trigger merely because it was available to the retired proxy.
- Resume dispatch only after the affected trigger inventory is reconciled. If a credential cannot be reconnected, leave that selection unavailable and the trigger disabled; do not substitute another credential silently.