Skip to main content

Credential proxy cutover and recovery

This is a breaking migration from the separate per-owner proxy and user-held unlock key to the Factorize backend proxy. The release removes the old proxy, unlock, pairing and run-bearer endpoints. Guest access requires a fresh vendor signature bound to the exact run VM; there is no Factorize VM token fallback. Credentials are encrypted with the server’s CREDENTIAL_ENCRYPTION_KEY. The backend can migrate its own managed credentials, but cannot recover secrets inside the old user encryption. Those records require owner re-entry. Do this during a maintenance window. Migration 0052_retire_credential_proxy refuses nonterminal runs or existing credential records without operator preparation. This document describes deployment work; repository tests do not mean the production cutover has already happened.

Before the cutover

  1. Announce a maintenance window and pause schedules, webhook events, and other dispatch sources for affected triggers. Prevent new runs from starting while the inventory is being prepared.
  2. Inventory each tenant’s triggers, selected portable integration IDs, model account IDs, and the owner of each credential. Keep this inventory private; it must contain identifiers and owners, never secret values.
  3. Let active runs finish. For runs that cannot finish, stop them through the Factorize run controls and verify that each is terminal before proceeding. Do not stop a shared execution app or VM to drain one tenant’s runs.
  4. Pause credential writes as well as all dispatch. With the pre-cutover schema (through migration 0051), run the read-only preparation plan:
    Supply DATABASE_URL and CREDENTIAL_ENCRYPTION_KEY through the operator’s secret environment. The report contains identifiers and trigger selections, not secret values. Confirm every recorded proxy_identity against vendor inventory; distinguish proxy-only apps from execution resources.
  5. Review the plan, then run the same command with --apply, preserving its report privately. This disables triggers selecting legacy user ciphertext, leaves their credential IDs and execution configuration intact, tombstones those unreadable credentials, clears the retired change feed and revokes old proxy control/enrollment state. Managed active credentials, GitHub and Linear connections, Fly connections and execution configurations survive.
  6. Keep dispatch and writes paused. Stop and verify deletion of only the inventoried old proxy Machines/VMs, shared proxy integrations and proxy-only apps through documented vendor controls. Record resource identifiers and absence checks. The preparation script does not delete vendor resources.
  7. Apply migration 0052 and deploy the backend and static application together following the package migration release gates. Do not deploy this release over an undrained database. The migration checks for credential writes after preparation and drops the obsolete proxy, enrollment, unlock and generation tables. Remove obsolete proxy image/build secrets and model browser capability signing keys from deployment configuration.
  8. Verify live canary runs on exe.dev and Fly: exact run identity, selected and unselected credential behavior, empty scopes, cancellation, artifacts and cleanup. exe.dev requires unique per-run WIF attachment and inventory access; Fly requires organization OIDC discovery/JWKS and guest Unix-socket tokens. Fix failed canaries before resuming dispatch.
If any run remains active, or a resource identity cannot be established, keep maintenance in place. Never stop a shared execution app to retire one proxy.

Reconnect credentials and repair triggers

  1. Keep existing GitHub and Linear connections; their upstream permissions do not expand. Owners reconnect unavailable model accounts using an OpenAI-compatible provider URL and optional access key, and re-enter unreadable custom bearer credentials. New IDs may need explicit trigger repair. Existing server-managed records retain their IDs and do not require unnecessary reconnection.
  2. Compare the new connection metadata with the private inventory. Update each affected trigger to select the corresponding new credential ID. The cutover must not select every tenant credential automatically. A trigger that cannot be mapped confidently stays disabled until its owner repairs it.
  3. Verify each saved selection belongs to that trigger’s tenant and that the selected account is connected. Run one canary per repaired trigger and confirm its outbound requests use only those selections. Confirm Factorize API scopes match the trigger’s configured scopes.
  4. Keep triggers with no portable selections unchanged. Do not add a credential to a trigger merely because it was available to the retired proxy.
  5. Resume dispatch only after the affected trigger inventory is reconciled. If a credential cannot be reconnected, leave that selection unavailable and the trigger disabled; do not substitute another credential silently.

Recovery

If a canary fails, keep dispatch paused and repair the new backend or the specific credential selection. After migration 0052, the old backend is not a compatible rollback target: it depends on removed tables and credentials. Never restore old credentials or user ciphertext to revive access. A rollback requires a separately reviewed deployment/database recovery plan with dispatch paused; ordinary recovery proceeds forward on the managed architecture.

Completion record

Record the maintenance window, terminal-run count, retired proxy app/Machine identifiers, repaired trigger count, triggers left disabled for owner repair, and canary results. Keep secret values, bearer tokens, unlock keys, and decrypted credential material out of the record.