> ## Documentation Index
> Fetch the complete documentation index at: https://docs.strata.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Break-Glass Procedures

This page documents emergency procedures for making Orchestrator configuration changes when you need to work outside the normal Console workflow. These procedures trade security guarantees (bundle signing, audit trails) for operational continuity.

<Warning>
  Break-glass procedures bypass normal security controls. Use them only when Console-managed changes are not possible or practical and configuration changes are urgently needed. Follow the [Recovery Checklist](#recovery-checklist) after returning to Console-managed operations to re-establish normal workflows.
</Warning>

## When to Use These Procedures

These procedures apply when you need to make Orchestrator configuration changes outside the normal Console workflow:

* **Console is unreachable** -- Network issues, DNS problems, regional connectivity disruption, or service interruption.
* **Account access issues** -- Your account is locked out, permissions have been revoked, or your SSO identity provider is down.
* **Team availability** -- The team member with Console access is unavailable during an emergency.

## Understanding the Constraint

The key constraint when working without Console access is **bundle signing**. The deployment's private signing key is stored only in the Console -- there is no API endpoint to export or download the private key. This means you cannot create new validly-signed bundles without Console access.

The Orchestrator verifies the ECDSA P-256 signature and SHA-256 hash of every file in the bundle before applying configuration. To apply configuration changes without Console access, you must switch to a local file configuration (Options 2 or 3 below), which uses a code path that does not perform signature verification.

The Orchestrator's tar.gz bundler always runs signature verification when loading a bundle -- there is no code path that skips verification based on an empty or missing public key. If `MAVERICS_BUNDLE_PUBLIC_KEY_FILE` is unset, the empty key causes a PEM decode failure rather than bypassing verification. This means you cannot load an unsigned tar.gz bundle by simply removing the signature and unsetting the key. The only way to bypass verification is to switch to a local file configuration, which uses a different bundler that does not perform signature checks.

## Immediate Impact: Working Without Console Access

When working without Console access:

* **No immediate disruption** -- The Orchestrator continues running with its last-known-good configuration.
* **No new publishes** -- Configuration cannot be published through the Console UI.
* **No new revisions** -- No diff previews, revision history, or audit log access.
* **Polling continues** -- The Orchestrator continues polling for new bundles but finds no changes.

If no configuration changes are needed, no action is required. The Orchestrator operates normally with its current configuration.

## Recovery Options

### Option 1: Continue with Current Configuration (No Action Needed)

The Orchestrator continues operating with the last successfully applied configuration. No intervention is required.

**How it works:**

* The Orchestrator keeps its current configuration in memory and continues serving traffic.
* If the Orchestrator restarts, it re-downloads and re-verifies the last bundle from the deployment provider.
* The signed bundle in the deployment provider's storage remains valid and available.

**When to use this option:**

* No configuration changes are needed while working without Console access.
* The current configuration is working correctly.

**Limitations:**

* No way to make any configuration changes until Console access is available.

***

### Option 2: Extract Configuration from Bundle and Switch to Local File

Download the current bundle from the deployment provider, extract the configuration file, and switch the Orchestrator to load it as a local file. This approach lets you make targeted changes to your existing configuration without needing to write a config from scratch.

**Steps:**

<Steps>
  <Step title="Download the current bundle">
    Download `maverics.tar.gz` from your deployment provider's storage location (S3 bucket, Azure Blob container, GCS bucket, or Git repository).

    ```bash theme={null}
    # Example: AWS S3
    aws s3 cp s3://your-bucket/path/maverics.tar.gz ./maverics.tar.gz

    # Example: Azure Blob
    az storage blob download --container-name your-container --name maverics.tar.gz --file ./maverics.tar.gz

    # Example: GCS
    gsutil cp gs://your-bucket/path/maverics.tar.gz ./maverics.tar.gz
    ```
  </Step>

  <Step title="Extract the archive">
    ```bash theme={null}
    mkdir bundle && cd bundle
    tar -xzf ../maverics.tar.gz
    ```
  </Step>

  <Step title="Modify the configuration">
    Edit `maverics.json` with the needed configuration changes.

    ```bash theme={null}
    # Edit the configuration
    vi maverics.json
    ```
  </Step>

  <Step title="Copy the configuration file to the Orchestrator">
    ```bash theme={null}
    # Copy the extracted config to the Orchestrator host
    scp maverics.json orchestrator-host:/etc/maverics/maverics.json
    ```
  </Step>

  <Step title="Switch to local file configuration">
    Point the Orchestrator to the local config file instead of the cloud/Git provider. Remote config source environment variables (`MAVERICS_AWS_CONFIG`, `MAVERICS_AZURE_CONFIG`, `MAVERICS_GCP_CONFIG`, `MAVERICS_GITHUB_CONFIG`, `MAVERICS_GITLAB_CONFIG`) take priority over `MAVERICS_CONFIG`. You **must** unset the remote provider variable for your deployment before setting `MAVERICS_CONFIG`, otherwise the Orchestrator will ignore the local file.

    ```bash theme={null}
    # Unset the remote config source for your provider (pick the one matching your deployment)
    # unset MAVERICS_AWS_CONFIG
    # unset MAVERICS_AZURE_CONFIG
    # unset MAVERICS_GCP_CONFIG
    # unset MAVERICS_GITHUB_CONFIG
    # unset MAVERICS_GITLAB_CONFIG

    export MAVERICS_CONFIG=/etc/maverics/maverics.json
    unset MAVERICS_BUNDLE_PUBLIC_KEY_FILE
    sudo systemctl restart maverics
    ```
  </Step>

  <Step title="Verify the Orchestrator is running">
    The Orchestrator now loads configuration from the local file instead of polling the deployment provider.

    ```bash theme={null}
    sudo systemctl status maverics
    ```
  </Step>
</Steps>

<Note>
  This approach is functionally equivalent to Option 3 but starts from your existing bundle configuration rather than a new YAML file. The extracted `maverics.json` file uses JSON format, which the Orchestrator accepts alongside YAML.
</Note>

**When to use this option:**

* You need to make targeted changes to the existing configuration.
* You want to preserve your current Console-managed configuration as a starting point.

**Limitations:**

* **Manual changes overwritten** -- When returning to Console-managed operations, re-publish from Console and switch the Orchestrator back to its deployment provider by unsetting `MAVERICS_CONFIG` and restoring `MAVERICS_BUNDLE_PUBLIC_KEY_FILE`.
* **Configuration is managed as a local file** -- No audit trail, revision history, or Console-managed features.

***

### Option 3: Switch to Local File Configuration

Replace the bundle-based configuration with a local YAML file. This is the simplest break-glass path for making urgent configuration changes.

**Steps:**

<Steps>
  <Step title="Prepare a local YAML configuration file">
    Create a configuration file for the Orchestrator. You can extract the `maverics.json` configuration from the current bundle (see [Option 2](#option-2-extract-configuration-from-bundle-and-switch-to-local-file) for extraction steps) and use it directly -- the Orchestrator accepts both YAML and JSON formats.

    ```bash theme={null}
    # Example: point to a local config file
    export MAVERICS_CONFIG=/etc/maverics/maverics.yaml
    ```
  </Step>

  <Step title="Disable bundle signature verification">
    Unset the bundle public key variable so the Orchestrator does not attempt to verify a bundle signature:

    ```bash theme={null}
    unset MAVERICS_BUNDLE_PUBLIC_KEY_FILE
    ```
  </Step>

  <Step title="Restart the Orchestrator">
    Restart the Orchestrator so it picks up the local configuration file instead of polling for bundles.

    ```bash theme={null}
    # systemd
    sudo systemctl restart maverics

    # Docker
    docker restart maverics
    ```
  </Step>
</Steps>

**When to use this option:**

* Urgent configuration changes are needed and you have a working YAML configuration.
* You prefer a clean, simple approach over modifying bundles.

**Limitations:**

* Manual configuration management -- no Console-managed objects, no diff previews.
* No audit trail for configuration changes made while working outside the Console.
* No revision history or rollback capability.
* You must switch back to bundle-based configuration when returning to Console-managed operations.

***

### Option 4: Direct Provider Modification (Git-Based Providers)

For GitHub and GitLab deployment providers, you can extract the configuration from the bundle and use Git for version tracking during the emergency.

<Warning>
  Pushing a modified tar.gz bundle to a Git repository has the same signature verification constraint described in [Understanding the Constraint](#understanding-the-constraint). You cannot load an unsigned tar.gz bundle by removing the signature file. Instead, follow these steps to extract the configuration and switch to a local file, using Git for version tracking.
</Warning>

**Steps:**

<Steps>
  <Step title="Clone the deployment repository">
    ```bash theme={null}
    git clone https://github.com/your-org/your-deployment-repo.git
    cd your-deployment-repo
    ```
  </Step>

  <Step title="Extract the configuration from the bundle">
    Extract `maverics.json` from the tar.gz bundle in the repository.

    ```bash theme={null}
    mkdir extracted && cd extracted
    tar -xzf ../maverics.tar.gz
    ```
  </Step>

  <Step title="Modify the configuration">
    Edit `maverics.json` with the needed configuration changes.

    ```bash theme={null}
    vi maverics.json
    ```
  </Step>

  <Step title="Track changes in Git">
    Commit the extracted config file (not the tar.gz) for auditability.

    ```bash theme={null}
    cp maverics.json ../
    cd ..
    git add maverics.json
    git commit -m "break-glass: emergency config change - [describe change]"
    git push
    ```
  </Step>

  <Step title="Switch the Orchestrator to local file configuration">
    Copy `maverics.json` to the Orchestrator and point it to the local config file, following the same approach as Options 2 and 3. You must unset the Git provider variable first -- remote config source variables take priority over `MAVERICS_CONFIG`.

    ```bash theme={null}
    scp maverics.json orchestrator-host:/etc/maverics/maverics.json

    # Unset the Git provider config source (pick the one matching your deployment)
    # unset MAVERICS_GITHUB_CONFIG
    # unset MAVERICS_GITLAB_CONFIG

    export MAVERICS_CONFIG=/etc/maverics/maverics.json
    unset MAVERICS_BUNDLE_PUBLIC_KEY_FILE
    sudo systemctl restart maverics
    ```
  </Step>
</Steps>

**When to use this option:**

* Your deployment uses a GitHub or GitLab provider and you want Git history to provide auditability during the emergency.

**Limitations:**

* **The Orchestrator loads from a local file, not from the Git repository directly** -- Git provides change tracking only.
* **Same limitations as Options 2 and 3** -- No Console-managed features during the emergency period.

## Recovery Checklist

After returning to Console-managed operations, follow this checklist to re-establish normal workflows.

<Steps>
  <Step title="Verify Console connectivity">
    Confirm that the Console is accessible and your organization's data is intact.
  </Step>

  <Step title="Re-publish from Console">
    Publish a new revision from the Console to overwrite any manual changes with a properly signed bundle. This ensures all Orchestrator instances are running Console-managed, signed configuration.
  </Step>

  <Step title="Re-enable signature verification">
    On every Orchestrator instance where verification was disabled, restore the `MAVERICS_BUNDLE_PUBLIC_KEY_FILE` environment variable to the public key path and restart the Orchestrator. Also unset `MAVERICS_CONFIG` if it was set during the emergency, so the Orchestrator returns to its deployment provider.

    ```bash theme={null}
    unset MAVERICS_CONFIG
    export MAVERICS_BUNDLE_PUBLIC_KEY_FILE=/etc/maverics/public_key.pem
    sudo systemctl restart maverics
    ```
  </Step>

  <Step title="Review audit logs">
    Check the Console audit logs for events that occurred while working outside the Console. There will be a gap in Console-side events during that period -- document this gap.
  </Step>

  <Step title="Document manual changes">
    Record what configuration changes were made manually while working outside the Console, why they were needed, and which recovery option was used. This documentation supports compliance requirements.
  </Step>
</Steps>

## Preparation: Maintaining Fallback Options

Prepare fallback options so you can respond quickly when Console access is unavailable.

* **Periodically download signed bundles** -- Download and archive signed bundles from the Console as backups. A valid signed bundle can be re-deployed without Console access (no signature verification bypass needed).
* **Document provider credentials separately** -- Store your deployment provider credentials (S3 access keys, Azure tokens, Git tokens) in a location accessible independently of the Console.
* **Record your public key path** -- Know where `MAVERICS_BUNDLE_PUBLIC_KEY_FILE` points and how to unset it on each Orchestrator instance.
* **Understand config source priority** -- Remote config source environment variables (`MAVERICS_AWS_CONFIG`, `MAVERICS_AZURE_CONFIG`, etc.) take priority over `MAVERICS_CONFIG`. When switching to local file mode, you must unset the active remote provider variable first, otherwise the Orchestrator will continue using the remote source.

## Related Pages

<CardGroup cols={2}>
  <Card title="Publishing Deployment Configs Overview" icon="upload" href="/reference/console/config-publishing">
    Bundle format, signing, deployment providers, and revision history
  </Card>

  <Card title="File Config Source" icon="file" href="/reference/orchestrator/configuration/config-sources/file">
    Load Orchestrator configuration from a local YAML file
  </Card>

  <Card title="Deploy to Production" icon="rocket" href="/guides/operations/deploy">
    Production deployment guide for the Orchestrator
  </Card>

  <Card title="Console Overview" icon="browser" href="/reference/console/overview">
    Console administration overview and capabilities
  </Card>
</CardGroup>