Skip to main content
This page documents emergency procedures for making Orchestrator configuration changes when you need to work outside the normal Console workflow. These procedures trade security guarantees (bundle signing, audit trails) for operational continuity.
Break-glass procedures bypass normal security controls. Use them only when Console-managed changes are not possible or practical and configuration changes are urgently needed. Follow the Recovery Checklist after returning to Console-managed operations to re-establish normal workflows.

​
When to Use These Procedures

These procedures apply when you need to make Orchestrator configuration changes outside the normal Console workflow:
  • Console is unreachable — Network issues, DNS problems, regional connectivity disruption, or service interruption.
  • Account access issues — Your account is locked out, permissions have been revoked, or your SSO identity provider is down.
  • Team availability — The team member with Console access is unavailable during an emergency.

​
Understanding the Constraint

The key constraint when working without Console access is bundle signing. The deployment’s private signing key is stored only in the Console — there is no API endpoint to export or download the private key. This means you cannot create new validly-signed bundles without Console access. The Orchestrator verifies the ECDSA P-256 signature and SHA-256 hash of every file in the bundle before applying configuration. To apply configuration changes without Console access, you must switch to a local file configuration (Options 2 or 3 below), which uses a code path that does not perform signature verification. The Orchestrator’s tar.gz bundler always runs signature verification when loading a bundle — there is no code path that skips verification based on an empty or missing public key. If MAVERICS_BUNDLE_PUBLIC_KEY_FILE is unset, the empty key causes a PEM decode failure rather than bypassing verification. This means you cannot load an unsigned tar.gz bundle by simply removing the signature and unsetting the key. The only way to bypass verification is to switch to a local file configuration, which uses a different bundler that does not perform signature checks.

​
Immediate Impact: Working Without Console Access

When working without Console access:
  • No immediate disruption — The Orchestrator continues running with its last-known-good configuration.
  • No new publishes — Configuration cannot be published through the Console UI.
  • No new revisions — No diff previews, revision history, or audit log access.
  • Polling continues — The Orchestrator continues polling for new bundles but finds no changes.
If no configuration changes are needed, no action is required. The Orchestrator operates normally with its current configuration.

​
Recovery Options

​
Option 1: Continue with Current Configuration (No Action Needed)

The Orchestrator continues operating with the last successfully applied configuration. No intervention is required. How it works:
  • The Orchestrator keeps its current configuration in memory and continues serving traffic.
  • If the Orchestrator restarts, it re-downloads and re-verifies the last bundle from the deployment provider.
  • The signed bundle in the deployment provider’s storage remains valid and available.
When to use this option:
  • No configuration changes are needed while working without Console access.
  • The current configuration is working correctly.
Limitations:
  • No way to make any configuration changes until Console access is available.

​
Option 2: Extract Configuration from Bundle and Switch to Local File

Download the current bundle from the deployment provider, extract the configuration file, and switch the Orchestrator to load it as a local file. This approach lets you make targeted changes to your existing configuration without needing to write a config from scratch. Steps:
1

Download the current bundle

Download maverics.tar.gz from your deployment provider’s storage location (S3 bucket, Azure Blob container, GCS bucket, or Git repository).
# Example: AWS S3
aws s3 cp s3://your-bucket/path/maverics.tar.gz ./maverics.tar.gz

# Example: Azure Blob
az storage blob download --container-name your-container --name maverics.tar.gz --file ./maverics.tar.gz

# Example: GCS
gsutil cp gs://your-bucket/path/maverics.tar.gz ./maverics.tar.gz
2

Extract the archive

mkdir bundle && cd bundle
tar -xzf ../maverics.tar.gz
3

Modify the configuration

Edit maverics.json with the needed configuration changes.
# Edit the configuration
vi maverics.json
4

Copy the configuration file to the Orchestrator

# Copy the extracted config to the Orchestrator host
scp maverics.json orchestrator-host:/etc/maverics/maverics.json
5

Switch to local file configuration

Point the Orchestrator to the local config file instead of the cloud/Git provider. Remote config source environment variables (MAVERICS_AWS_CONFIG, MAVERICS_AZURE_CONFIG, MAVERICS_GCP_CONFIG, MAVERICS_GITHUB_CONFIG, MAVERICS_GITLAB_CONFIG) take priority over MAVERICS_CONFIG. You must unset the remote provider variable for your deployment before setting MAVERICS_CONFIG, otherwise the Orchestrator will ignore the local file.
# Unset the remote config source for your provider (pick the one matching your deployment)
# unset MAVERICS_AWS_CONFIG
# unset MAVERICS_AZURE_CONFIG
# unset MAVERICS_GCP_CONFIG
# unset MAVERICS_GITHUB_CONFIG
# unset MAVERICS_GITLAB_CONFIG

export MAVERICS_CONFIG=/etc/maverics/maverics.json
unset MAVERICS_BUNDLE_PUBLIC_KEY_FILE
sudo systemctl restart maverics
6

Verify the Orchestrator is running

The Orchestrator now loads configuration from the local file instead of polling the deployment provider.
sudo systemctl status maverics
This approach is functionally equivalent to Option 3 but starts from your existing bundle configuration rather than a new YAML file. The extracted maverics.json file uses JSON format, which the Orchestrator accepts alongside YAML.
When to use this option:
  • You need to make targeted changes to the existing configuration.
  • You want to preserve your current Console-managed configuration as a starting point.
Limitations:
  • Manual changes overwritten — When returning to Console-managed operations, re-publish from Console and switch the Orchestrator back to its deployment provider by unsetting MAVERICS_CONFIG and restoring MAVERICS_BUNDLE_PUBLIC_KEY_FILE.
  • Configuration is managed as a local file — No audit trail, revision history, or Console-managed features.

​
Option 3: Switch to Local File Configuration

Replace the bundle-based configuration with a local YAML file. This is the simplest break-glass path for making urgent configuration changes. Steps:
1

Prepare a local YAML configuration file

Create a configuration file for the Orchestrator. You can extract the maverics.json configuration from the current bundle (see Option 2 for extraction steps) and use it directly — the Orchestrator accepts both YAML and JSON formats.
# Example: point to a local config file
export MAVERICS_CONFIG=/etc/maverics/maverics.yaml
2

Disable bundle signature verification

Unset the bundle public key variable so the Orchestrator does not attempt to verify a bundle signature:
unset MAVERICS_BUNDLE_PUBLIC_KEY_FILE
3

Restart the Orchestrator

Restart the Orchestrator so it picks up the local configuration file instead of polling for bundles.
# systemd
sudo systemctl restart maverics

# Docker
docker restart maverics
When to use this option:
  • Urgent configuration changes are needed and you have a working YAML configuration.
  • You prefer a clean, simple approach over modifying bundles.
Limitations:
  • Manual configuration management — no Console-managed objects, no diff previews.
  • No audit trail for configuration changes made while working outside the Console.
  • No revision history or rollback capability.
  • You must switch back to bundle-based configuration when returning to Console-managed operations.

​
Option 4: Direct Provider Modification (Git-Based Providers)

For GitHub and GitLab deployment providers, you can extract the configuration from the bundle and use Git for version tracking during the emergency.
Pushing a modified tar.gz bundle to a Git repository has the same signature verification constraint described in Understanding the Constraint. You cannot load an unsigned tar.gz bundle by removing the signature file. Instead, follow these steps to extract the configuration and switch to a local file, using Git for version tracking.
Steps:
1

Clone the deployment repository

git clone https://github.com/your-org/your-deployment-repo.git
cd your-deployment-repo
2

Extract the configuration from the bundle

Extract maverics.json from the tar.gz bundle in the repository.
mkdir extracted && cd extracted
tar -xzf ../maverics.tar.gz
3

Modify the configuration

Edit maverics.json with the needed configuration changes.
vi maverics.json
4

Track changes in Git

Commit the extracted config file (not the tar.gz) for auditability.
cp maverics.json ../
cd ..
git add maverics.json
git commit -m "break-glass: emergency config change - [describe change]"
git push
5

Switch the Orchestrator to local file configuration

Copy maverics.json to the Orchestrator and point it to the local config file, following the same approach as Options 2 and 3. You must unset the Git provider variable first — remote config source variables take priority over MAVERICS_CONFIG.
scp maverics.json orchestrator-host:/etc/maverics/maverics.json

# Unset the Git provider config source (pick the one matching your deployment)
# unset MAVERICS_GITHUB_CONFIG
# unset MAVERICS_GITLAB_CONFIG

export MAVERICS_CONFIG=/etc/maverics/maverics.json
unset MAVERICS_BUNDLE_PUBLIC_KEY_FILE
sudo systemctl restart maverics
When to use this option:
  • Your deployment uses a GitHub or GitLab provider and you want Git history to provide auditability during the emergency.
Limitations:
  • The Orchestrator loads from a local file, not from the Git repository directly — Git provides change tracking only.
  • Same limitations as Options 2 and 3 — No Console-managed features during the emergency period.

​
Recovery Checklist

After returning to Console-managed operations, follow this checklist to re-establish normal workflows.
1

Verify Console connectivity

Confirm that the Console is accessible and your organization’s data is intact.
2

Re-publish from Console

Publish a new revision from the Console to overwrite any manual changes with a properly signed bundle. This ensures all Orchestrator instances are running Console-managed, signed configuration.
3

Re-enable signature verification

On every Orchestrator instance where verification was disabled, restore the MAVERICS_BUNDLE_PUBLIC_KEY_FILE environment variable to the public key path and restart the Orchestrator. Also unset MAVERICS_CONFIG if it was set during the emergency, so the Orchestrator returns to its deployment provider.
unset MAVERICS_CONFIG
export MAVERICS_BUNDLE_PUBLIC_KEY_FILE=/etc/maverics/public_key.pem
sudo systemctl restart maverics
4

Review audit logs

Check the Console audit logs for events that occurred while working outside the Console. There will be a gap in Console-side events during that period — document this gap.
5

Document manual changes

Record what configuration changes were made manually while working outside the Console, why they were needed, and which recovery option was used. This documentation supports compliance requirements.

​
Preparation: Maintaining Fallback Options

Prepare fallback options so you can respond quickly when Console access is unavailable.
  • Periodically download signed bundles — Download and archive signed bundles from the Console as backups. A valid signed bundle can be re-deployed without Console access (no signature verification bypass needed).
  • Document provider credentials separately — Store your deployment provider credentials (S3 access keys, Azure tokens, Git tokens) in a location accessible independently of the Console.
  • Record your public key path — Know where MAVERICS_BUNDLE_PUBLIC_KEY_FILE points and how to unset it on each Orchestrator instance.
  • Understand config source priority — Remote config source environment variables (MAVERICS_AWS_CONFIG, MAVERICS_AZURE_CONFIG, etc.) take priority over MAVERICS_CONFIG. When switching to local file mode, you must unset the active remote provider variable first, otherwise the Orchestrator will continue using the remote source.

Publishing Deployment Configs Overview

Bundle format, signing, deployment providers, and revision history

File Config Source

Load Orchestrator configuration from a local YAML file

Deploy to Production

Production deployment guide for the Orchestrator

Console Overview

Console administration overview and capabilities