Telemetry
The orchestrator gathers and emits state information (telemetry) using an open-source project known as OpenTelementry (OTLP). You can read more about OTLP online.
To collect and review orchestrator telemetry, we recommend using a vendor's OTLP collector (DataDog, NewRelic, etc.).
After configuring the orchestrator to emit metrics (below), you can setup a telemetry collector using our documentation.
Each section section of the orchestator telemetry configuration is explained below. A complete example can be found at the end of this document.
The
telemetry key is the main config starting point for telemetry. When omitted, the orchestrator does not emit metrics.The
metrics key is required below telemetry. It contains all of the telemetry metrics configuration, described as keys below.enabledTo emit metrics, add key
enabled: true below metrics. To disable metrics output, simply change the enabled key value to false.intervalOptionally configure the metrics reporting interval, add an
interval key followed by a time duration: interval: 15m The default interval is 5m.exporterThe required
exporter key references an exporter name from the exporters section below. Orchestrator metrics will be emitted to the specified exporter value. The name specified here must match one defined in exporters configuration below. See the example at the end of this document.The
health key is required below telemetry.enabledTo emit health, add key
enabled: true below health. To disable health output, simply change the enabled key value to false.exporterThe required
exporter key references an exporter name from the exporters section below. Orchestrator health will be emitted to the specified exporter value. The name specified here must match one defined in exporters configuration below. See the example at the end of this document.This required key describes where and how to export the Orchestrator metrics data. For more information about collectors where you can export telemetry data, see Collectors
The orchestrator currently outputs only to OTLP; the following section is required under
exporters: otlp:
protocols:
Exporter names must all start with
otlp and be unique, E.G. otlp-1 and otlp-2, or otlp-newrelic and otlp-datadog. See the example at the end of this document.The orchestrator exports OTLP telemetry using an HTTP (only) exporter. Under
http, add the endpoint key and specify where the orchestrator should emit metrics data: http:
endpoint: localhost:4318
By default, OTLP collectors listen to the URL path
/v1/metrics for metrics data. However, OTLP collectors can also be configured instead to listen at different URL paths. When using an OTLP collector listening at a different or custom URL path, you must update the exporter configuration with the URL path expected by the collector.To optionally specify a URL path for an exporter, use the
urlPath key in the http section: http:
endpoint: localhost:4318
urlPath: /customPath
This is an optional parameter, and should not be specified unless the OTLP collector is configured to listen at a non-default URL path.
headersOptionally add http headers which are emitted to the OTLP collector. Add
headers key, specifying the header name and value to send with the HTTP request: http:
headers:
api-key: auth-header-value
x-company-header: company-value
By default, no additional headers are output with metrics.
Most public OTLP collector vendors require either the
api-key or x-api-key header.Here is an example of a running orchestrator telemetry configuration:
tls:
telem-tls:
certFile: telemetry/localhost-client.pem
keyFile: telemetry/localhost-client-key.pem
caFile: telemetry/rootCA.pem
telemetry:
metrics:
enabled: true
interval: 5m
exporter: otlp/datadog
health:
enabled: true
exporter: otlp/local
exporters:
otlp/local:
protocols:
http:
endpoint: localhost:4318
insecure: true
otlp/datadog:
protocols:
http:
tls: telem-tls
endpoint: otlp.nr-data.net:4318
headers:
api-key: <Your company's NewRelicAPIKey>
Last modified 2mo ago