Telemetry (Local development)

The orchestrator gathers and emits state information (telemetry) using an open-source project known as OpenTelementry (OTLP). You can read more about OTLP online.

To collect and review orchestrator telemetry, we recommend using a vendor's OTLP collector (DataDog, NewRelic, etc.).

After configuring the orchestrator to emit metrics (below), you can setup a telemetry collector using our documentation.

Enabling telemetry

The Orchestator telemetry configuration uses OpenTelemetry's otelconf configuration which is a high-level configuration helper for setting up OpenTelemetry's SDKs.

A example can be found at the end of this document.

telemetry

The telemetry key is the main config starting point for telemetry.  When omitted, the orchestrator does not emit metrics.

metrics

The metrics key is required as child of telemetry. It contains all the telemetry metrics configuration and is a direct replacement of the meter_provider key defined in the otelconf.
This key serves as a wrapper config key around the actual otelconf configurations.

Full configuration of the otelconf configuration can be found here

Most public OTLP collector vendors require either the api-key or x-api-key header.

Complete example

Here is an example of a running orchestrator telemetry configuration:

telemetry:
  metrics:
    readers:
      - periodic:
          exporter:
            otlp:
              protocol: http/protobuf
              endpoint: https://localhost:4318/v1/metrics
              insecure: false
              timeout: 5000
              headers:
                - name: x-api-key
                  value: "some-api-key"
          interval: 5000
          timeout: 1000