Telemetry collectors

The Orchestrator emits telemetry using OpenTelemetry (OTLP). To setup your Orchestrator to emit telemetry, first see Telemetry Config.

This document lists examples of how to configure certain OTLP collectors to receive Orchestrator telemetry.

The Orchestrator emits telemetry in OTLP format.

Several vendors offer native OTLP collectors you can use to review Orchestrator telemetry.

Users may deploy a co-located OpenTelemetry collector to collect, aggregate and process telemetry from the Orchestrator.

Please review the local dashboard section to see how an example of running a observability stack locally to visualize the telemetry from an Orchestrator.

New Relic

New Relic is a cloud-based observability platform.

This guide shows the basic steps for connecting the Orchestrator to New Relic. See also the New Relic OpenTelemetry guide for additional detail.

Account setup

To setup New Relic to collect Orchestrator Telemetry, you’ll first need to setup an online account.

API key

To connect the Orchestrator to your New Relic account, first create an API Key. The Orchestrator sends this API Key as a header in the data transmitted to New Relic. New Relic verifies the API Key with your account, then allows the Orchestrator metrics data into your online datastore.

TLS certificates

New Relic requires TLS for HTTP security. You can reuse the certificate/key pair defined for the http server (normally named maverics), or set up TLS certificates specifically for the Telemetry service. See the Transport Security (TLS) section for instructions on setting up certificates and keys.

Orchestrator configuration

Once you have a New Relic account, your API Key, and the TLS certificates, you can configure the Orchestrator to output metrics to your New Relic account.

Edit your maverics.yaml Orchestrator configuration file to include a section similar to the following:

telemetry:
  metrics:
    enabled: true
    interval: 5m
    exporter: otlp
  exporters:
    otlp:
      protocols:
        http:
          tls: telemetryTLS # defined in the tls section 
          endpoint: otlp.nr-data.net:4318
          headers:
            api-key: <NewRelicAPIKey>

DataDog

DataDog is “the essential monitoring and security platform for cloud applications.”

This guide will show the basic steps to emit Orchestrator metrics to DataDog.

DataDog also has OpenTelemetry support documentation online. For more technical information about OpenTelemetry and the DataDog agent, refer to their documentation.

Account setup

To connect the Orchestrator to DataDog, you must first have a DataDog account. You can get a free account on the DataDog website by clicking on the “Get Started” button.

API key

To emit Orchestrator metrics to your DataDog account, the Orchestrator needs to know your DataDog application (API) key. To create a new application key, first log in to DataDog, then navigate to your organization application keys. Use a descriptive name for your key, such as “Maverics Orchestrater OTLP Machine 1” to remind you what the key is used for.

DataDog agent

Install the DataDog agent

To collect data from the Orchestrator, DataDog requires you to run their agent. You must install the DataDog agent for your OS onto the same system where the Orchestator is running. The DataDog agent installation instructions can be found on their documentation.

At the bottom of the installation page for your OS, you’ll see a “step-by-step” instruction link. Expand this link and find the location of the datadog.yaml file on your system; you’ll need to update this file in the next section.

Configure the DataDog agent

To enable the DataDog agent to collect OTLP metrics, you must edit your datadog.yaml agent configuration file, and restart the agent. (See the section above for finding the datadog.yaml file on your system.)

In datadog.yaml find the section named OpenTelemetry Configuration section and update it to include the following lines:

otlp_config:
  receiver:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

In the original datadog.yaml file, we recommend you simply add the above yaml lines, and leave the existing otlp_config section commented out, for future reference.

For more information on the DataDog OTLP, refer to their documentation. You can also reference the DataDog installation instructions above for more DataDog agent information.

Restart the DataDog agent

With the datadog.yaml file updated to enable OTLP metrics, you’ll need to restart the DataDog Agent. Follow the directions for restarting your agent from the Agent Installation section above.

For example, on a Fedora system, run sudo systemctl restart datadog-agent

Orchestrator configuration

Now that the DataDog agent is installed and running, configure your Orchestrator to emit metrics to the agent, listening on the endpoint you configured above.

Edit your maverics.yaml Orchestrator configuration file to include a section similar to the following:

telemetry:
  metrics:
    enabled: true
    interval: 5m
    exporter: otlp
  exporters:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318

Restart the Orchestrator to read your new telemetry configuration, then navigate to your DataDog account to see Orchestrator metrics.

Reading Orchestrator metrics in DataDog

You’ll need to setup either a dashboard or other monitor to read Orchestrator metrics. Refer to their dashboard documentation for more information.

To ensure DataDog is receiving Orchestrator metrics, navigate to the Metrics Explorer and search for metrics from service: io.strata.maverics.

Local dashboard

This configuration is useful for basic local testing and can be used as a starting point for configuring your own OTLP stack.

You can view Orchestrator metrics locally by using Docker Compose and some basic configuration for Prometheus, Grafana, and the OpenTelemetry Collector provided in a downloadable archive.

1. Download and extract the grafana_dashboard.zip archive.
2. From within the directory, run docker compose up.
3. Start the Orchestrator with telemetry configured. An example Orchestrator configuration file is included in the archive. The OpenTelemetry collector will be listening on http://localhost:4318.
4. Once all containers have started successfully, navigate to the Maverics System Dashboard in Grafana.