Maverics observability

The Orchestrator emits telemetry data from the running process by using OpenTelemetry.

To configure your Maverics instance to emit telemetry, see Telemetry.

Metrics

Metrics are numerically-based observations (counters, totals, etc.)

ℹ️

All metrics emit both their value, and a list of shared resource attributes, described in the Shared resource attributes section.

Orchestrator metrics

The following custom metrics are emitted by the orchestrator:

Active Requests

Active Requests (simultaneous) is described here

Name	Description	Type
`"http.server.active_requests"`	Measures the number of concurrent HTTP requests that are currently in-flight.	`UpDownCounter (Non-monotonic)`

Attributes:

http.request.method HTTP request method (string)
server.address Name of the local HTTP server that received the request (string)
server.port local HTTP server port (int)
url.scheme http or https (string)

For example:

{
    __name__="http_server_active_requests",
    ..., 
    service_name="io.strata.maverics", 
    service_version="0.18.14", 
    ...,
    http_request_method="GET",
    server_address="maverics.sonarsystems.com", 
    server_port="7474",
    url_scheme="http",
...
}

Server Latency (request duration)

Server Latency (aka request duration) is described here

Name	Description	Type
`"http.server.duration"`	Measures the duration of all HTTP requests, as a histogram.	`Histogram`

Attributes:

http.request.method HTTP request method (string)
server.address Name of the local HTTP server that received the request (string)
server.port local HTTP server port (int)
url.scheme http or https (string)

For example:

{
    __name__="http_server_duration",
    ..., 
    service_name="io.strata.maverics", 
    service_version="0.18.14", 
    ...,
    http_request_method="GET",
    server_address="maverics.sonarsystems.com", 
    server_port="7474",
    url_scheme="http",
...
}

Request Count

Request count is a running total of all requests received by the Orchestrator.

Name	Description	Type
`"http.server.request_count"`	Measures the total running count of all Orchestrator requests.	`Int64Counter (Monotonic)`

Attributes:

status_code HTTP response status (int)

For example:

{
    __name__="http_server_request_count",
    ..., 
    service_name="io.strata.maverics", 
    service_version="0.18.14", 
    ...,
    http_request_method="GET",
    server_address="maverics.sonarsystems.com", 
    server_port="7474",
    url_scheme="http",
...
}

Session Count

The Orchestrator Session Count metric displays the number of known sessions.

Name	Description	Type
`"io.strata.maverics.sessions"`	Measures the number of known sessions.	`UpDownCounter (Non-monotonic)`

For example:

{
    __name__="io_strata_maverics_sessions", 
    ..., 
    service_name="io.strata.maverics", 
    service_version="0.18.14", 
...
}

ℹ️

Sessions are removed from this count only when a user with an expired session attempts to navigate to a protected URL.

OTLP metrics

The Orchestrator also uses built-in metrics that are defined within OTLP.

To review the OTLP documentation on the semantics of these metrics, please see the OpenTelemetry repo.

For more complete information on each of these metrics, please see the OTLP system metrics documentation.

Runtime metrics

Name	Type	Description
runtime.uptime	(ms)	Milliseconds since application was initialized
process.runtime.go.cgo.calls	-	Number of cgo calls made by the current process
process.runtime.go.gc.count	-	Number of completed garbage collection cycles
process.runtime.go.gc.pause_ns	(ns)	Amount of nanoseconds in GC stop-the-world pauses
process.runtime.go.gc.pause_total_ns	(ns)	Cumulative nanoseconds in GC stop-the-world pauses since the program started
process.runtime.go.goroutines	-	Number of goroutines that currently exist
process.runtime.go.lookups	-	Number of pointer lookups performed by the runtime
process.runtime.go.mem.heap_alloc	(bytes)	Bytes of allocated heap objects
process.runtime.go.mem.heap_idle	(bytes)	Bytes in idle (unused) spans
process.runtime.go.mem.heap_inuse	(bytes)	Bytes in in-use spans
process.runtime.go.mem.heap_objects	-	Number of allocated heap objects
process.runtime.go.mem.heap_released	(bytes)	Bytes of idle spans whose physical memory has been returned to the OS
process.runtime.go.mem.heap_sys	(bytes)	Bytes of heap memory obtained from the OS
process.runtime.go.mem.live_objects	-	Number of live objects is the number of cumulative Mallocs - Frees

ℹ️

Monitoring the metric runtime.uptime is the best way to ensure your Orchestrator is up and running. The value should continually increment, while the Orchestrator is up and functioning.

Host metrics

ℹ️

These metrics are only emitted on Linux and Windows systems.

Name	Value
process.cpu.time	state=user or system
system.cpu.time	state=user or system or other or idle
system.memory.usage	state=used or available
system.memory.utilization	state=used or available
system.network.io	direction=transmit or receive

HTTP Metrics

The following HTTP Metrics are collected and emitted by the Orchestrator:

Name	Type	Description
http.server.request_count	-	Number of requests observed by the Orchestrator
http.server.duration	(ms)	Orchestrator latency as a Histogram

Shared resource attributes

For every metric above, the Orchestrator also adds the following shared resource attributes:

Name	Value
`service_name`	`"io.strata.maverics"` (always)
`service_version`	Orchstrator Version (SemVer E.G. `0.9.16`)
`service_instance_id`	UUID of each running Orchstrator Instance. Reset on startup.

`os_type`	Type of OS running Orchestrator (One of `linux`, `windows`, or `darwin`)
`os_description`	Longer description of OS Type. May include OS version and build information.

`host_name`	Name of host running Orchestrator

`process_executable_name`	Name of command-line executable used to start the Orchestrator
`process_executable_path`	Full pathname of the running Orchestrator executable
`process_command_args`	Command-line arguments used to start the Orchestrator
`process_owner`	OS User that started the Orchestrator
`process_pid`	OS Process ID of the Orchestrator
`process_runtime_name`	`"go"` (always)
`process_runtime_version`	Version of GoLang used to compile the Orchestrator (E.G. “go1.19”)
`process_runtime_description`	Detailed information about the Go version running the Orchestrator

For example, a complete runtime.uptime metric with resource attributes may look like the following:

runtime_uptime{
    container_id="45d64ce11347", 
    exported_job="io.strata.maverics", 
    host_name="localhost.localdomain", 
    instance="otelcol:9464", 
    job="otel", 
    os_description="Fedora Linux 35 (Workstation Edition) (Linux localhost.localdomain 5.17.7-200.fc35.x86_64 #1 SMP PREEMPT Thu May 12 14:56:48 UTC 2022 x86_64)", 
    os_type="linux", 
    process_command_args="["bin/maverics_linux_amd64","-v","-config","../maverics.yaml"]", 
    process_executable_name="maverics_linux_amd64", 
    process_executable_path="/etc/bin/maverics_linux_amd64", 
    process_owner="user1", 
    process_pid="12345", 
    process_runtime_description="go version go1.19 linux/amd64",
    process_runtime_name="go", 
    process_runtime_version="go1.19", 
    service_instance_id="e83daeb0-0dee-4d80-a3f6-5f9d4cb9ff3", 
    service_name="io.strata.maverics", 
    service_version="0.20.4"
}, 123456

Telemetry Orchestrator health