Observability

GoAkt provides built-in observability through metrics (system and per-actor), event streams, and dead letters. Metrics are available as programmatic snapshots and, when configured, as OpenTelemetry instruments for integration with dashboards and exporters.

Overview

System metrics: Node-level statistics via ActorSystem.Metric.
Per-actor metrics: Actor-level statistics via pid.Metric.
Event stream: System and cluster events (lifecycle, membership, dead letters).
OpenTelemetry: Metrics exported when WithMetrics is used.
Tracing: Zero-instrumentation actor traces via goakt-ebpf.

All metrics are local: they reflect the current node only. Metrics are point-in-time snapshots; values do not update after the struct is returned.

System metrics

ActorSystem.Metric(ctx) returns a *Metric struct with node-level statistics. Returns nil when the actor system is not started.

DeadlettersCount: Type: int64. Description: Total messages dropped to dead letters on this node.
ActorsCount: Type: int64. Description: Total number of live actors on this node.
Uptime: Type: int64. Description: Seconds since the actor system started.
MemoryUsed: Type: uint64. Description: Used memory in bytes (host).
MemorySize: Type: uint64. Description: Total memory in bytes (host).
MemoryAvailable: Type: uint64. Description: Available memory in bytes (host).

System metrics

m := sys.Metric(ctx)
if m != nil {
    log.Printf("actors=%d deadletters=%d uptime=%ds",
    m.ActorsCount(), m.DeadlettersCount(), m.Uptime())
}

Per-actor metrics

pid.Metric(ctx) returns an *ActorMetric struct with actor-level statistics. Returns nil when the actor is not running. For remote actors, metrics are fetched via remoting; the same fields are available.

ProcessedCount: Type: uint64. Description: Messages handed to the handler since the last restart (excludes PostStart; a message is counted when its handling starts, failed ones included).
ChidrenCount: Type: uint64. Description: Number of direct child actors.
RestartCount: Type: uint64. Description: Cumulative restarts, by supervision or by Restart/ReSpawn.
Uptime: Type: int64. Description: Seconds since the actor, or its latest incarnation, started.
LatestProcessedDuration: Type: time.Duration. Description: Time elapsed since the most recent message started processing (how long the actor has been idle).
StashSize: Type: uint64. Description: Current number of stashed messages.
DeadlettersCount: Type: uint64. Description: Messages addressed to this actor that went to dead letters, including those it rejected with Unhandled.
FailureCount: Type: uint64. Description: Cumulative failures the supervisor acted on (suspensions and resumed failures); survives restarts.
ReinstateCount: Type: uint64. Description: Cumulative reinstatements (suspended → resumed).
UnhandledCount: Type: uint64. Description: Cumulative messages the actor rejected with ReceiveContext.Unhandled.

Per-actor metrics

m := pid.Metric(ctx)
if m != nil {
    log.Printf("actors=%s processed=%d restarts=%d stash=%d",
    pid.Name(), m.ProcessedCount(), m.RestartCount(), m.StashSize())
}

Event stream

The event stream publishes system and cluster events (lifecycle, membership, dead letters). It is separate from application-level PubSub. See Event Streams for full documentation.

Event stream

subscriber, err := sys.Subscribe()
if err != nil {
    return err
}
defer sys.Unsubscribe(subscriber)

for msg := range subscriber.Iterator() {
    switch e := msg.Payload().(type) {
        case *actor.ActorStarted:
        // ...
        case *actor.Deadletter:
        // ...
    }
}

Event types: ActorStarted, ActorStopped, ActorChildCreated, ActorPassivated, ActorRestarted, ActorSuspended, ActorReinstated, Deadletter, NodeJoined, NodeLeft.

Dead letters

Messages sent to stopped or non-existent actors are captured by the dead-letter actor and published to the event stream. The Deadletter payload includes sender, receiver, message, and reason. Subscribe to the event stream to observe or log dead letters.

OpenTelemetry

When WithMetrics() is passed at actor system creation, the framework registers OpenTelemetry observable counters. You must initialize the OpenTelemetry SDK (MeterProvider, exporter) before starting the actor system; otherwise metrics are not exported.

System-level instruments

Attributes: actor.system.

actorsystem.deadletters.count: Type: Int64ObservableCounter. Description: Total dead letters on this node.
actorsystem.actors.count: Type: Int64ObservableGauge. Description: Live actors on this node.
actorsystem.grains.count: Type: Int64ObservableGauge. Description: Grains activated on this node.
actorsystem.uptime: Type: Int64ObservableGauge (s). Description: Actor system uptime in seconds.
actorsystem.peers.count: Type: Int64ObservableGauge. Description: Connected peers (cluster mode).

Scheduler instruments

Attributes: actor.system.

scheduler.scheduled.count: Type: Int64ObservableCounter. Description: Messages successfully scheduled for later delivery.
scheduler.cancelled.count: Type: Int64ObservableCounter. Description: Scheduled messages successfully cancelled.

Both counters move on the scheduling API calls only: a call that fails, such as a cancellation for an unknown reference, does not count.

Cluster membership instruments

Attributes: actor.system. Registered only when the actor system runs in cluster mode.

cluster.members.joined.count: Type: Int64ObservableCounter. Description: Members this node has seen join the cluster.
cluster.members.left.count: Type: Int64ObservableCounter. Description: Members this node has seen leave the cluster.

Each node reports the churn it observed, so a three node cluster reports a single join on three series. Rate either counter to alert on membership instability; actorsystem.peers.count gives the current membership the churn resolves to.

Actor lifecycle instruments

Attributes: actor.system, actor.kind. These three totals count events that remove an actor from the tree, so they are reported by the actor system rather than by a PID, one series per actor kind. System actors (guardians, the dead-letter actor, topic actors) are excluded.

actor.spawned.count: Type: Int64ObservableCounter. Description: Actors of this kind whose spawn completed.
actor.stopped.count: Type: Int64ObservableCounter. Description: Actors of this kind that were shut down.
actor.passivated.count: Type: Int64ObservableCounter. Description: Actors of this kind that passivated.

actor.stopped.count counts terminal stops only. The teardown a restart performs before re-initializing an actor is reported by actor.restart.count, and a passivation is reported by actor.passivated.count, so no lifecycle event is counted twice.

Per-actor instruments

Attributes: actor.system, actor.name, actor.kind, actor.address by default, or actor.system and actor.kind under WithLowCardinalityMetrics. Per-actor metrics are collected by a single callback that walks the actor tree on every scrape, so the number of meter registrations stays constant regardless of the actor population. An actor appears in scrapes once it is running and has processed its PostStart message; suspended, stopping, and passivating actors are omitted until they are running again.

actor.children.count: Type: Int64ObservableGauge. Description: Current child actors.
actor.stash.size: Type: Int64ObservableGauge. Description: Messages stashed.
actor.deadletters.count: Type: Int64ObservableCounter. Description: Dead letters addressed to this actor, per message type.
actor.restart.count: Type: Int64ObservableCounter. Description: Restarts.
actor.last.received.duration: Type: Int64ObservableGauge (ms). Description: Time since the last message started processing.
actor.processed.count: Type: Int64ObservableCounter. Description: Messages processed.
actor.uptime: Type: Int64ObservableGauge (s). Description: Actor uptime in seconds.
actor.failure.count: Type: Int64ObservableCounter. Description: Failures.
actor.reinstate.count: Type: Int64ObservableCounter. Description: Reinstatements.
actor.unhandled.count: Type: Int64ObservableCounter. Description: Messages rejected with Unhandled.
actor.mailbox.size: Type: Int64ObservableGauge. Description: Messages waiting in the mailbox.

actor.deadletters.count carries one extra attribute, message.type, naming the type of the dropped message, and is reported once per message type the actor dropped. An actor that has dropped nothing reports a single zero without the attribute. To get an actor’s total across all types, sum the series:

Per-actor instruments

sum by (actor_name) (actor_deadletters_count)

actor.mailbox.size is the one instrument reported outside the gate described above. A suspended or restarting actor still emits it, and only it, because a backlog piling up behind an actor that has stopped consuming is exactly the condition the metric exists to expose. It counts user messages that are enqueued and not yet dispatched, so it excludes the message being processed, the stashed messages reported by actor.stash.size, and the internal control-plane queue.

Average time a message waits in the mailbox

GoAkt records no mailbox wait histogram: measuring one would mean timestamping every message on the message path. The average wait follows from Little’s law instead, dividing the queue length by the rate at which the actor drains it:

Average time a message waits in the mailbox

actor_mailbox_size / rate(actor_processed_count[5m])

The result is in seconds. A value that climbs while the processed rate stays flat means work is arriving faster than the actor can handle it.

Reporting one series per actor kind

By default each instrument above produces one time series per live actor. That is precise, but the series count grows with the actor population, which becomes expensive to store and query once a system runs thousands of actors. WithLowCardinalityMetrics reports the same instruments once per actor kind instead. The scrape aggregates the live actors of each kind and observes each instrument once, carrying only actor.system and actor.kind:

Reporting one series per actor kind

sys, _ := actor.NewActorSystem("app", actor.WithMetrics(actor.WithLowCardinalityMetrics()))

The series count then depends on how many actor kinds the system declares, not on how many actors it runs. System actors (guardians, the dead-letter actor, topic actors) are excluded from the per-kind series; they only appear in the default per-actor mode.

The per-actor attributes actor.name and actor.address are dropped in this mode, so an individual actor can no longer be singled out in a dashboard or an alert. Queries and alerts written against the default mode must be rewritten to group by actor_kind.

Each instrument is reduced across the live actors of the kind:

actor.children.count: Reduction: sum. Meaning per kind: Children held by actors of this kind.
actor.stash.size: Reduction: sum. Meaning per kind: Stashed messages across the kind.
actor.restart.count: Reduction: sum. Meaning per kind: Restarts across the kind.
actor.processed.count: Reduction: sum. Meaning per kind: Messages processed by the kind.
actor.failure.count: Reduction: sum. Meaning per kind: Failures across the kind.
actor.reinstate.count: Reduction: sum. Meaning per kind: Reinstatements across the kind.
actor.unhandled.count: Reduction: sum. Meaning per kind: Unhandled messages across the kind.
actor.deadletters.count: Reduction: sum per kind and message type. Meaning per kind: Drops by the kind, still broken down by type.
actor.uptime: Reduction: max. Meaning per kind: Age of the oldest live actor of the kind.
actor.last.received.duration: Reduction: min. Meaning per kind: Time since the kind last processed a message.
actor.mailbox.size: Reduction: sum. Meaning per kind: Messages waiting across the kind.

The two time gauges cannot be summed. The maximum uptime answers how long the kind has been running, and the minimum time since the last received message answers whether the kind is still active.

The gate is unchanged: only running actors that have processed their PostStart contribute, so a kind whose actors are all suspended drops out of the series until one resumes. actor.mailbox.size is the exception it is in the default mode too: every actor of the kind adds its backlog to the sum whatever its lifecycle state, so a kind whose actors are all suspended keeps reporting that one instrument while the others go quiet. A kind that has dropped nothing reports a single zero for actor.deadletters.count without the message.type attribute, exactly as an actor does in the default mode. The system-level instruments and the actor lifecycle instruments already report one series per system and per kind, so they are identical in both modes.

Example

import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/sdk/metric"
)

// Initialize SDK before creating actor system
exporter, _ := otlpmetrichttp.New(context.Background(), otlpmetrichttp.WithInsecure())
provider := metric.NewMeterProvider(metric.WithReader(metric.NewPeriodicReader(exporter)))
otel.SetMeterProvider(provider)

sys, _ := actor.NewActorSystem("app", actor.WithMetrics())

Tracing

For zero-instrumentation distributed tracing of actor message flows, use goakt-ebpf. It runs as a sidecar process, attaches to your GoAkt v4 application via eBPF uprobes, and exports spans to any OTLP-compatible backend (Jaeger, Grafana Tempo, Honeycomb, etc.): no code changes or SDK dependency in your app.

Quickstart

Actors

Grains

AI Agents

Remoting

Clustering

Advanced

Architecture

Observability

Overview

System metrics

Per-actor metrics

Event stream

Dead letters

OpenTelemetry

System-level instruments

Scheduler instruments

Cluster membership instruments

Actor lifecycle instruments

Per-actor instruments

Average time a message waits in the mailbox

Reporting one series per actor kind

Example

Tracing

​Overview

​System metrics

​Per-actor metrics

​Event stream

​Dead letters

​OpenTelemetry

​System-level instruments

​Scheduler instruments

​Cluster membership instruments

​Actor lifecycle instruments

​Per-actor instruments

​Average time a message waits in the mailbox

​Reporting one series per actor kind

​Example

​Tracing

Overview

System metrics

Per-actor metrics

Event stream

Dead letters

OpenTelemetry

System-level instruments

Scheduler instruments

Cluster membership instruments

Actor lifecycle instruments

Per-actor instruments

Average time a message waits in the mailbox

Reporting one series per actor kind

Example

Tracing