Remoting

Overview

Remoting lets actors on different nodes exchange messages over TCP. Enable it with actor.WithRemote and a remote.Config. From inside the actor system, use ActorOf and the usual Tell / Ask APIs; the framework routes remote messages automatically. For callers outside an actor system (CLI, API servers, batch jobs), use the Client package. Current peers speak a multiplexed duplex protocol: persistent lane connections per peer, correlation-driven asks, chunked large messages, and credit-based flow control. A legacy unary protobuf-over-TCP path remains for mixed-version rollouts. Delivery at the transport is at-most-once; use reliable delivery when you need confirmed delivery.

Quick example

cfg := remote.NewConfig("0.0.0.0", 8080)
sys, err := actor.NewActorSystem("my-system", actor.WithRemote(cfg))
if err != nil {
    panic(err)
}
if err := sys.Start(ctx); err != nil {
    panic(err)
}
defer sys.Stop(ctx)

Configuration

Create a remote.Config with remote.NewConfig(bindAddr, bindPort, opts...). The bind address must be a concrete IP (127.0.0.1, 0.0.0.0, ::, …), not a hostname. Pass the config via actor.WithRemote(cfg). With a wildcard bind address (0.0.0.0 or ::) the remoting server listens on every interface, and the node is advertised to peers, through ActorSystem.Host(), actor addresses and cluster membership, as the first private IP of the host, or its first public IP when it has no private one, or loopback when it has neither, so an offline machine still starts. :: accepts both IPv4 and IPv6 peers. Cluster gossip and the registry bind the advertised address rather than the wildcard. To listen on one interface only, and to choose the advertised address, bind to that interface’s IP.

WithProtocolPin: Wire protocol on dial and accept: auto, legacy, or duplex. Default: auto.
WithOrdinaryLanes: Ordinary duplex lanes per peer for user tell/ask. Default: 1.
WithLargeMessageDestinations: Actor-path patterns that prefer the large lane. Default: none.
WithChunkSize: Logical size above which senders emit CHUNK frames (16 KiB–4 MiB). Default: 256 KiB.
WithMaxMessageSize: Cap on a reassembled logical frame (≥ max frame, ≤ 4 GiB−1; negotiated). Default: 16 MiB.
WithMaxFrameSize: Max whole-frame size (16 KiB–16 MiB; HELLO pairwise minimum). Default: 16 MiB.
WithMaxConcurrentLargeTransfers: Concurrent CHUNK reassembly groups per connection (negotiated). Default: 4.
WithCreditWindow: Unreclaimed in-flight byte budget per duplex connection (≥ chunk size). Default: 16 MiB.
WithWriteTimeout: Duplex outbound admission bound when the caller has no deadline. Default: 10s.
WithReadIdleTimeout: Duplex PING interval; two missed PONGs close the connection (0 = off). Default: 10s.
WithDialTimeout: How long a dial to a peer may take before it fails (must be > 0). Default: 5s.
WithCompression: None, Gzip, Zstd, or Brotli. Default: None.
WithSerializers: Bind a Serializer to a message type or interface. Default: protobuf.
WithSerializables: Bulk-register types with the built-in CBOR serializer. Default: none.
WithJSONSerializables: Bulk-register types with the built-in JSON serializer. Default: none.
WithContextPropagator: Propagate trace IDs, auth tokens, correlation IDs. Default: none.
WithTLS: TLS for remoting (and cluster transports when clustering). See TLS. Default: off.

Message serialization is covered on the Serialization page.

Protocol pin

WithProtocolPin selects which remoting wire protocol this node dials and accepts. It applies to both the remoting listener and the outbound client.

auto: Constant: remote.ProtocolPinAuto. Behavior: Default. Accept: sniff the first byte after TLS (0x02 → duplex, otherwise legacy). Dial: prefer duplex, fall back to legacy on EOF or connection reset before HELLO_ACK; a peer classified legacy is re-probed for duplex after 30 seconds.
legacy: Constant: remote.ProtocolPinLegacy. Behavior: Unary protobuf-over-TCP only. No duplex sniff, dial, or fallback.
duplex: Constant: remote.ProtocolPinDuplex. Behavior: Multiplexed duplex only. Non-duplex first bytes are refused; no legacy fallback.

Use auto for mixed-version rolling upgrades. Pin legacy or duplex for homogeneous clusters, deterministic rollout, or when first-byte discrimination is unsafe: legacy brotli has no magic byte and can collide with the duplex 0x02 discriminator under auto.

Protocol pin

cfg := remote.NewConfig("0.0.0.0", 8080,
    remote.WithProtocolPin(remote.ProtocolPinDuplex),
)

Lanes and ordering

Per peer, duplex remoting opens a small set of persistent connections:

Control: System traffic (watch/unwatch, spawn/stop, cluster RPCs, liveness) so user floods cannot delay death-watch.
Ordinary: User tells and asks; hashed by the receiver’s full canonical address across WithOrdinaryLanes.
Large: Destinations listed in WithLargeMessageDestinations.

Ordering is per sender–target pair FIFO. With the default single ordinary lane, that matches today’s effective ordering except that control may overtake user traffic (deliberate) and large destinations ride their own lane (opt-in). Raising WithOrdinaryLanes (1–254) narrows the FIFO domain in exchange for parallel sends.

Large messages

Logical frames larger than WithChunkSize are split into CHUNK frames and reassembled by correlation ID, up to WithMaxMessageSize. Concurrent reassembly groups per connection are capped by WithMaxConcurrentLargeTransfers. Oversize and cap violations return in-band errors; the connection stays usable when the peer supports chunking (capability revision ≥ 2). WithChunkSize is local and not negotiated (it is clamped down to the negotiated max frame size when larger); peers with different chunk sizes still interoperate. WithMaxFrameSize, WithMaxMessageSize, and WithMaxConcurrentLargeTransfers are HELLO-advertised and take the pairwise minimum, so raising only one side has no effect if the peer advertises less. WithLargeMessageDestinations lists hierarchical actor-path glob patterns that route matching user traffic onto the large lane. Patterns match the path after host:port, not the full goakt:// URI. The path lists every ancestor of the actor (for example orders/cart/item for the actor item under cart under orders), and each * matches exactly one segment: orders/* matches the direct children of orders, and orders/*/* their children. It is a performance and isolation knob, not a correctness gate: an oversized message to an unlisted destination still chunks in place on its ordinary lane and stays in order with surrounding small messages to that actor.

Large messages

cfg := remote.NewConfig("0.0.0.0", 8080,
    remote.WithLargeMessageDestinations("bulk/*", "archive/exporter"),
    remote.WithChunkSize(256*1024),
    remote.WithMaxMessageSize(64*1024*1024),
)

Flow control and send semantics

WithCreditWindow is advertised as HELLO initial_credits and negotiated as the pairwise minimum. It sizes the local outbound admission queue on every duplex session. End-to-end send-window enforcement (writer park / CREDIT grants) applies only when both peers negotiate capability revision ≥ 4; older peers keep an unlimited send window but still honor the admission cap. The value must be greater than zero and at least WithChunkSize. On the duplex path:

Enqueue success returns nil (fire-and-forget does not wait for the peer).
A full outbound admission queue blocks until the caller’s deadline, or WithWriteTimeout when none is set, then returns errors.ErrRemoteSendBackpressure.
Transport failures dead-letter with an event; messages are not silently dropped for flow control.
Slow receivers slow senders: DATA/CHUNK writes spend credit; the receiver returns credit when a tell leaves the mailbox for dispatch, at ask worker-pool handoff, or on CHUNK reassembly append. Because a tell’s credit is returned only as the message leaves the mailbox, flow control tracks mailbox residency: a stalled consumer stops returning credit and its senders park instead of the receiver absorbing at wire speed. When credit is exhausted the writer parks.

Asks are multiplexed by correlation ID on the same connection: the socket read loop stays free while asks run on a worker pool, so one slow actor does not block the connection. The pool has no upper bound on workers: each ask to a stalled actor holds a goroutine until the ask’s deadline, so bound the timeout of asks toward actors that can stall. When a node stops, its remoting server also ends the duplex connections its peers opened. From the start of the stop it admits no new ask: a late ask is refused with ErrRemotingDisabled and does not run. Asks already running finish and reply within the shutdown timeout, then the connections are closed, so a peer sees the node gone (a closed connection, then a refused dial) instead of answers from a stopped node.

The credit window is shared by every receiver on a duplex connection. Because a tell’s credit is not returned until the message leaves the mailbox, a single stalled consumer can hold the window and slow senders to other actors on the same peer that hash to the same ordinary lane. Two mitigations:

Give an actor you expect to be slow a bounded mailbox. When it fills, the receiver refuses the remote message and returns the credit, which frees the shared window for siblings and backpressures that actor’s own sender. This holds for every bounded mailbox, BoundedMailbox included: its blocking applies to local senders only, never to the connection’s inbound dispatcher.
Raise WithOrdinaryLanes so receivers shard across more connections, each with its own window, making a collision between a slow and a fast receiver less likely.

Guaranteed per-receiver isolation (a stalled receiver backpressuring only its own sender) is planned as a follow-up.

Compression

WithCompression selects None (default), Gzip, Zstd, or Brotli. On the duplex path HELLO requires an exact match of the configured codec on both sides; otherwise the session falls back to None. On the legacy path both sides must be configured alike; a mismatch produces garbage. Prefer WithProtocolPin away from auto if you still use brotli on legacy peers (see Protocol pin).

Deadlines and liveness

WithWriteTimeout (default 10s) bounds duplex outbound queue admission when the caller’s context has no deadline, and bounds duplex socket writes.
WithReadIdleTimeout (default 10s) arms duplex PING/PONG liveness probes. After two missed PONGs the connection is closed. Set to 0 to disable probes. PINGs also refresh the server’s connection idle deadline, so a healthy idle lane is not reclaimed while liveness is succeeding.
WithDialTimeout (default 5s) bounds the connection of the remoting client to a peer, on the duplex and the legacy path alike. A dial to an address that silently drops packets, such as the IP of a departed pod, waits out the whole timeout. Keep it below the timeout of the calls that may have to dial, so the dial fails on its own timeout instead of the call failing on its deadline.
The config idle timeout (default 20 minutes; no public option) is the legacy unary read/write deadline and the duplex connection reclaim window. When both idle timeout and WithReadIdleTimeout are set, read-idle must be strictly less than idle timeout.

WithWriteTimeout and WithReadIdleTimeout apply to the duplex path. The legacy unary path uses the idle timeout alone.

Transport

Dual-protocol listener: duplex and legacy on the same remoting port
Duplex: persistent control / ordinary / large lanes, credit window, chunking
Legacy: length-prefixed unary frames and the send coalescer (legacy peers only)
Optional TLS (see TLS)
Compression as above

Using remoting

Once remoting is configured, ActorOf(ctx, name) returns a *PID whether the actor is local or remote. Use Tell and Ask as usual; no application code changes are required for remote vs local. Batch APIs and control RPCs use the same remoting stack.

TLS

TLS is configured on the remote config with remote.WithTLS. It takes a tls.Info from the GoAkt tls package, which carries a standard crypto/tls configuration for each side of a connection:

TLS

type Info struct {
    ClientConfig *tls.Config // used for outbound connections
    ServerConfig *tls.Config // used by listeners accepting connections
}

For an actor system both fields must be set; NewActorSystem fails with ErrInvalidTLSConfiguration when either is nil. Every node dials other nodes and accepts connections from them, so each node is always both a client and a server. The TLS settings are validated and applied only when remoting is enabled; without remoting they are ignored.

The actor system option actor.WithTLS is deprecated in favor of remote.WithTLS. It keeps working for existing systems, and when both are set the remote config settings take precedence.

What gets encrypted

One remote.WithTLS call covers every transport the node opens:

Remoting server and all outbound remoting clients: Port: remoting port. Mode: always.
Cluster engine traffic (registry reads/writes, replication): Port: peers port. Mode: cluster mode.
Membership gossip (memberlist transport): Port: discovery port. Mode: cluster mode.

There is no per-channel opt-out: a node either runs fully encrypted or fully in plaintext.

Enabling TLS

package main

import (
    "crypto/tls"
    "crypto/x509"
    "os"

    "github.com/tochemey/goakt/v4/actor"
    "github.com/tochemey/goakt/v4/remote"
    gtls "github.com/tochemey/goakt/v4/tls"
)

func main() {
    caPEM, err := os.ReadFile("ca.pem")
    if err != nil {
        panic(err)
    }

    caPool := x509.NewCertPool()
    caPool.AppendCertsFromPEM(caPEM)

    cert, err := tls.LoadX509KeyPair("node.pem", "node-key.pem")
    if err != nil {
        panic(err)
    }

    info := &gtls.Info{
        ServerConfig: &tls.Config{
            Certificates: []tls.Certificate{cert},
            ClientCAs:    caPool,
            ClientAuth:   tls.RequireAndVerifyClientCert, // enforce mTLS
        },
        ClientConfig: &tls.Config{
            Certificates: []tls.Certificate{cert},
            RootCAs:      caPool,
        },
    }

    remoteConfig := remote.NewConfig("0.0.0.0", 8080, remote.WithTLS(info))

    system, err := actor.NewActorSystem("secure-system",
        actor.WithRemote(remoteConfig),
    )
    if err != nil {
        panic(err)
    }

    _ = system
}

Mutual TLS

GoAkt passes your tls.Config values through unchanged, so mutual TLS is a matter of standard crypto/tls settings, as in the example above: the server sets ClientCAs and ClientAuth: tls.RequireAndVerifyClientCert, and the client presents its own certificate in Certificates. Without those, you get server-only TLS: traffic is encrypted and clients authenticate the server, but any client can connect.

Cluster requirements

Same root CA everywhere. All nodes must trust the same CA, and every certificate must chain to it, otherwise handshakes between nodes fail.
All nodes or none. TLS and plaintext nodes cannot talk to each other. Enabling TLS on a running plaintext cluster (or the reverse) is a full-restart change, not a rolling one.
Certificate names must match how nodes address each other. Nodes dial each other by the advertised host, so certificates need that host in their SANs, or the client config must relax verification.

Discovery providers (Consul, etcd, Kubernetes, NATS) talk to their own backends and secure those connections through their own provider-specific configuration; remote.WithTLS does not apply to them. The standalone Client package picks up the TLS settings from the remote.Config passed to its nodes.

Context propagation

When messages cross node boundaries, request-scoped metadata (trace IDs, auth tokens, correlation IDs) must travel with them. Implement remote.ContextPropagator and pass it via WithContextPropagator on the remote config.

The ContextPropagator interface

package remote

type ContextPropagator interface {
    Inject(ctx context.Context, headers http.Header) error
    Extract(ctx context.Context, headers http.Header) (context.Context, error)
}

Inject: Write context values into headers for outgoing requests. Called before sending a remote message.
Extract: Read headers from an incoming request and return a context with propagated values. Called when receiving a remote message.

The carrier is http.Header as a string-keyed map; the transport is TCP, not HTTP. The receiving actor’s ReceiveContext.Context() or GrainContext.Context() will contain the propagated values.

Implementation notes

Be stateless and safe for concurrent use
Use stable, well-known header keys
Avoid leaking sensitive data unless required
Validate inputs to guard against injection or oversized metadata

Quickstart

Actors

Grains

AI Agents

Clustering

Advanced

Architecture

Remoting

Overview

Quick example

Configuration

Protocol pin

Lanes and ordering

Large messages

Flow control and send semantics

Compression

Deadlines and liveness

Transport

Using remoting

TLS

What gets encrypted

Enabling TLS

Mutual TLS

Cluster requirements

Context propagation

The ContextPropagator interface

Implementation notes

​Overview

​Quick example

​Configuration

​Protocol pin

​Lanes and ordering

​Large messages

​Flow control and send semantics

​Compression

​Deadlines and liveness

​Transport

​Using remoting

​TLS

​What gets encrypted

​Enabling TLS

​Mutual TLS

​Cluster requirements

​Context propagation

​The ContextPropagator interface

​Implementation notes

Overview

Quick example

Configuration

Protocol pin

Lanes and ordering

Large messages

Flow control and send semantics

Compression

Deadlines and liveness

Transport

Using remoting

TLS

What gets encrypted

Enabling TLS

Mutual TLS

Cluster requirements

Context propagation

The ContextPropagator interface

Implementation notes