Use cases
Reach for point-to-point reliable delivery when one actor must hand work to another and the default at-most-once semantics are not enough:- Message loss is unacceptable. Two specific actors need at-least-once or effectively-once processing instead of best-effort Tell. A checkout actor handing accepted orders to a fulfillment processor is the running example on this page: every order must arrive, in order, exactly once in effect, across restarts, redeploys, and node loss.
- Flow-controlled producer-consumer pipelines. A fast producer risks overwhelming a slower consumer or saturating
the network. The producer sends only when its controller grants demand through
RequestNext, so the consumer-side buffer stays bounded no matter how fast the producer runs and mailboxes never grow without limit. - Ordered, sequential processing. Deliveries arrive in sequence order, and the next message reaches the consumer only after the current one is confirmed; the consumer controller buffers anything that arrives in the meantime. A coordinator streaming jobs to a worker knows each job was processed, not merely enqueued, before the next one lands.
- Cross-node event forwarding. An actor on one cluster node feeds an actor on another, and the receiver must see every event in order even though the network hop is lossy.
- Crash-resilient handoff. With a durable queue on the producer side, messages that were stored but not yet confirmed are redelivered after a producer crash, restart, or relocation. Combined with an idempotent consumer, this gives outbox-style, crash-surviving handoff between two parts of your system without operating an external broker.
Ask),
and for best-effort telemetry where plain Tell is enough.
How it works
One flow connects one producer to one consumer. The actor system spawns an internal controller next to each endpoint; your actors talk only to their own controller, and the controllers run the protocol between themselves: The producer controller sends only when the consumer controller has granted demand, sequences every message, and resends until the consumer confirms. Registration, session management, resends, and deduplication after restarts are entirely the controllers’ job. The full path of one message:Guarantees
- Neither side crashes, controllers connected: Effectively-once, in order, within the consumer-driven flow-control window.
- Message loss or consumer-side restart: At-least-once; the consumer handler must be idempotent.
- Producer-side restart without a durable queue: Messages already handed to the controller but unconfirmed are lost; the flow resumes cleanly.
- Producer-side restart with a durable queue: Stored messages reload and redeliver; unacknowledged submissions are resubmitted.
Enabling a flow
Each side names its peer’s actor name. There are no controller names or handles anywhere in your code.Enabling a flow
The controllers carry internal, activation-scoped identities under the reserved
GoAkt prefix. They are invisible to
Actors, ActorOf, Kill, ReSpawn, and every other actor-management API, and they impose no naming convention on
your own actors.The producer contract
The producer buffers incoming work and answers three protocol messages:RequestNextgrants one tokenized send permission. Hold the grant until there is something to send; it never expires.- Answer a grant with exactly one
Produced, built withactor.NewProduced. A retried grant carrying a token you already answered must receive the sameProducedagain, never a second message. Storedacknowledges your submission. Reply withactor.NewStoredAck.
The producer contract
IsAuthorizedFor check matters: RequestNext grants the right to hand over a business message, so honoring a
spoofed grant would let another actor pull work out of your producer.
Backpressure surfaces in the pending buffer. A slow consumer means the buffer grows; bounding or shedding it is an
application policy decision.
Knowing when the consumer confirmed
By default the producer learns that a message was stored, never that it was processed. EnableWithReliableDeliveryConfirmation and the controller tells the producer a DeliveryConfirmed for every message the
consumer confirms:
Knowing when the consumer confirmed
submitters map[string]*actor.PID field
keyed by MessageID and filled in the *OrderCreated case. The handler then relays with any message your submitter
understands:
Knowing when the consumer confirmed
Tell that carries no reply address, so relaying the outcome to the original submitter is your
code, using correlation the producer recorded at ingress.
Feeding the flow
The producer is a normal actor, so work reaches it like any other message: from application code withactor.Tell, from
another actor’s Receive, or from a remote node. Nothing in the sender knows reliable delivery exists:
Feeding the flow
Delivery’s sender is the controller, never the business origin.
The consumer contract
The consumer processes eachDelivery idempotently, then confirms to the sender:
The consumer contract
One-way flows and Ask
Every hop inside the machinery is aTell; the flow carries one-way, ordered, confirmed transfers. Ask still works at
the edges with bounded meaning:
- A caller may
Askthe producer, but the producer can only answer from local knowledge, such as “accepted into my buffer”.Storedacknowledges controller storage; it is not consumer confirmation. When the producer needs to know the consumer confirmed, enableWithReliableDeliveryConfirmationand handleDeliveryConfirmed(see Knowing when the consumer confirmed). Relaying that signal to an original submitter remains application code:Askat the edge still cannot mean “delivered”. - The consumer cannot reply to the original submitter through the flow, because
Delivery’s sender is the controller. Carry correlation in the payload and reply through ordinary messaging, useDeliveryConfirmedat the producer when a completion signal is enough, or run a second flow in the opposite direction when the reply itself must be reliable.
Payload serialization
Every reliable payload is encoded before it is sequenced or stored, even on a single node, using the same serializer dispatch as remoting. Protobuf payloads work with no configuration. Any other payload type must be registered:Payload serialization
Large messages
One reliable payload rides one remoting transport frame. The remotingmaxFrameSize cap is 16 MiB, and an oversized
frame closes the connection without an error frame, so a large reliable payload can wedge the flow in a resend loop that
kills the connection on every attempt. Enable chunking when payloads may approach that bound or monopolize a shared
connection:
Large messages
MinReliableChunkSize, MaxReliableChunkSize] (1 KiB through 16 MiB minus envelope headroom).
Each chunk consumes one sequence number; the consumer controller reassembles before Delivery, so your consumer still
sees one business message. A message must fit in the consumer’s flow-control window worth of chunks: the consumer
confirms nothing mid-message, so a message needing more chunks than the window can never drain. A violation fails the
flow terminally and names the remedy (raise WithReliableFlowControlWindow or the chunk size).
With a durable queue, the producer controller stores the whole chunked message through StoreChunked so a crash
mid-message cannot mix two encodings of the same payload. A resubmission after a producer crash recovers the stored
shape even when the re-encoded payload crosses the chunk threshold in either direction: the first write stays
authoritative. Queue implementors must treat the batch as atomic; see Durable delivery. The derived
chunk identities live in a reserved namespace: an application MessageID must not start with GoAktChunk:, and
NewProduced rejects one that does.
Durable delivery
Without a durable queue, a producer-side restart loses the messages already handed to the controller but not yet confirmed. To survive producer crashes, attach aDurableProducerQueue:
Durable delivery
Durable delivery
Load returns a new positive epoch that fences every earlier
one, and stale writers receive ErrQueueFenced; the first Store / StoreChunked for a business MessageID wins, so
retries and nondeterministic serializers cannot create conflicts; a Store addressing a MessageID owned by a retained
chunked batch returns ErrQueueChunkedBatch so the controller can recover the batch instead of appending a duplicate;
state integrity violations return ErrQueueConflict; other backend errors are retried under WithReliableQueueRetry.
The full contract is documented on the interface.
A producer backed by its own recoverable source completes the loop: it durably marks or removes the submission before
replying StoredAck, and after a crash it resubmits every unmarked item with its original MessageID. The queue’s
first-write ownership then guarantees one durable sequence per message.
Deduplication and exactly-once effects
MessageID is the canonical deduplication key. The producer generates it once, when work enters its pending buffer, and
it identifies the same business message across retries, controller restarts, sessions, and durable recovery. Seq only
orders messages within one sequencing history.
Redelivery is normal, not exceptional. A lost confirmation is indistinguishable from a lost delivery, so the consumer
controller resends and the consumer sees the same message twice:
MessageID also closes the ambiguous-acceptance window: if the producer crashes after the controller stored a message
but before the handshake completed, resubmitting under the same MessageID returns the original sequence instead of
storing a duplicate.
For exactly-once business effects, the consumer must commit the MessageID and the business mutation in one
transaction:
Deduplication and exactly-once effects
seen map, as in the example above, deduplicates within one consumer lifetime only.
Configuration
- WithReliableFlowControlWindow (consumer): Demand granted per request and the consumer-side buffer capacity; maximum 10,000. Default: 50.
- WithReliableResendInterval (consumer): Consumer controller tick: re-registration and resend cadence. Default: 2s.
- WithReliableDurableQueue (producer): Durable queue for producer-crash survival. Default: none.
- WithReliableQueueRetry (producer): Durable operation retries before the controller raises a reliability error. Default: 3 attempts, 100ms.
- WithReliableRetryInterval (producer):
RequestNextandStoredretry cadence toward the producer actor. Default: 500ms. - WithReliableChunking (producer): Split large payloads into sequenced chunks; size in [
MinReliableChunkSize,MaxReliableChunkSize]. Default: off. - WithReliableDeliveryConfirmation (producer): Tell the producer a
DeliveryConfirmedwhen the consumer confirms a message. Default: off. - WithReliableRemoteConsumer (producer): Remoting address of the node hosting the consumer endpoint in a remoting-only flow. Default: none.
- WithReliableRemoteProducer (consumer): Remoting address of the node hosting the producer endpoint in a remoting-only flow. Default: none.
Clustered flows
A flow can cross nodes in one of two modes: both systems join the same GoAkt cluster and resolve each other through its registry, or both run remoting-only and name each other’s address explicitly (see Remoting-only flows). This section covers cluster mode, where controller discovery and endpoint relocation use the cluster registry. Remote placement of a reliable endpoint on a remoting-only system fails fast withErrReliableClusterRequired instead of spawning a flow that never connects:
The spawn calls are unchanged; each node spawns its endpoint and the controllers find each other through the registry:
Clustered flows
- Register the payload types on every node.
- A durable queue is a serializable dependency: register its type on every node eligible to host the producer, and make sure the reconstructed instance observes the same durable state from every node.
- Run the cluster with a replica count of at least 2 when reliable endpoints are relocatable. With a single copy, registry state owned by a lost node disappears with it and the peer lookup cannot recover.
Remoting-only flows
Two nodes with remoting enabled and no cluster form a flow by naming each other’s address explicitly: the producer side carriesWithReliableRemoteConsumer(host, port) with the consumer node’s remoting address, and the consumer side
carries WithReliableRemoteProducer(host, port) with the producer node’s. Each endpoint is spawned locally on its own
node; there is no remote placement in this mode.
Remoting-only flows
A flow follows exactly one resolution authority. A peer address requires remoting (
ErrReliablePeerRemotingRequired
otherwise) and cannot be combined with clustering (ErrReliablePeerClusterConflict), both rejected at spawn.- There is no relocation: nothing exists to relocate through. Node loss is recovered by restarting the peer process at its configured address; the restarted endpoint has a new incarnation and the ordinary registration resync reconnects the flow.
- The loss boundary equals a process crash: with a durable queue the producer reloads stored state and redelivers, without one controller-unconfirmed messages are lost.
- Work-pulling does not support peer addresses; a worker set spanning nodes requires cluster mode.
- Payload types must be registered on both nodes, exactly as in cluster mode.
Failure handling and recovery
Transient trouble heals itself: lost messages, unreachable peers, and controller restarts are recovered by the protocol’s timers, and retriable durable-queue errors restart the producer controller, which reloads authoritative queue state. Deterministic conditions that no retry can fix take a different path: Non-recoverable conditions stop the affected controller and publish exactly oneReliableDeliveryFailed event on the
event stream, while your endpoint actor stays alive:
- ReliableDeliveryStageLoad: Durable queue state could not be loaded on a controller restart.
- ReliableDeliveryStageStore: Durable store failed terminally (fenced writer or state conflict).
- ReliableDeliveryStageAccept: Durable acceptance failed terminally.
- ReliableDeliveryStageConfirm: Durable confirmation failed terminally.
- ReliableDeliveryStageProtocol: Deterministic contract violation, including an unregistered or undecodable payload type.
Failure handling and recovery
ReliableControllerRoleProducer or
ReliableControllerRoleConsumer). The flow stays disabled until an operator fixes the cause, for example registering
the missing serializer or restoring queue ownership, and then calls ReSpawn on the endpoint:
Failure handling and recovery
ReSpawn restarts the endpoint, recreates its controller, and for a durable producer acquires a fresh queue epoch.
Submissions the producer kept buffered during the outage flow again once the controller is back.