Use cases
Reach for point-to-point reliable delivery when one actor must hand work to another and the default at-most-once semantics are not enough:- Message loss is unacceptable. Two specific actors need at-least-once or effectively-once processing instead of best-effort Tell. A checkout actor handing accepted orders to a fulfillment processor is the running example on this page: every order must arrive, in order, exactly once in effect, across restarts, redeploys, and node loss.
- Flow-controlled producer-consumer pipelines. A fast producer risks overwhelming a slower consumer or saturating the network. The producer sends only when its controller grants demand through
RequestNext, so the consumer-side buffer stays bounded no matter how fast the producer runs and mailboxes never grow without limit. - Ordered, sequential processing. Deliveries arrive in sequence order, and the next message reaches the consumer only after the current one is confirmed; the consumer controller buffers anything that arrives in the meantime. A coordinator streaming jobs to a worker knows each job was processed, not merely enqueued, before the next one lands.
- Cross-node event forwarding. An actor on one cluster node feeds an actor on another, and the receiver must see every event in order even though the network hop is lossy.
- Crash-resilient handoff. With a durable queue on the producer side, messages that were stored but not yet confirmed are redelivered after a producer crash, restart, or relocation. Combined with an idempotent consumer, this gives outbox-style, crash-surviving handoff between two parts of your system without operating an external broker.
Ask), and for best-effort telemetry where plain Tell is enough.
How it works
One flow connects one producer to one consumer. The actor system spawns an internal controller next to each endpoint; your actors talk only to their own controller, and the controllers run the protocol between themselves: The producer controller sends only when the consumer controller has granted demand, sequences every message, and resends until the consumer confirms. Registration, session management, resends, and deduplication after restarts are entirely the controllers’ job. The full path of one message:Guarantees
Effectively-once holds only on the fault-free path. Any lost message, restart, or relocation legitimately redelivers, so consumer processing must always be idempotent. The framework never claims exactly-once business effects; see Deduplication for how to build them.
Confirmation is business-level. The consumer confirms after processing a delivery, not when it is enqueued, and a delivery is retried until confirmed. It is never dropped, skipped, or marked successful without confirmation.
The guarantee begins at the producer’s handoff to its controller, and with a durable queue at the point the message is stored. The hop into the producer is ordinary at-most-once messaging. A producer that needs ingress reliability feeds itself from its own durable source, as described in Durable delivery.
Enabling a flow
Each side names its peer’s actor name. There are no controller names or handles anywhere in your code.The controllers carry internal, activation-scoped identities under the reserved
GoAkt prefix. They are invisible to Actors, ActorOf, Kill, ReSpawn, and every other actor-management API, and they impose no naming convention on your own actors.The producer contract
The producer buffers incoming work and answers three protocol messages:RequestNextgrants one tokenized send permission. Hold the grant until there is something to send; it never expires.- Answer a grant with exactly one
Produced, built withactor.NewProduced. A retried grant carrying a token you already answered must receive the sameProducedagain, never a second message. Storedacknowledges your submission. Reply withactor.NewStoredAck.
IsAuthorizedFor check matters: RequestNext grants the right to hand over a business message, so honoring a spoofed grant would let another actor pull work out of your producer.
Backpressure surfaces in the pending buffer. A slow consumer means the buffer grows; bounding or shedding it is an application policy decision.
Knowing when the consumer confirmed
By default the producer learns that a message was stored, never that it was processed. EnableWithReliableDeliveryConfirmation and the controller tells the producer a DeliveryConfirmed for every message the consumer confirms:
submitters map[string]*actor.PID field keyed by MessageID and filled in the *OrderCreated case. The handler then relays with any message your submitter understands:
Tell that carries no reply address, so relaying the outcome to the original submitter is your code, using correlation the producer recorded at ingress.
Feeding the flow
The producer is a normal actor, so work reaches it like any other message: from application code withactor.Tell, from another actor’s Receive, or from a remote node. Nothing in the sender knows reliable delivery exists:
Delivery’s sender is the controller, never the business origin.
The consumer contract
The consumer processes eachDelivery idempotently, then confirms to the sender:
One-way flows and Ask
Every hop inside the machinery is aTell; the flow carries one-way, ordered, confirmed transfers. Ask still works at the edges with bounded meaning:
- A caller may
Askthe producer, but the producer can only answer from local knowledge, such as “accepted into my buffer”.Storedacknowledges controller storage; it is not consumer confirmation. When the producer needs to know the consumer confirmed, enableWithReliableDeliveryConfirmationand handleDeliveryConfirmed(see Knowing when the consumer confirmed). Relaying that signal to an original submitter remains application code:Askat the edge still cannot mean “delivered”. - The consumer cannot reply to the original submitter through the flow, because
Delivery’s sender is the controller. Carry correlation in the payload and reply through ordinary messaging, useDeliveryConfirmedat the producer when a completion signal is enough, or run a second flow in the opposite direction when the reply itself must be reliable.
Payload serialization
Every reliable payload is encoded before it is sequenced or stored, even on a single node, using the same serializer dispatch as remoting. Protobuf payloads work with no configuration. Any other payload type must be registered:Large messages
One reliable payload rides one remoting transport frame. The remotingmaxFrameSize cap is 16 MiB, and an oversized frame closes the connection without an error frame, so a large reliable payload can wedge the flow in a resend loop that kills the connection on every attempt. Enable chunking when payloads may approach that bound or monopolize a shared connection:
MinReliableChunkSize, MaxReliableChunkSize] (1 KiB through 16 MiB minus envelope headroom). Each chunk consumes one sequence number; the consumer controller reassembles before Delivery, so your consumer still sees one business message. A message must fit in the consumer’s flow-control window worth of chunks: the consumer confirms nothing mid-message, so a message needing more chunks than the window can never drain. A violation fails the flow terminally and names the remedy (raise WithReliableFlowControlWindow or the chunk size).
With a durable queue, the producer controller stores the whole chunked message through StoreChunked so a crash mid-message cannot mix two encodings of the same payload. A resubmission after a producer crash recovers the stored shape even when the re-encoded payload crosses the chunk threshold in either direction: the first write stays authoritative. Queue implementors must treat the batch as atomic; see Durable delivery. The derived chunk identities live in a reserved namespace: an application MessageID must not start with GoAktChunk:, and NewProduced rejects one that does.
Durable delivery
Without a durable queue, a producer-side restart loses the messages already handed to the controller but not yet confirmed. To survive producer crashes, attach aDurableProducerQueue:
Load returns a new positive epoch that fences every earlier one, and stale writers receive ErrQueueFenced; the first Store / StoreChunked for a business MessageID wins, so retries and nondeterministic serializers cannot create conflicts; a Store addressing a MessageID owned by a retained chunked batch returns ErrQueueChunkedBatch so the controller can recover the batch instead of appending a duplicate; state integrity violations return ErrQueueConflict; other backend errors are retried under WithReliableQueueRetry. The full contract is documented on the interface.
A producer backed by its own recoverable source completes the loop: it durably marks or removes the submission before replying StoredAck, and after a crash it resubmits every unmarked item with its original MessageID. The queue’s first-write ownership then guarantees one durable sequence per message.
Deduplication and exactly-once effects
MessageID is the canonical deduplication key. The producer generates it once, when work enters its pending buffer, and it identifies the same business message across retries, controller restarts, sessions, and durable recovery. Seq only orders messages within one sequencing history.
Redelivery is normal, not exceptional. A lost confirmation is indistinguishable from a lost delivery, so the consumer controller resends and the consumer sees the same message twice:
MessageID also closes the ambiguous-acceptance window: if the producer crashes after the controller stored a message but before the handshake completed, resubmitting under the same MessageID returns the original sequence instead of storing a duplicate.
For exactly-once business effects, the consumer must commit the MessageID and the business mutation in one transaction:
seen map, as in the example above, deduplicates within one consumer lifetime only.
Configuration
The flow-control window bounds how far the producer may run ahead of confirmations. The producer controller never sends beyond the granted demand, which keeps the consumer-side buffer bounded regardless of producer speed.
Clustered flows
A flow can cross nodes in one of two modes: both systems join the same GoAkt cluster and resolve each other through its registry, or both run remoting-only and name each other’s address explicitly (see Remoting-only flows). This section covers cluster mode, where controller discovery and endpoint relocation use the cluster registry. Remote placement of a reliable endpoint on a remoting-only system fails fast withErrReliableClusterRequired instead of spawning a flow that never connects:
The spawn calls are unchanged; each node spawns its endpoint and the controllers find each other through the registry:
- Register the payload types on every node.
- A durable queue is a serializable dependency: register its type on every node eligible to host the producer, and make sure the reconstructed instance observes the same durable state from every node.
- Run the cluster with a replica count of at least 2 when reliable endpoints are relocatable. With a single copy, registry state owned by a lost node disappears with it and the peer lookup cannot recover.
Remoting-only flows
Two nodes with remoting enabled and no cluster form a flow by naming each other’s address explicitly: the producer side carriesWithReliableRemoteConsumer(host, port) with the consumer node’s remoting address, and the consumer side carries WithReliableRemoteProducer(host, port) with the producer node’s. Each endpoint is spawned locally on its own node; there is no remote placement in this mode.
A flow follows exactly one resolution authority. A peer address requires remoting (
ErrReliablePeerRemotingRequired otherwise) and cannot be combined with clustering (ErrReliablePeerClusterConflict), both rejected at spawn.- There is no relocation: nothing exists to relocate through. Node loss is recovered by restarting the peer process at its configured address; the restarted endpoint has a new incarnation and the ordinary registration resync reconnects the flow.
- The loss boundary equals a process crash: with a durable queue the producer reloads stored state and redelivers, without one controller-unconfirmed messages are lost.
- Work-pulling does not support peer addresses; a worker set spanning nodes requires cluster mode.
- Payload types must be registered on both nodes, exactly as in cluster mode.
Failure handling and recovery
Transient trouble heals itself: lost messages, unreachable peers, and controller restarts are recovered by the protocol’s timers, and retriable durable-queue errors restart the producer controller, which reloads authoritative queue state. Deterministic conditions that no retry can fix take a different path: Non-recoverable conditions stop the affected controller and publish exactly oneReliableDeliveryFailed event on the event stream, while your endpoint actor stays alive:
ReliableControllerRoleProducer or ReliableControllerRoleConsumer). The flow stays disabled until an operator fixes the cause, for example registering the missing serializer or restoring queue ownership, and then calls ReSpawn on the endpoint:
ReSpawn restarts the endpoint, recreates its controller, and for a durable producer acquires a fresh queue epoch. Submissions the producer kept buffered during the outage flow again once the controller is back.