Context Pools

​

1. The problem it exists to solve

1. Reuse objects so the allocator and GC are not on the hot path.
2. No global lock so concurrent actors do not serialize.
3. Bounded memory so a burst of messages cannot grow the pool forever.

​

2. What is being pooled

• the message envelope (message, sender, self, response, …)
• an intrusive mailbox node via next

​

3. How the pool is built (top down)

process
  └── contextPool  (one, package-global)
        └── N shards  (power of two, 8..128)
              └── each shard: a 512-slot MPMC ring

​

Layer A: shard count

var contextShardCount = func() uint32 {
	n := uint32(8)

	for n < uint32(2*runtime.GOMAXPROCS(0)) && n < 128 {
		n <<= 1
	}

	return n
}()

func nextContextShard() uint32 {
	return contextShardCounter.Add(1) & contextPool.mask
}

​

Layer B: a shard is Vyukov’s array MPMC queue

type contextShard struct {
	enqueuePos atomic.Uint64
	_          CacheLinePadding

	dequeuePos atomic.Uint64
	_          CacheLinePadding

	cells [contextShardCapacity]poolCell
}

type poolCell struct {
	seq atomic.Uint64
	ctx *ReceiveContext
}

• writers store ctx then store seq
• readers load seq then load ctx

​

Layer C: sequence numbers (the part people get wrong)

​

4. Object lifecycle (what actually happens at runtime)

1. Spawn. newPID calls nextContextShard() and stores it on pid.ctxShard.
2. Send. Local toReceiveContext / PID.Tell / PID.Ask call getContext(to.ctxShard).
   • Hit: pop from that shard. get does not restamp poolShard; the object already carries it from allocation.
   • Miss: new(ReceiveContext) and stamp poolShard.
   • Then build(...) fills message fields.
   Remote Tell on the sender does not touch this pool. A few internal paths (newReceiveContext, used by PipeTo) allocate outside it; if those objects later pass through a recycling mailbox they re-enter as poolShard == 0.
3. Mailbox. The context is enqueued. For UnboundedMailbox, next links it into the MPSC list. It is now owned by the mailbox, not the pool.
4. Dispatch. dispatchOne runs the actor’s Receive against that object and must not return it to the pool. Recycle is owned by the mailbox Dequeue that handed it out.
5. Recycle. Only some mailboxes return contexts to the pool. Know which:
   • UnboundedMailbox (default): the dequeued node becomes the new sentinel. The previous sentinel is reset() + contextPool.put inside Dequeue. The context the actor just processed is not recycled yet; it is the new sentinel and will be recycled on the next dequeue. The mailbox’s initial sentinel is new(ReceiveContext) with poolShard == 0, so the first put lands on shard 0. UnboundedFairMailbox uses one UnboundedMailbox per sender, so it inherits this path.
   • recycleContext (one-dequeue-later): NonBlockingBoundedMailbox, BoundedPriorityMailbox, BoundedStablePriorityMailbox, and UnboundedStablePriorityMailbox keep the last dequeued context in prev and call recycleContext on the next Dequeue. That helper reset()s, clears next, then puts.
   • No recycle: BoundedMailbox, UnboundedPriorityMailBox, and UnboundedSegmentedMailbox return the pointer from Dequeue and never put it back. Those envelopes become GC work after the turn. That is intentional: they do not use the sentinel or prev scheme.
6. Home-shard return. put always uses ctx.poolShard, not the caller’s shard hint. An object allocated for shard 2 always goes back to shard 2, even if some code path passed a different hint.

​

5. Invariants you must not break as a maintainer

1. poolShard is immutable after allocation. reset() must not touch it. put returns to home shard only (TestContextPool_PutReturnsToHomeShardOnly).
2. A context is in exactly one place: pool, or one mailbox, or in-hand on a dispatcher. Never pool an object that is still linked (next != nil). recycleContext clears next for that reason; UnboundedMailbox.Dequeue does it itself.
3. Never hold a pointer into a recycled context. After Ask enqueues, the comment in pid.go is explicit: capture the response channel, then forget the context. It can be rebuilt for an unrelated message immediately after dequeue.
4. Overflow must drop, not block or panic. TestContextPool_OverflowDropsForGC fills past capacity. If you add blocking, you can deadlock a dispatcher waiting to return a context.
5. Get must never return nil. Empty or contended → allocate. Callers do not nil-check.
6. Power-of-two sizes are load-bearing. Capacity 512, shard count 8/16/…/128. If you change either to a non-power-of-two, every & (n-1) becomes a silent bug.
7. Publication order: write ctx, then seq; on pop, observe seq, then read ctx, then nil ctx, then advance seq. Reverse that and you have a data race the race detector may or may not catch depending on timing.
8. cloneContext must clear responseClosed. The test TestCloneContext_ClearsResponseClosed exists because a FIFO leftover from a previous Ask will hide a missing Store(false) if you are not careful.
9. This pool is not the grain pool. Same geometry, separate instance. See Grain sibling. If you change contextShardCapacity, poolSpinLimit, or contextShardCount, decide whether the grain rings (and the grain ack-channel ring) move with them.

​

6. How to think about a change

• Am I changing topology (shard count, capacity, padding)? Measure pairwise Tell and many-actors-many-cores. The original bug was “looks fine on one actor.”
• Am I changing ownership? Trace every getContext to its matching put. UnboundedMailbox.Dequeue and the prev-based mailboxes recycle one dequeue later; BoundedMailbox, UnboundedPriorityMailBox, and UnboundedSegmentedMailbox never put back. dispatchOne must not put.
• Am I adding a field to ReceiveContext? Decide: does reset() clear it? Does cloneContext copy it? Does build() set it? Those three sites are a triple; miss one and you leak identity across messages.
• Am I tempted to use sync.Pool or a channel again? That is the design we left. The comments at the top of the file are the postmortem.

​

7. Mental model to keep

A ReceiveContext is a ticket. The actor’s ctxShard is which window issues tickets. Each window is a 512-slot lock-free ring. If the window is empty you print a new ticket; if it is full you throw the returned ticket away. Tickets always go back to the window that printed them.

​

8. 512-slot MPMC ring

​

Ring

• enqueuePos: where the next spare context is stored (put / push)
• dequeuePos: where the next spare context is taken (get / pop)

cells[pos & (capacity - 1)]

​

512-slot

type poolCell struct {
	seq atomic.Uint64
	ctx *ReceiveContext
}

​

MPMC

• Producers (put): every mailbox consumer recycling a context. Many actors, many dispatcher workers.
• Consumers (get): every Tell/Ask to actors on this shard. Many senders.

1. Load my cursor (enqueuePos or dequeuePos).
2. Look at that cell’s seq.
3. If seq says the cell is ready for this pos, CAS the cursor forward (pos → pos+1). Winning the CAS is how you claim the slot. Losing means another thread took it; reload and retry, up to poolSpinLimit (4) loop iterations total.
4. Only the winner writes or reads ctx, then publishes a new seq.

cells:  [0]    [1]    [2]    [3]
seq:     0      1      2      3     ← constructed empty
enqueuePos = 0, dequeuePos = 0

• 0: this cell matches this cursor; try to claim it
• < 0: empty (get) or full (put); stop
• > 0: someone else already moved past this cell; reload the cursor

​

Put the three words back together

A shard is not a list and not a channel. It is a fixed revolving array that many threads can fill and drain at once, and when it is empty or full it refuses instead of waiting.

​

9. ABA protection

1. You observe A.
2. While you are preempted, other threads change A to B, then back to A.
3. You resume, CAS succeeds, and you finish an operation against a structure that has been emptied, reused, and refilled. You think you were first. You were last.

​

10. Pairwise Tell

​

Three Tell shapes, and why they are not interchangeable

​

Why “pair”

​

How it connects back to the pool

• Each receiver is assigned a ctxShard at spawn (round-robin, shared with other actors).
• Each pair’s getContext/put uses the receiver’s home shard, not a process-wide channel.
• With GOMAXPROCS pairs and 2 × GOMAXPROCS shards, pairs tend to land on different rings, which is why this workload exposes a global lock. It does not prove exclusive per-actor rings; when actor count exceeds shard count, sharing is guaranteed.

​

11. Grain sibling

• grainMailbox.Dequeue is the sentinel path, same idea as UnboundedMailbox: the previous head is reset(), next cleared, grainContextPool.put.
• releaseGrainContext is for failed enqueue (bounded mailbox full, timer tick dropped, passivation pill refused). It is not the steady-state drain path.

• Topology constants are shared. Changing them is a both-or-reasoned decision, including the ack-channel ring.
• Recycle / GrainContext fields / TellGrain acks: leave context_pool.go alone.
• Scaling surface: BenchmarkGrainTellPairwise and BenchmarkGrainAskPairwise are the grain analogue of pairwise Tell. Single-grain tell cannot see a process-wide pool lock.

​

See also

• Dispatcher Pool: the scheduler that drains mailboxes; dispatchOne must not return contexts to the pool
• Mailboxes: UnboundedMailbox sentinel recycle vs recycleContext
• Stashing: cloneContext when a message must enter a second mailbox
• Code Map: actor/context_pool.go and actor/grain_context_pool.go
• Grains: identity-addressed activation; TellGrain still blocks for a processed ack
• Design Decisions: rationale for sharding instead of a process-wide channel