Multi-Datacenter

​

Overview

​

Architecture

• Control plane: Manages datacenter registration, heartbeats, state transitions, and event watching. Implementations: NATS JetStream, etcd.
• DC leader: Only the cluster leader in each DC runs the datacenter controller. It registers the local DC with the control plane using cluster members’ remoting addresses.
• DataCenterRecord: Each DC has a record: ID, metadata (Name, Region, Zone, Labels), Endpoints (remoting addresses), State, LeaseExpiry, Version.

​

When to use

• Geographic distribution: Run actors close to users or data in different regions.
• Disaster recovery and failover: Replicate or fail over to another DC.
• DC-aware placement: Spawn actors in a specific datacenter via SpawnOn with WithDataCenter.

​

Control planes

• NATS: Package: datacenter/controlplane/nats. Use case: Lightweight, good for cloud-native setups.
• etcd: Package: datacenter/controlplane/etcd. Use case: Strong consistency, Kubernetes-friendly.

​

NATS

import "github.com/tochemey/goakt/v4/datacenter/controlplane/nats"

natsCfg := &nats.Config{
    URL:             "nats://127.0.0.1:4222",
    Bucket:          "goakt_datacenters",
    TTL:             30 * time.Second,
    Timeout:         5 * time.Second,
    ConnectTimeout:  5 * time.Second,
}
natsCfg.Sanitize()
cp, err := nats.NewControlPlane(natsCfg)

​

etcd

import "github.com/tochemey/goakt/v4/datacenter/controlplane/etcd"

etcdCfg := &etcd.Config{
    Endpoints:   []string{"http://127.0.0.1:2379"},
    Namespace:   "/goakt/datacenters",
    TTL:         30 * time.Second,
    DialTimeout: 5 * time.Second,
    Timeout:     5 * time.Second,
}
etcdCfg.Sanitize()
cp, err := etcd.NewControlPlane(etcdCfg)

​

Configuration

import (
    "github.com/tochemey/goakt/v4/actor"
    "github.com/tochemey/goakt/v4/datacenter"
    "github.com/tochemey/goakt/v4/datacenter/controlplane/nats"
)

// 1. Create control plane
natsCfg := &nats.Config{URL: "nats://127.0.0.1:4222", TTL: 30 * time.Second, ...}
natsCfg.Sanitize()
cp, _ := nats.NewControlPlane(natsCfg)

// 2. Create datacenter config
dcCfg := datacenter.NewConfig()
dcCfg.ControlPlane = cp
dcCfg.DataCenter = datacenter.DataCenter{
    Name:   "dc-1",
    Region: "us-east-1",
    Zone:   "az-a",
}

// 3. Wire into cluster config
clusterCfg := actor.NewClusterConfig().
    WithDiscovery(discoveryProvider).
    WithDataCenter(dcCfg)

system, _ := actor.NewActorSystem("app",
    actor.WithCluster(clusterCfg),
    actor.WithRemote(remoteCfg),
)

​

Datacenter config options

• HeartbeatInterval: How often the DC renews its liveness lease. Default: 10s.
• CacheRefreshInterval: How often the DC registry cache is refreshed. Default: 10s.
• MaxCacheStaleness: Max age of cached DC data before routing is restricted. Default: 30s.
• FailOnStaleCache: If true, SpawnOn with WithDataCenter returns ErrDataCenterStaleRecords when cache is stale. If false, proceeds with best-effort routing. Default: true.
• WatchEnabled: Use watch-based cache refresh when the control plane supports it. Default: true.

​

Data center states

• REGISTERED: Record exists but not yet eligible for routing.
• ACTIVE: Eligible for routing as long as lease is valid.
• DRAINING: Avoid new placements; existing traffic may continue.
• INACTIVE: Not eligible for routing.

​

DC-aware actor placement

targetDC := &datacenter.DataCenter{Name: "dc-2", Region: "eu-west-1"}
err := system.SpawnOn(ctx, "analytics-worker", NewWorkerActor(), actor.WithDataCenter(targetDC))
if err != nil {
    // ErrDataCenterNotReady, ErrDataCenterStaleRecords, ErrDataCenterRecordNotFound, etc.
    return err
}
// Actor is created on a node in dc-2. Use ActorOf to resolve the PID.
pid, err := system.ActorOf(ctx, "analytics-worker")

​

Messaging across DCs

​

Readiness and errors

• system.DataCenterReady: Returns whether the multi-DC controller is operational.
• system.DataCenterLastRefresh: Time of last successful DC cache refresh.
• ErrDataCenterNotReady: Controller not ready; SpawnOn with WithDataCenter fails.
• ErrDataCenterStaleRecords: Cache older than MaxCacheStaleness and FailOnStaleCache is true.
• ErrDataCenterRecordNotFound: Target DC not in active records or not in ACTIVE state.

​

Custom control plane

• Register, Heartbeat, SetState, ListActive, Watch, Deregister

​

See also

• Clustering: Cluster setup and discovery
• Relocation: Actor relocation within a cluster
• Remoting: Cross-node messaging