8. High-Availability Cluster

Scenario: authentication must not stop. Three rradius servers form a cluster, with data replicated to each of them automatically; when any one stops (a failure, an upgrade, a reboot), the other two keep serving, and the NAS move to the nodes that are alive.

Example directory: rradius-server/examples/scenarios/ha-cluster/

FileContents
node1.toml, node2.toml, node3.tomlThe three nodes’ configurations, differing only in node_id, listen and [metrics] bind
raft.secret, api.secretSecrets internal to the cluster, the same on all three
nas.secretThe NAS shared secret, the same on all three
migrations/1_schema.sqlTables, the same as the quick start, with no demo accounts

The accounting script reuses ../quick-start/accounting.lua from the quick start, so this chapter covers only the cluster itself. You add accounts yourself; see the steps below.

Architecture

             NAS (primary: node1, backups: node2, node3)
              │ RADIUS 1812/1813
   ┌──────────┼───────────────┐
 node1      node2           node3       each holds all the data
 10.99.0.1  10.99.0.2       10.99.0.3   (WireGuard addresses)
   └──── raft 8100 / api 8200 ────┘     nodes talk only over WireGuard
  • Writes (sql_execute, migrations) go through Raft and succeed only when a majority (2 of 3) agrees, so every node ends up with the same result.
  • Reads (sql_query, the sql handler) read the local replica. They are fast, but may lag slightly behind data just written.
  • Three is the smallest highly available size: with one stopped there is still a majority; with two nodes, stopping one leaves none. Five nodes can lose two at once.
SituationAuthenticationAccounting
All three upNormalNormal
One stoppedNormal (the NAS ask the others). If the one stopped is the leader, for the few seconds until a new leader is elected, logins that need to write get no reply and accounting writes time outNormal; if the leader stopped, those few seconds of accounting go to the spool (possibly once more than needed, see below)
Two stopped, one leftAuthentication that only reads data (sql, ldap) reads the local replica and answers as usual; logins that need to write (one-time codes, IP assignment, SIM cards) get no reply, so the NAS retransmits or asks another node. With stop_without_leader_ms set, past that time even read-only authentication and Status-Server get no replyCannot be written. With the accounting spool, it is written locally and answered, then replayed when the cluster recovers
  • When the database is unusable, there is no reply rather than a reject. When a login cannot be decided because the cluster database timed out or there is no leader, RADIUS does not reply and TACACS+ answers ERROR, so NAS and devices ask another node, and users never see “wrong password”. This is counted in rradius_radius_discarded_total{reason="database"}. A reject is sent only when the account certainly does not exist or the password is certainly wrong, and every database call of the script succeeded; a reject that follows a failed database call in the script also counts as undecided. When a script returns ignore after a database failure, the next handler runs as usual. EAP conversations in progress are kept, and continue when the NAS retransmits.
  • stop_without_leader_ms ([cluster], unset by default): when a node has been without a leader for longer than this, even read-only logins and Status-Server get no reply (TACACS+ answers ERROR), counted as reason="no_leader", so NAS and load balancers treat it as failed and ask the other side of the partition. The price is that when this is the only node left in the cluster, it does not serve either. Accounting is unaffected. Changing it needs a restart.
  • A node that was disconnected for a long time may need one restart when it comes back. When more than 100,000 writes (snapshot_every) happened while it was away, hiqlite cannot read its own log when it returns; rradius logs exiting so that a restart recovers it and exits with status 1, and after a restart it catches up with the cluster. So every node needs automatic restarts (systemd Restart=on-failure, Docker restart: unless-stopped).
  • Do not restart the last node left. The running node keeps serving read-only logins, but once restarted it can only come up when there is a leader (this is how hiqlite behaves), and with migrations_dir it exits with status 1 after migrations_wait_ms. Bring the other nodes back first.
  • A partitioned node reads stale data. While it is cut off from the majority, accounts disabled on the other side, NAS removed and keys withdrawn all remain valid on it, and SIGHUP does not help, until the network recovers. If that is unacceptable, set stop_without_leader_ms.
  • Writes handed to the cluster wait for their outcome. When the database is slow (the leader is taking a snapshot, for example) or the leader changes, a write the script has already handed to the cluster is not abandoned midway; it waits for the cluster’s answer (at most 30 seconds), so the write is never “reported as failed but actually written”. Accounting can still be recorded twice, though: with no outcome within 30 seconds, accounting gets no reply and the NAS retransmits; and when a write waits longer than the time limit, later database calls in the same request time out at once, so the record goes to the spool and runs again on replay. Statements with RETURNING still follow the time limit. So accounting scripts must still be safe to repeat (for example with INSERT OR IGNORE, keyed on Acct-Session-Id).

Steps

1. Prepare each host

  1. Build the WireGuard network 10.99.0.0/24 with the three hosts as peers, and use the firewall so that 8100 and 8200 accept connections only from wg0. The full steps are in usage section 2.2.

  2. Copy quick-start/ and ha-cluster/ from examples/scenarios/ to each host (for example to /etc/rradius/), keeping their relative positions so node*.toml can find ../quick-start/accounting.lua.

  3. Add your own accounts: set up [accounts] and add them with rradius-server user add, which writes to the whole cluster from any node; or add 2_users.sql to ha-cluster/migrations/, in the format shown in chapter 2 and chapter 19. Do not copy the quick start’s 2_demo_users.sql; its demo accounts have published passwords.

  4. raft.secret, api.secret, nas.secret and migrations/ must be the same on every host. Generate new secrets first, then copy them over:

    (umask 077; openssl rand -base64 32 > raft.secret; openssl rand -base64 32 > api.secret)
    

2. Start

Start each of the three with its own configuration, in any order:

rradius-server --config /etc/rradius/ha-cluster/node1.toml   # on node1
rradius-server --config /etc/rradius/ha-cluster/node2.toml   # on node2
rradius-server --config /etc/rradius/ha-cluster/node3.toml   # on node3
  • On the first start, each node waits for the cluster to form (a majority online) and then applies the migrations, waiting at most migrations_wait_ms (120 seconds in the example). When one comes up before the others, it waits; that is not a failure.
  • Once you see RADIUS Server listening, it serves.
  • NAS configuration: set all three as RADIUS servers, as primary/backup or round robin, with the same shared secret. One EAP login must complete on one node (chapter 3), so with EAP use primary/backup, or have a load balancer pin by source.

3. Check

RADIUS uses the nodes’ physical addresses, not the WireGuard addresses, and requests must come from a host in the client list (10.80.0.0/16 in the example), or there is no reply. From a host in that range, with an account added in item 3 above:

# every node accepts logins (replace 192.0.2.11 to .13 with the three physical addresses)
for ip in 192.0.2.11 192.0.2.12 192.0.2.13; do radtest <user> <password> $ip 0 "$(cat nas.secret)"; done

# each node's metrics, on the node or from inside the WireGuard network
curl -s http://10.99.0.1:9812/metrics | grep rradius_

When the management host is not in the client range, add a temporary [[radius.clients]] with only its address (applied with SIGHUP), and remove it after testing.

Send a Start to node1 and a Stop to node3, and the accounting table has a single row: writes go through the cluster, so it does not matter which node received them.

The configuration explained

[cluster]
node_id = 1
listen = "10.99.0.1"
migrations_dir = "migrations"
migrations_wait_ms = 120000

[[cluster.nodes]]
id = 1
raft = "10.99.0.1:8100"
api = "10.99.0.1:8200"
# ...the same for ids 2 and 3
  • Every node lists all three nodes (itself included), and node_id says which one it is. There must be a node 1.
  • Set listen to the WireGuard address. With [[cluster.nodes]], the default is 0.0.0.0, which opens raft and api on every interface, and connections on those two ports are not encrypted.
  • [lua] query_timeout_ms = 2000: after losing the majority, a write fails only after this long, and only then does accounting go to the spool. The default of 5 seconds is longer than many NAS’s retransmission interval, and retransmissions would be dropped as still in progress.
  • [radius.accounting_spool]: each node replays only its own spool files, so put it on a persistent disk. rradius_acct_spool_bytes above 0 means accounting that has not reached the database yet, which is useful for alerts.
  • [log] format = "json": easier to handle when the three nodes send logs to the same collector.

What belongs to each node

Tables replicate automatically, but the following do not. Go through them one by one when planning several nodes:

ItemPer nodeWatch out for
Configuration files, scripts, dictionary, migrations/One copy eachContents must be the same. After a change, update every node and send each a SIGHUP or restart it
raft.secret, api.secret, NAS shared secret filesOne copy eachContents must be the same
Key files: aka.key (chapter 13), dpsk.key (chapter 12), ticket.seal (chapter 15)One copy eachContents must be the same, and they are not in the database; copy them to every node yourself, and back them up separately
Certificates, private keys, revocation listsOne copy eachWhen updating, replace them on every node and send SIGHUP
Accounting spoolEach replays only its own filesPut it on a persistent disk; when a node dies, whatever it has not replayed is lost
EAP conversations in progressOnly on the node that received themOne login must stay on one node
TLS sessions for resumptionBy default only on the node that issued themWith shared_tickets = true, EAP-TLS can resume across nodes (chapter 15)
Remembered results: dynamic clients (chapter 14), the key matched for a per-person passphrase (chapter 12), upstreams found by DNS (chapter 11)Each remembers its ownWhen a table change must take effect at once, send SIGHUP to every node. A partitioned node reads an old replica, and SIGHUP does not help
Cluster stateEach sees its ownrradius_cluster_leader, rradius_cluster_is_leader, rradius_cluster_applied_index, rradius_cluster_seconds_without_leader in /metrics
RadSec connectionsOn whichever node they connected toOnly the node holding a NAS’s own connection can send CoA over it (chapter 10)
EAP-AKA’ rate limits (only with next_sqn_query)Counted separately by defaultWith shared_budget = true, shared across the cluster; with current_sqn_query and proven_sqn_query there is no such limit (chapter 13)
Upstream down stateEach decides on its own
Metrics, [log.trace] timingsEach its ownCluster totals are summed by the monitoring system; tracing must be set on every node involved

Tuning

  • read_connections: the number of connections reading the local database, twice the CPU count by default (at most 32). Logins are mostly reads, and queue when there are not enough connections. Lower it when memory is tight; each connection has its own cache.
  • snapshot_every: a node takes a snapshot every time this many writes accumulate, 100000 by default. Lower values make restarts faster after an abnormal exit (power loss, SIGKILL), but with many writes, frequent snapshots slow the service down.
  • More nodes do not raise the write ceiling. Every write goes through the same leader and waits for a majority to confirm. Logins (reads) are handled by each node on its own, so more nodes are faster. Between separate hosts, every write also waits for a round trip between nodes.

Measurements and explanations for these three are in usage section 2.2. Changing any of them needs a restart.

Day-to-day operations

Upgrades and reboots

One at a time: stop a node → upgrade → start → confirm it is back in the cluster → do the next. Stopping only one at a time keeps the cluster’s majority throughout.

  • Confirming it is back: the log says cluster node N has caught up with the cluster, and rradius_cluster_leader in /metrics is 1. On startup a node waits for a leader, then for its local replica to catch up with the data the cluster has written, before it starts answering, so accounts added while it was down are visible from its very first answer. The wait is limited by migrations_wait_ms; without migrations_dir, if the wait runs out, it logs a warning and serves anyway (possibly with stale data).

  • Stop the leader last. Do the node whose rradius_cluster_is_leader is 1 last: stopping the leader causes an election, and writes during those seconds time out.

  • Stop with SIGTERM or Ctrl+C, which closes the database cleanly. With several nodes this takes about 10 seconds (hiqlite waits 9.5 seconds first), so use docker stop -t 30 or set stop_grace_period: 30s. SIGKILL (kill -9, a container being forced to stop) does not, and on the next start the local database is rebuilt from the other nodes, which is slower.

Adding a migration

  1. Put the new file (for example 3_add_column.sql) on every node.
  2. Restart the nodes one by one. The first one to restart applies it to the whole cluster; the others see it already applied when they restart.
  3. Do not start nodes whose directories differ at the same time; see usage section 2.5.

Replacing a failed node

Start the new host with the same node_id, the same addresses and configuration, and an empty data/; it takes the data from the other nodes. This procedure has not been tested for real; rehearse it in a test environment first. Adding or removing nodes while running is not supported yet.

Common problems

SymptomCause
No listening after startup, and finally an exit with status 1No majority reached: the other nodes are not started, WireGuard does not connect, a firewall blocks it, or raft.secret differs
Login results on one node differ from the othersThat node’s replica lags (compare rradius_cluster_applied_index with the others), it is partitioned (rradius_cluster_leader is 0), or its migrations/ differs; check its log
Accounting slows down after stopping one nodeIt should not if the stopped node was not the leader. If it was the leader, writes time out and go to the spool during the few seconds of the election. With two stopped, after 5 seconds without a leader writes fail at once and go to the spool
Some logins get no reply, and reason="database" keeps risingThat node cannot reach the cluster database (no leader, partitioned) or the database is too slow. Check rradius_cluster_leader and the log
One node does not answer at all, and reason="no_leader" keeps risingstop_without_leader_ms is set and that node has been without a leader too long. Restore the network between it and the other nodes
Wi-Fi logins occasionally fail, with the log mentioning StateThe second half of an EAP conversation reached another node. Change the NAS to primary/backup, or have the load balancer pin by source
All SIM card or per-person passphrase logins fail on one nodeThat node’s key file (aka.key, dpsk.key) differs from the others
After a table change, some nodes pick it up at once and some do notEach node remembers the results it looked up. Send SIGHUP to every node
Built with Hugo
Theme Stack designed by Jimmy