14. Hundreds of Branch NAS: Dynamic Clients

Scenario: a retail chain has hundreds of branches, each with an AP or gateway or two, and every month new stores open and old ones replace equipment. With the approach of the earlier chapters, every NAS needs its own [[radius.clients]] in the configuration file, and every change means editing every node’s configuration and reloading.

Instead, put the NAS in a table:

  • For requests whose source is not in the configuration file, rradius looks the source address up in the table, and when found, handles the request with the name and shared secret it found.
  • Adding one takes a single rradius-server client add; once written to the table, the whole cluster sees it, with no configuration change and no restart.
  • Each NAS can have its own shared secret.

When you do not need this: with a dozen or so NAS that rarely change, the configuration file is fine. Nor with NAS on RadSec, which can already be recognised by certificate alone (chapter 10).

Example directory: rradius-server/examples/scenarios/branch-nas/

FileContents
rradius.toml[radius.dynamic_clients], [accounts.clients] for the client command
migrations/1_schema.sqlnas (the branches’ NAS), accounts and accounting (as in the quick start)
migrations/2_demo_data.sqlOne demo account clerk1, one demo NAS

How a packet is recognised

source address
  ├─ matches some [[radius.clients]] ─────────────────► the configuration file's (always first)
  ├─ not in the ranges of networks ───────────────────► dropped, no reply
  └─ in networks
       ├─ this address is remembered (found, not expired) ─► the remembered name and secret
       ├─ "not found" is remembered (not expired) ─────────► dropped
       └─ neither → look up the table
            ├─ found ─► remembered for lifetime_ms, this packet handled
            ├─ not found ─► remembered for negative_lifetime_ms, dropped
            └─ database error or too slow ─► dropped, not remembered, the next packet looks up again

Steps

1. Start

cd rradius-server
./target/release/rradius-server --config examples/scenarios/branch-nas/rradius.toml

2. Add a NAS

printf '%s\n' 'store-0107-secret-change-me' \
  | ./target/release/rradius-server client add --config examples/scenarios/branch-nas/rradius.toml \
      --name store-0107-ap1 10.20.1.7
# added client 10.20.1.7

The row is now written across the cluster. From then on, requests from 10.20.1.7 with this shared secret are handled, under the name store-0107-ap1.

The example’s networks is 10.20.0.0/16, and local test requests can only come from 127.0.0.1. So to see a dynamic lookup locally, first change networks to ["127.0.0.0/8"], then client add --name test-ap 127.0.0.1, and send requests from this machine with that shared secret. That is what the automated test does.

3. Check

curl -s http://127.0.0.1:9812/metrics | grep rradius_dynamic_client
  • rradius_dynamic_client_lookups_total{result="found"}: the number of successful lookups. A NAS is looked up only once within lifetime_ms.
  • rradius_dynamic_clients: how many are currently remembered.

The configuration explained

[radius.dynamic_clients]
networks = ["10.20.0.0/16"]
query = "SELECT name, secret FROM nas WHERE address = ?"
lifetime_ms = 300000
negative_lifetime_ms = 60000
max_clients = 10000
  • networks (required): only sources in these ranges are looked up. The narrower the better: a single packet from any address in the range triggers a database query.
  • query (required): one ?, filled with the source address as text (10.20.1.5). The first column is the name, the second the shared secret.
    • An empty name means the source address is used as the name.
    • An empty secret, or a name equal to a client in the configuration file, is treated as not found, and a line is logged.
  • lifetime_ms: how long a found NAS is remembered (5 minutes by default). Changes to the table within that time are not seen by this node.
  • negative_lifetime_ms: how long “not found” is remembered (1 minute by default), during which it is not looked up again, so unknown sources cannot keep triggering queries.
  • max_clients: the most that are remembered. When full, expired entries are cleared first, then “not found” ones.
  • The query’s time limit is [lua] query_timeout_ms. The example sets it to 2 seconds.

The first request of a new NAS

While a source is being looked up, its packets are dropped; they are also dropped when more than 16 lookups are running at once. So the first request of a new NAS usually gets a reply only after one retransmission; after that it is as fast as a client in the configuration file. This is normal, and rradius_dynamic_client_lookups_total{result="busy"} counts it.

How dynamic clients differ from configured ones

ConfiguredDynamic
Authentication chainCan be chosen with authAlways the default chain
EAP methodsCan be chosen with eap_methodsAll of the server’s methods
Message-Authenticator settingsCan be overridden per clientThose of [radius]
As a CoA target (send_coa, CoA forwarding)YesNo
RadSecYesNo, UDP only
Per-client metricsYesNo, only the two totals above

To send CoA to particular NAS, or have them use another chain, put those NAS in the configuration file. Configured clients always come first.

Adding, changing and removing one

With the client command

# Add one: the shared secret is read from standard input, not the command line
printf '%s\n' '<this NAS's shared secret>' | rradius-server client add --config rradius.toml --name store-0405-ap1 10.20.4.5

# Change the shared secret
printf '%s\n' '<the new shared secret>' | rradius-server client secret --config rradius.toml 10.20.4.5

# Show, remove
rradius-server client show --config rradius.toml 10.20.4.5
rradius-server client delete --config rradius.toml 10.20.4.5

The SQL each action runs is set in [accounts.clients], and the table structure is up to you:

[accounts.clients]
add = "INSERT INTO nas (address, name, secret) VALUES (?1, ?2, ?3)"
secret = "UPDATE nas SET secret = ?2 WHERE address = ?1"
delete = "DELETE FROM nas WHERE address = ?1"
show = "SELECT address, name FROM nas WHERE address = ?1"
  • ?1 is the address; ?2 of add is the name and ?3 the shared secret; ?2 of secret is the new shared secret. The full rules are in section 2.14 of the reference document.
  • Checked before sending; when something is wrong, nothing is written and it exits with status 2: the address must be one IP (not a range), must fall inside networks, and must not fall inside the address or range of any configured client (it would never be looked up); the name must not equal a configured client’s (the server would treat it as not found).
  • A shared secret shorter than 16 characters gives a warning, but is written anyway, as for configured clients.
  • The write goes through the cluster, and it can run on any node. The command connects with [cluster]’s secret_api, so make the configuration file readable only by administrators.
  • Results: added client 10.20.4.5, secret changed for client 10.20.4.5, deleted client 10.20.4.5, status 0; no client 10.20.4.5 with status 1 when there is no such row; status 1 when the database refuses because the address or name already exists.

With a migration file

-- 3_new_stores.sql
INSERT INTO nas (address, name, secret) VALUES
  ('10.20.4.5', 'store-0405-ap1', '<this NAS''s shared secret>'),
  ('10.20.4.6', 'store-0405-ap2', '<this NAS''s shared secret>');

Put it on every node and restart them one by one (chapter 8). This suits importing a large batch the first time and keeping the list in version control; later changes use the client command.

How soon changes take effect

ActionWhen this node sees it
Adding a new oneIts next packet (if never looked up before); if it was looked up before and “not found”, at most negative_lifetime_ms
Changing the shared secretAt most lifetime_ms. Meanwhile, packets the NAS sends with the new secret fail verification and get no reply
Removing oneAt most lifetime_ms, during which it still works

The client command only changes the table and does not notify nodes, so the table above still applies. To take effect at once, send SIGHUP to every node: reloading clears everything remembered. Secret changes and removals usually need this.

Before going live

  • Shared secrets are stored in plain text in the table. Anyone who can read the database, and scripts under [lua], can see them. Control access to the data directory, backups and scripts.
  • Source addresses can be forged. UDP source addresses are unreliable; the shared secret and Message-Authenticator are the real verification. The example keeps the [radius] default require_message_authenticator = "yes", which applies to dynamic clients too.
  • The cost of scanning: when someone sends packets from many addresses in the range (or forged sources), each address triggers a lookup and is remembered as “not found”. The limit on concurrent lookups (16) and negative_lifetime_ms bound the load, but keep networks as small as possible anyway, and let the firewall pass only the ranges that really have NAS.
  • When the database has problems: remembered NAS keep being served until lifetime_ms expires; after that, lookups fail and packets are dropped (counted as result="failed"), without being remembered as “not found”. The lookup reads the local replica and by design does not need the cluster’s majority (this case has not been tested specifically).
  • The accounting spool: once accounting from a dynamic client goes into the spool (chapter 7), replay looks the source address up again at that time, and handles it only if the name is the same. So do not rename a NAS while the spool still holds data, or that accounting moves to rejected.log.
  • The client column in the accounting table is the name from the table, so names must be unique and must not change once set. nas.name in the example is UNIQUE.
  • Monitoring: a sudden rise in rradius_dynamic_client_lookups_total{result="not_found"} means unregistered devices are sending requests, or someone is scanning; a rise in result="failed" means a database problem.
  • Not tested: real NAS. The scenario test checks locally that logins and accounting work after client add, that after client secret the node keeps using the remembered secret until SIGHUP, and that after client delete plus SIGHUP there is no reply.

Common problems

SymptomCause
A new NAS gets no reply at allThe source is not in networks; the table’s address differs from the source rradius sees (NAT, several interfaces); or it was just not found and is still within negative_lifetime_ms
A new NAS’s first request gets no reply, and a retransmission worksNormal: packets are dropped during the lookup
The log says the name is the same as a configured clientThe table’s name must not equal the name of any [[radius.clients]]
A NAS stops working after its secret was changedThe node still remembers the old secret; send SIGHUP or wait lifetime_ms
It still works after removalSame as above
EAP for a dynamic client offers methods it should notDynamic clients cannot set eap_methods and get all of the server’s methods. To restrict them, put that NAS in the configuration file
rradius_radius_discarded_total{reason="invalid"} increasesThe shared secret in the table differs from the one on the NAS
Built with Hugo
Theme Stack designed by Jimmy