20. Troubleshooting by Symptom

This chapter starts from “what you see” and tells you where to look first. The “Common problems” at the end of each chapter cover problems specific to that scenario; this one covers problems across scenarios. The full table of messages is in usage section 8.

Have these four ready

ToolWhat it showsHow to use it
The authentication logThe result of every login, and who decidedLog lines starting with authentication protocol=..., present by default
MetricsWhat leaves no log line: dropped packets, overload, upstream statecurl -s http://127.0.0.1:9812/metrics, with [metrics] set
TracingThe full requests and responses of one user or one NAS[log.trace] plus SIGHUP; see chapter 19
--checkProblems in the configuration file, certificates or script syntaxrradius-server --check --config ..., without affecting the running service

The authentication log records only requests that reached a decision. Dropped packets (unknown source, bad signature, overload, retransmissions) leave no log line; look at the metrics. This is the most common misunderstanding: nothing in the log does not mean the packet never arrived.

The NAS sent a request and got no reply at all

rradius deliberately does not reply in many cases (a reply would give an attacker information, or make the NAS think the password was wrong). First see which label of this metric is rising:

curl -s http://127.0.0.1:9812/metrics | grep -E 'rradius_radius_(discarded|duplicates|packets)_total'
What is risingMeaningWhat to do
Nothing movesThe packet does not reach rradiusThe firewall, routing, the address and port set on the NAS; whether [radius] bind listens only on 127.0.0.1; confirm with tcpdump -ni any udp port 1812
discarded{reason="unknown_client"}The source address is not in [[radius.clients]]The log shows unknown client with the source address. When the NAS is behind NAT, rradius sees the translated address. With dynamic clients, see chapter 14
discarded{reason="invalid"}Wrong shared secret, bad format, or no Message-AuthenticatorWhen the log shows lacking a Message-Authenticator: old devices do not send this attribute; set require_message_authenticator = "no" or "auto" on that client (chapter 1). Otherwise check the shared secret, watching for extra whitespace at the end of the file
discarded{reason="overload"}Requests handled at once reached the limitSee “Logins are slow, or only sometimes work” below
duplicatesRetransmissions: the original request is still being handledHandling is too slow for the NAS to wait. See “Logins are slow, or only sometimes work” below
packets rises but there is no replyThe request was handled, but the decision was not to replyForwarded by realm and no upstream answered (chapter 9); the accounting script did not return 'accept'; EAP’s max_sessions_per_device is full

Use Status-Server to separate “network and secret” from “the account”:

echo 'Message-Authenticator=0x00' | radclient -t 2 -r 1 <address>:1812 status <shared secret>

A reply means the network works, the source is recognised and the shared secret is right, so the problem is in the request itself. No reply means the problem is one of those three. Send this from a host in the client list.

Logins are rejected

Find that login’s line in the authentication log and look at handler and account:

authentication protocol="radius" client="wlc" source=10.30.0.10 user="Staff1@corp.example.com" account="staff1" chain="default" handler="sql" decision="reject"
handlerMeaningCommon causes
sqlThe sql handler rejectedWrong password; the password column is NULL or empty; more than one row found; the password’s format cannot be used with this login method (CHAP needs plain text, MS-CHAPv2 plain text or {nthash})
ldapThe ldap handler rejectedWrong password, not in required_groups; when the directory cannot be reached or the service account fails to log in, there is also a log message starting with auth ldap handler:
lua, scriptA script rejectedThe script’s logic, or a script error (the log shows Lua script error, with the file and line)
noneEvery handler skippedThe account does not exist (sql found nothing, ldap did not find it). First check that account is the name you expect
no_password_checkNobody in an all_accept chain checked a passwordThe account is not in sql or ldap; only a later script said yes
normalizationThe name was stopped before the chainThe name was empty after processing, or does not match [auth.username] allowed
eap-tls, eap-peap, eap-ttls, eap-aka-primeThe EAP conversation ended in a rejectThe line before usually has EAP conversation with client ... failed or refused with the reason; see “Wi-Fi (802.1X)” below
dpskA per-person passphraseNobody’s passphrase matched (account is empty), or it matched but that person is disabled (chapter 12)
A realm name, with chain="proxy"The upstream rejectedInvestigate on the upstream’s side

The password is definitely right:

  • Is account the same as the name in the table? Case, and whether it carries a realm. See [auth.username] (usage section 3.6).
  • Does the log show RADIUS response attribute error? That means a reply attribute’s name or value is wrong (reply_query, group_reply, the script’s set_attr), which rejects even with the right password.
  • Was the log line perhaps not written by this rradius at all? With several nodes, look on each.
  • Was a name with @ forwarded by [proxy]? Then chain is proxy.

No log line for this login at all: go back to the previous section; it was dropped. Or [log] auth = false has turned the log off.

Logins are slow, or only sometimes work

What to look atMeans
RADIUS authentication overloaded: discarded N packets in the logRequests handled at once reached max_auth_inflight (4096 by default). This is the effect, not the cause: something is making each request slow
rradius_radius_duplicates_total risingThe NAS is retransmitting; handling takes longer than its retransmission interval
sql_query timed out or sql_execute timed out in the logThe database is slow, or the cluster has no majority. Such logins get no reply (rradius_radius_discarded_total{reason="database"}), and the NAS retransmits or asks another node
Timeouts from auth ldap handler: in the logThe directory is slow or unreachable
rradius_script_http_requests_total{result="timeout"} risingAn API called by a script is slow (chapter 18)
rradius_proxy_requests_total{result="timeout"} risingAn upstream is slow or down

Finding the slow part:

  • Password hashing: bcrypt and argon2 are deliberately slow, verifying no more at once than there are CPU cores, with the rest queueing. At peaks, a few hundred logins a second already queue. For high login volumes, switch to sha512-crypt (chapter 19).
  • Writes: scripts that write to the database on every login (quotas, IP assignment, two-step verification) go through the cluster and are slower than reads alone. Right after the cluster loses its majority, writes fail only after [lua] query_timeout_ms; after more than 5 seconds without a leader they fail at once. These logins all get no reply rather than a reject.
  • The first upstream does not answer: when forwarding by realm, every request first waits for the first server to time out before trying the next, until it is marked down. probe_interval_ms notices earlier (chapter 11).
  • Raising max_auth_inflight usually does not help. The NAS gives up after a few seconds, and a longer queue just means more wasted work. Fix the cause of the slowness first.
  • Accounting slowing logins down: the number of accounting scripts running at once is limited (CPU cores minus 2), leaving threads for logins; but both share the database, so heavy accounting scripts still affect logins.

Accounting keeps being retransmitted, or accounting is missing

The NAS keeps retransmitting the same accounting: the accounting script did not return 'accept', so rradius did not reply.

  • Lua script error (radius/accounting) in the log: the script failed; look at the file and line.
  • Lua script did not accept accounting in the log: the script ended normally but returned something else. Packets you do not want to record must also return 'accept'.
  • sql_execute timed out in the log: the database cannot take writes. With the accounting spool set up, this case is written locally first and then answered (chapter 7).

Accounting is missing:

CheckMeans
rradius_acct_spool_bytes above 0Still in the spool waiting for replay, not yet in the database. It empties by itself once the database recovers and the script works
rejected.log in the spool directoryRequests given up during replay: rejected by the script, SQL syntax or constraint errors, a client that no longer exists. The file is not cleaned up automatically, and never replayed
Files ending in .corrupt or truncated.log in the spool directoryLeft by power loss or disk problems; the requests in them were not replayed
rradius_acct_spool_full_total risingThe spool is full or writing failed; these requests got no reply and rely on NAS retransmission
RADIUS accounting overloaded in the logmax_acct_inflight was reached, and packets were dropped
The accounting is forwardedLook at rradius_proxy_acct_spool_* (chapter 11)

The spool never goes down (rradius_acct_spool_bytes stays above 0): replay is stuck on the first record. The database has not recovered, or the script fails every time. Look at log messages starting with Lua script and accounting spool; once the script is fixed and SIGHUP sent, it continues by itself.

The same accounting was recorded twice: the script is not safe to repeat. When a NAS updates Acct-Delay-Time and retransmits, it uses a new Identifier, so rradius cannot tell it is a retransmission; spool replay can also run it again. Writes must use INSERT ... ON CONFLICT or INSERT OR IGNORE (chapter 1).

Wi-Fi (802.1X)

SymptomCause
EAP conversation with client ... failed in the log, with a reason mentioning StateThe second half of the conversation reached another node. One login must stay on one node: primary/backup on the NAS, or a load balancer pinning by source (chapter 8)
... failed with a TLS error as the reasonThe client broke off because it does not trust the server certificate (no CA on the device, a server name mismatch), or TLS versions or cipher suites do not match
... refusedThe certificate (or card) passed, but the name does not match, or the chain did not accept it: certificate_query found nothing, or ldap has no certificate_filter
the chain did not accept the inner nameWrong PEAP or TTLS password, or no such account
the chain accepted the inner name without checking a passwordThe handler that accepted is a lua handler without verifies_password
this client has no chain to check a password withPEAP and TTLS need an authentication chain, and this client has none
PEAP keeps failing, while a PAP test worksThe password is not plain text or {nthash}; or ldap uses bind mode
The client picks a method you did not want to offerThat client’s eap_methods lists it. To accept only certificates, list only ["tls"]
rradius_eap_sessions_total{result="timeout"} is highClients did not finish: users cancelled after a certificate warning, a poor signal, or the NAS’s EAP timeout is too short
result="evicted" keeps risingConversations at once exceed max_sessions or max_sessions_per_client. When every user goes through the same controller, raise the latter
result="overload"One device opened too many conversations at once (max_sessions_per_device), or EAP-AKA’ rate limits
Revocation and session resumption problemsChapter 15

The inner name cannot be seen: in the authentication log for PEAP and TTLS, user is the anonymous outer name, and account is the real account. [log.trace] matches outer names, so trace the whole NAS with clients = [...], or list the outer names actually used.

TLS connections (RadSec, TACACS+ over TLS, RadSec upstreams)

SymptomCause
RadSec handshake with ... failedThe other side has no certificate, or it is not signed by client_ca_file, revoked or expired, or the TLS versions do not match
The handshake succeeds but the connection is closed at onceThe certificate passed but matched no client: see whether the log says fits no client, is not valid for client ...'s tls_name or needs tls_name
Every device fails to connect at the same timeThe server certificate expired; crl_file is missing a CA; ocsp = "hard" and the responder is down; RadSec has reject_expired_crl on and the CRL expired
A replaced certificate or CRL does not take effectSIGHUP is needed, on every node
proxy server ...: cannot connect ... TLS:Connecting to an upstream failed. The rest of the message says whether the other side’s certificate is not trusted by ca_file, is not valid for tls_name, or the other side does not accept the certificate you presented
accepting a connection failed N timesFile descriptors ran out. Raise ulimit -n, or lower max_connections

openssl s_client can check the handshake on its own; see chapter 10.

Forwarding by realm

SymptomCause
no proxy server answered a request for realm ...No upstream gave a valid answer: down, a firewall, a wrong shared secret, or the upstream’s answer had no Message-Authenticator
proxy server ... is not answering on ...This server was marked down, and the next is used for now. Later a Status-Server checks whether it is back
An upstream is marked down after a mistyped passwordtimeout_ms is shorter than the upstream’s deliberate delay before rejecting
a request for realm ... came back to this serverThe forwarding configuration loops
Local accounts are forwardedThere is a * or an entry covering your own domain; with DNS discovery, your own realms are not listed in local_realms
Every request for some realm is droppedThe DNS lookup failed (chapter 11)

Disconnects (Disconnect, CoA)

The Error-Cause of a NAK states the reason directly:

ResultMeaning
No reply at allThe sender is not in [[radius.coa.clients]], the shared secret is wrong, or more than 256 requests at once. rradius_coa_relay_discarded_total increases
NAK 502No NAS found: the request’s NAS-IP-Address or NAS-Identifier matches no client; or a NAS recognised only by certificate has no connection open
NAK 503Only a user name was given, and session_query found no open session
NAK 505The NAS gave no valid answer (CoA not enabled, wrong port, rradius’s source not accepted, a different shared secret); session_query failed; or the request contains attributes that must be encrypted
NAK with another Error-CauseThe NAS’s own answer, passed back as is

send_disconnect and send_coa returning "timeout", and rradius-server disconnect printing timeout, have the same causes as the first item of 505. The command sends from the host it runs on, so the NAS must accept that source address.

TACACS+

SymptomCause
The connection is closed at once, with TACACS+ connection from unknown client in the logThe source is not in [[tacacs.clients]]
Authentication answers ERROR, with a body did not decode, so the shared secret may be wrong in the logThe shared secrets differ. rradius_tacacs_malformed_total increases too
Logins work, but every command is refusedThe authorization script refuses, or no rule of the authorization group holds (chapter 6)
enable always fails, with TACACS+ enable request refused in the logEnable is refused by default. Do not put rradius in the device’s enable methods, or set enable = "login"
CHAP and MS-CHAPv2 logins get RESTARTThis client has no authentication chain, or the chain only has a script without verifies_password
Logins keep answering ERRORMore than 8 unfinished logins on one connection, or a single login taking more than 60 seconds
TACACS+ overloaded: closed N connectionsmax_connections was reached. When devices open connections and never close them, lower idle_timeout_ms
TLS problemsChapter 16

Startup fails

Look at the exit status:

StatusMeaningCommon causes
2The configuration is wrong, and nothing has startedThe message names the file and field: a misspelt field, a missing file, a script syntax error, a certificate and key that do not match, key file permissions too broad, problems in migration file names or contents
1The configuration is fine, but the environment does not allow itThe RADIUS UDP ports or the CoA port cannot be bound (in use, or no permission; the message gives the address and port); the cluster was not ready within migrations_wait_ms (other nodes not up, or unreachable)
  • When TACACS+, RadSec or metrics ports cannot be bound, the process does not exit; it logs one error line, and other services run as usual. So with TACACS+ on port 49 and no permission, the process looks healthy but devices cannot connect; after startup, confirm the log has listening for every port.
  • migration file '...' changed after it was applied: an applied file was changed. Restore its original contents, and add a file with the next number for changes. In development you can stop the server, delete data/ and start over.
  • A node keeps exiting with status 1, with cluster was not ready to apply migrations in the log: the majority of nodes is not online yet; or migrations_wait_ms is shorter than the slowest SQL file takes to run.
  • Raft node has not been initialized keeps appearing: the node has not joined the cluster yet. Have the other nodes started? Can they reach each other’s raft and api ports? Is raft.secret the same?
  • --check before starting catches most of the status 2 problems, but it does not connect to the database or directory, so wrong SQL or a wrong LDAP password only shows up after startup.

A reload did not take effect

  • reload failed, keeping the running configuration: ... in the log: the new configuration has an error, and the service keeps using the old one. Fix it and send SIGHUP again. rradius_reload_total{result="error"} increases.
  • these need a restart to apply: ... in the log: the settings listed changed but apply only after a restart; the others have been applied. The list is in chapter 19.
  • Only one node took effect: each node reloads on its own; files must go on every node and the signal to every node.
  • The dictionary file changed: the dictionary is not reread; restart.
  • The migration directory changed: new files are applied only at startup.

The cluster

SymptomCause
Login results on one node differ from the othersThat node’s migrations/, scripts or configuration file differ from the others’; or it is partitioned (rradius_cluster_leader is 0) and reads stale data
Some logins on one node get no replyrradius_radius_discarded_total{reason="database"}: it cannot reach the cluster database; reason="no_leader": stop_without_leader_ms is set and it has been without a leader too long (chapter 8)
A restarted node takes a while to start answeringNormal: it waits for its local replica to catch up with the cluster before answering, logging has caught up with the cluster. After a long downtime the catch-up takes longer, up to migrations_wait_ms
Data just written cannot be readsql_query reads the local replica, which may lag slightly behind writes. To know whether a write succeeded, use sql_execute’s return value instead of querying again
Every write times outThe majority was lost (one of three left). Read-only logins work as usual; logins that need to write get no reply and TACACS+ answers ERROR; accounting goes to the spool when there is one, until the nodes come back
A node exits by itself with status 1, with exiting so that a restart recovers it in the logIts Raft core stopped (commonly because the cluster wrote more than snapshot_every while it was disconnected), or hiqlite panicked internally. Restarting it lets it catch up; with automatic restarts set up there is nothing to do (chapter 8)
Stopping a node takes about 10 secondsNormal: with several nodes, hiqlite waits 9.5 seconds before closing
Restarted while it was the last node, it never comes up (exiting with status 1 when there are migrations)A restarted node waits for a leader. Bring the other nodes back first (chapter 8)
Startup is slow after an abnormal exitThe writes since the last snapshot have to be replayed. See snapshot_every in usage section 2.2
Features that behave differently on each nodeThe accounting spool, EAP conversations, sessions without shared_tickets, results remembered for dynamic clients and per-person passphrases, RadSec connections, and tracing timings all belong to each node

The log itself

  • rradius_log_dropped_lines_total rising: the log is written faster than its destination takes it, or writing fails (a full disk, the receiving process exited). Dropped lines are gone; the service is unaffected. Check where stderr goes; with heavy traffic, turn off [log] auth, or do not trace whole NAS.
  • Messages ending in (N since the last line): messages anyone can trigger (unknown sources, failed handshakes) appear at most once per second of each kind, and N is the total for that period.
  • WARN and ERROR from hiqlite: a few lines appear when the cluster starts and stops, such as No metadata exists inside the DB yet, and are normal.
  • Lines starting with script ...: ... are written by scripts with log(), and are not errors.
  • Every message goes to stderr. The results of management commands (user, disconnect, coa) go to stdout.
Built with Hugo
Theme Stack designed by Jimmy