3. Configuration

The daemon reads one TOML file, /etc/emfor/emfor.toml by default (chapter 2). Every key is optional except the services’ check and backends. examples/emfor.toml shows every option with comments.

Value formats

KindFormatExamples
DurationA string with a unit"500ms", "2s", "1m"
SizeA string with a unit"64MiB"
File modeAn octal string"0644"
PercentageA string ending in %"10%"
AddressIPv4:port or [IPv6]:port"10.0.0.11:8080", "[2001:db8::11]:8080"

Unknown keys anywhere in the file are errors, so a misspelled key never goes unnoticed.

Top level

site = "dc1"
KeyDefaultMeaning
sitenoneThe data centre this host is in. Used by balance = "locality", and required with [peers]. 1–63 characters of a-z, 0-9 and - (lowercased when read). Must come before the first table.
include_dirnoneA directory of further .toml files that are merged into this config (below). Only allowed in the main file. Must come before the first table.

[store]

[store]
path = "/var/lib/emfor/state.mdb"
map_size = "64MiB"
file_mode = "0644"
lock_file_mode = "0666"
sync = false
KeyDefaultMeaning
path/var/lib/emfor/state.mdbThe store file. Clients look here unless EMFOR_STORE says otherwise.
map_size"64MiB"Largest size the store can grow to; at least 1MiB, rounded up to a multiple of 64KiB (LMDB needs a multiple of the page size). Commits fail with MDB_MAP_FULL when it is reached.
file_mode"0644"Mode the store file is created with. Applications need to read it.
lock_file_mode"0666"Mode of <path>-lock. Applications need to write it. See who can use the store before keeping the default.
syncfalsetrue flushes every commit to disk. See the store on disk.

[log]

KeyDefaultMeaning
level"info"trace, debug, info, warn or error. When the environment variable RUST_LOG is set, it overrides this for the whole run.

[dns]

The DNS server runs only when the table is present; an empty [dns] table turns it on with the defaults. See chapter 7.

KeyDefaultMeaning
listen["127.0.0.1:5300"]Addresses to answer on, UDP and TCP.
zone"svc.emfor"Only names under the zone are answered. At most 242 characters. A service in the zone must leave room for the SRV target name of each backend, <ip label>.<service> in at most 253 characters (an IPv6 label is 39).
ttl"1s"TTL of answers and of the zone SOA; whole seconds from 0s to 3600s.
rate_limit100UDP responses a second for each client network (IPv4 /24, IPv6 /56); 0 is no limit. Loopback clients and TCP are never limited. See Rate limit.

[metrics]

The Prometheus endpoint runs only when the table is present.

KeyDefaultMeaning
listen["127.0.0.1:5301"]Addresses to serve /metrics on.

[peers]

Optional. Makes this daemon a member of a peer group: the emfor daemons of one site that list each other here. Members send each other a heartbeat every second, and when one of them sees a backend go from healthy to unhealthy it tells the others, which recheck that backend at once and every 200ms (up to fall checks) instead of waiting for their next interval. With three or more live members, three of them are assigned to each backend: they check it every interval and send their results to the others; two down results among those three mark the backend down on every member. The other members let two agreeing results decide and check the backend themselves about every third interval, and recheck every 200ms when their own check fails while the others see the backend up. The vote and the split need no setting of their own; they use the services' interval, and only members whose check, check_options and timeout match count. Clients still read only the local store. Without [peers] nothing of this runs. See chapter 1.

A service with peers = false (in its table or in [defaults]) is left out of the group on this host: it sends no reports or results, ignores those it receives, and always decides from its own checks. The other services stay in the group. The host announces the services it leaves out in its heartbeats (as short hashes of their names, at most 80 of them), and the other members then do not count it as one of those services’ assigned checkers, so their backends keep three where enough members are live. Each host can therefore leave out its own services. Peer groups and this announcement both came in 0.3.0, and 0.2.0 refuses a config with [peers], so every member reads it. With more than 80 services left out the host logs a warning and the others still count it for the rest.

site = "dc1"                               # required with [peers]

[peers]
listen = "10.0.0.5:5302"
members = ["10.0.0.5:5302", "10.0.0.6:5302", "10.0.0.7:5302"]
secret_file = "/etc/emfor/peers.key"
KeyDefaultMeaning
listenrequiredThis daemon’s UDP address in the group, one of members. A specific address, not 0.0.0.0 or ::; other members check that messages come from it. 5302 is the usual port.
membersrequiredEvery member’s listen, this daemon included: 2 to 64 addresses, no repeats, all IPv4 or all IPv6 like listen. The same list can go on every host of the site.
secret_filerequiredAbsolute path to the shared secret that authenticates and encrypts every message: at least 32 bytes after one trailing line feed is removed, not readable by other users and not writable by group or others (chmod 600 or 640).
previous_secret_filenoneOptional. Absolute path to the secret secret_file held before its last change, with the same conditions. Read only at start, so that a daemon restarted during a secret change keeps signing with and accepting the previous secret as if it had not restarted (see below).
encrypttruetrue sends every message encrypted, with keys derived from the secret: EMP3 (XChaCha20-Poly1305, 24-byte random nonce) to members that run this release, EMP2 (ChaCha20-Poly1305, 12-byte random nonce) to the others, picked per member as described below. false sends them only signed (EMP1: HMAC-SHA256), so backend names, addresses and reasons can be read on the network. All three are accepted either way; with true the daemon logs a warning, once a minute per member, for each member that still sends EMP1.
accept_plaintexttruefalse drops every EMP1 message (counted as plaintext in emfor_peer_messages_rejected_total), so nobody who learns the secret can make this daemon read a signed but unencrypted message. Needs encrypt = true. Set it once every member sends EMP2 or EMP3.

Members only share reports for a service when its check is defined the same way on both: the same check, check_options and timeout. Other settings (interval, rise, fall, the backend list) may differ.

The secret file is read again every second. To change the secret, write the new one on every host within a minute of each other: after its own file changed, a host still signs with the previous secret for 60 seconds and accepts it for 120 seconds, so no message is dropped meanwhile. If the file changes again within those 60 seconds (say, to fix a wrong secret), the host keeps signing with the secret it signed with before, for 60 seconds after the latest change.

The previous secret is kept only in memory. A daemon restarted during a change knows only the new secret: members whose file has not changed yet drop its messages (auth), and it theirs, until their files change. To restart safely during a change, keep the old secret in a second file and name it in previous_secret_file, for example:

cp -p /etc/emfor/peers.key /etc/emfor/peers.key.old  # keeps the mode
cat new.key > /etc/emfor/peers.key  # the same new secret on every host; keeps the mode

A daemon that starts within 120 seconds of the change to secret_file (by the file’s modification time) then signs with the old secret until 60 seconds after that change and accepts it until 120 seconds after it. Later starts ignore previous_secret_file, so it can stay in the config.

Messages more than 5 seconds off the receiver’s clock are dropped, so keep the hosts’ clocks in sync (NTP).

With encrypt = true the format is picked per member. Every EMP2 message this release sends carries "emp3": true in its encrypted payload; a member is sent EMP3 once the latest encrypted message accepted from it is EMP3 or such an EMP2, and EMP2 otherwise: before any message from it was accepted, after a start or a reload that added it, and when it sends EMP2 without the field (0.4.0) or EMP1. Each change is logged at info (“sending EMP3 to peer … from now on”). EMP3 exists because a 12-byte random nonce is safe for about 2^32 messages per secret, which a large group can send within days; a 24-byte one has no such limit. A group running 0.4.0 needs no configuration change to upgrade: upgrade the hosts one at a time, and members of this release use EMP3 between each other after their first message, EMP2 with 0.4.0 members. Going back to 0.4.0 needs no change either, with encrypt on or off: the other members send the host EMP2 again once they accept its first message. Until then, about one second after it starts, it drops their EMP3 messages (counted as malformed).

Encryption was added in 0.4.0. Members running 0.3.0 send and accept only EMP1 and drop encrypted messages (counted as malformed), so to upgrade a group from 0.3.0, set encrypt = false on each host as it is upgraded; once every member runs this release, remove the line (or set encrypt = true) and reload each daemon. Once no member logs the plaintext warning, you can set accept_plaintext = false everywhere and reload. The encryption has no forward secrecy: whoever later gets the secret can read messages recorded before. Rotate the secret regularly, as above, and keep the file readable only by the daemon.

[defaults]

Values every service uses unless it sets its own. The built-in values:

[defaults]
interval = "2s"
timeout = "1s"
rise = 2
fall = 3
jitter = "10%"
on_all_down = "open"
balance = "round-robin"
peers = true
slow_start = "0s"
KeyRuleMeaning
interval100ms to 24hTime from the end of one check of a backend to the start of the next.
timeoutat least 10ms, at most intervalHow long one check may take. A plugin gets it in whole milliseconds, less a small margin.
riseat least 1Consecutive passes before a backend is healthy.
fallat least 1Consecutive failures before a backend is unhealthy.
jitter0% to 50%Random spread of interval, so checks do not line up.
on_all_downopen or closedWhat lookups do when no backend is healthy (chapter 5).
balancesee chapter 5How a backend is chosen.
peerstrue or falseWhether the service takes part in the peer group. No effect without [peers].
slow_start0s (off), or 1s to 1hHow long a backend that recovers gets a growing share of traffic (chapter 5).

[services.“name”]

[services."api.svc.emfor"]
check = "http"
backends = ["10.0.0.11:8080", "10.0.0.12:8080"]
interval = "1s"
dns_ttl = "5s"

[services."api.svc.emfor".check_options]
path = "/healthz"
KeyRequiredMeaning
checkyestcp, http, exec, or the name of a declared plugin.
backendsyesA non-empty list of backends (below).
check_optionsnoOptions of the checker (chapter 4).
interval, timeout, rise, fall, jitter, on_all_down, balance, peers, slow_startnoOverride [defaults] for this service.
dns_ttlnoOverrides [dns] ttl for this service; whole seconds, 0s to 3600s.

Service names are host-name-like: lowercased, one trailing dot removed, at most 253 characters of dot-separated labels, each 1–63 characters of a-z, 0-9 and - not starting or ending with -. Two names that differ only in case are duplicates. Clients normalize the name they look up the same way.

Backends are strings, or tables when they need a weight, a site, backup or drain:

backends = [
  "10.0.4.11:9200",
  { addr = "10.0.4.12:9200", weight = 3 },
  { addr = "10.1.4.11:9200", site = "dc2" },
  { addr = "10.2.4.11:9200", backup = true },
  { addr = "10.0.4.13:9200", drain = true },
]
KeyDefaultMeaning
addrrequiredIPv4:port or [IPv6]:port, port 1–65535. Unique within the service.
weight11 to 100. Used by balance = "weighted", and by "sticky" with an affinity key.
sitenoneThe backend’s data centre. Used by balance = "locality".
backupfalsetrue or false. A backup is used only while no other backend passes its checks (chapter 5). At least one backend of a service must not be a backup.
drainfalsetrue or false. A drained backend is never chosen, by any balance or in DNS, while its checks go on (chapter 5).

weight, site, backup and drain are accepted under every balance, so you can switch balance without editing the backends.

[plugins.“name”]

[plugins."redis"]
command = ["/usr/local/libexec/emfor-check", "redis"]
KeyMeaning
commandRequired. The program and its arguments; the first element is an absolute path to an executable. No element may contain NUL. Run without a shell.

The name is 1–63 characters of a-z, 0-9, _ and -, and not tcp, http or exec. A service uses the plugin with check = "<name>". Every declared plugin is started, used or not. See chapter 4.

Splitting the config over a directory

With include_dir = "conf.d" in the main file, the daemon reads the main file and then every .toml file in that directory, and merges them into one config before checking it. A relative path is relative to the directory of the main file (/etc/emfor/conf.d for /etc/emfor/emfor.toml). This lets a package own emfor.toml while each team or configuration tool owns the files for its services:

/etc/emfor/emfor.toml            include_dir = "conf.d", [store], [dns]
/etc/emfor/conf.d/01-plugins.toml
/etc/emfor/conf.d/10-web.toml
/etc/emfor/conf.d/99-local.toml

Which files, in what order. Regular files, and links to regular files, whose name ends in .toml and does not start with .. They are read in the byte order of their names, not the order of your locale, and a later file overrides an earlier one; the main file is the lowest layer. Write the numbers with leading zeros: 9-a.toml sorts after 10-a.toml. Everything else is ignored (web.toml~, .web.toml.swp, web.toml.disabled, subdirectories), so renaming a file takes it out of the config. A name that is not valid UTF-8 is skipped with a warning in the log. An empty directory is fine.

How files merge. Two tables merge key by key, as deep as they go. Anything else is replaced as a whole by the later value: a number, a string, a list (including backends and command), or a value of another type.

# emfor.toml
[services."api.svc.emfor"]
check = "tcp"
backends = ["10.0.0.1:80", "10.0.0.2:80"]
interval = "2s"
# conf.d/99-local.toml
[services."api.svc.emfor"]
interval = "5s"

The service keeps its check and both backends and gets interval = "5s". To change the backends, give the whole list again. A later file cannot remove a service, a plugin or a backend; take it out of the file that defines it, or rename that file so it is ignored. A service whose merged table has no check or no backends is an error, as in a single file. The merged config is checked, started, reloaded and shared with peers exactly like the same text in one file; how it is split does not change the service definitions peers compare.

Only in the main file. [store] and include_dir itself. emfor status and emfor lookup read only the main file, so a fragment cannot move the store, and a fragment cannot include another directory. Anyone who only runs status and lookup needs to read the main file, not the directory. Everything else, including site, [dns], [peers] and [plugins], may be in any file.

Permissions. The directory and every file read from it must not be writable by group or others (mode 0755 and 0644 are fine, 0775 and 0664 are not); otherwise the config is refused and the error names the path and mode. The owner must be root or the user running emfor (for a link, the owner of the file it points to); otherwise the config is refused too, because a file owned by anyone else can be changed by that user. A fragment can declare a plugin command or an exec check, so whoever can write there can run programs as the daemon’s user. Run emfor check-config as the user the daemon runs as (the systemd unit and the FreeBSD configtest do); run by hand as root, it accepts only files owned by root. The main file is not checked this way, so protect it yourself (0644, owned by root).

The directory must exist. A package creates it. A config whose include_dir does not exist is an error: the daemon logs it and exits 2 at start, and a reload is rejected and the old config keeps running.

Errors name their file. A TOML syntax error is reported as file:line:column, and any other error as its TOML path followed by the file that set that key (services."api.svc.emfor".interval: ... (in /etc/emfor/conf.d/99-local.toml)). When an error comes from values in two files, such as a timeout in one and an interval in another, the file named is the later one.

Writing the files. Write a file to a temporary name outside the directory or ending in something other than .toml, then rename it into place. A reload that reads a half-written file is rejected and the old config stays, but an atomic rename avoids the failed reload.

Checking a config

$ emfor check-config
config ok: 8 services, 19 backends

emfor check-config checks the whole file and prints every error, each with its TOML path, for example:

services."api.svc.emfor".timeout: timeout must not exceed interval

With include_dir set, it then lists the files it read, the main file first and the others in the order they were merged:

config ok: 8 services, 19 backends
read /etc/emfor/emfor.toml
read /etc/emfor/conf.d/10-web.toml
read /etc/emfor/conf.d/99-local.toml

The daemon logs the same list at start and after each reload. Without include_dir the output is the single line.

Seeing the merged config. emfor check-config --show checks the config the same way, and then prints the merged config as TOML on standard output, each value followed by the file that last set it:

# include_dir = "conf.d" (its files are merged below)

[services."api.svc.emfor"]
backends = ["10.0.0.1:80", "10.0.0.2:80"] # /etc/emfor/emfor.toml
check = "tcp" # /etc/emfor/emfor.toml
interval = "5s" # /etc/emfor/conf.d/99-local.toml

A list such as backends is set as a whole, so its comment names the file that set the list. The config ok and read lines go to standard error, so emfor check-config --show > merged.toml saves a file that checks again anywhere, because the include_dir line is written as a comment (its files are merged already); comments aside the file has the same content. Relative paths in the config other than include_dir do not depend on the file’s directory. An invalid config prints nothing on standard output and exits 2. Without include_dir the output starts with one comment naming the file and has no other comments. The built-in defaults are not added: a key you did not write does not appear. Nothing is redacted; the config holds password file paths, not passwords, but a plugin command is printed as written, so do not put a secret there. The comment format is for people; do not parse it.

It exits 0 when the config is valid and 2 otherwise (1 when it cannot write its output, for example to a full disk; a closed pipe is not an error). It starts the declared plugins so that they can check their services’ check_options, and stops them before it exits. It does not open the store. A daemon started with an invalid config logs the same errors and exits 2.

What a reload changes

SIGHUP (systemctl reload emfor, service emfor reload) re-reads the file, and the files of include_dir with it. A config with any error, in any file, or with an include_dir that has gone missing, is rejected as a whole: the daemon logs every error and keeps running the old config.

ChangeOn reload
Services and backends added or removedApplied; new backends start unknown.
check or check_options of a serviceApplied; its backends start over from unknown.
interval, timeout, rise, fall, jitter, on_all_down, peersApplied; backends keep their state. A new interval or jitter also applies to the wait already under way: the next check is due the new wait after the last one ended, at once when that is past.
balance, weight, site (top level and backends), backup, drain, dns_ttl, slow_startApplied; backends keep their state, and a backend in slow start keeps its recovery time.
[dns] zone, [dns] ttl, [dns] rate_limitApplied.
[log] levelApplied (unless RUST_LOG is set).
A plugin’s commandThe plugin is restarted, and the services that use it start over from unknown. Unchanged plugins keep running.
Anything in [store]Rejected; needs a restart.
[dns] listen, adding or removing [dns]Rejected (dns listen settings require restart).
[metrics] listen, adding or removing [metrics]Rejected (metrics listen settings require restart).
[peers] members, [peers] secret_file, [peers] encrypt, [peers] accept_plaintextApplied; added members start as not live. previous_secret_file is checked, but read only at start.
[peers] listen, adding or removing [peers]Rejected (peers listen settings require restart).
Built with Hugo
Theme Stack designed by Jimmy