A client library has one call: look up a service, get back one backend address. It reads the local store, so it needs the emfor daemon running on the same host, and nothing else.
The lookup
Every client takes a service name and an optional affinity key, and returns a string:
ip:portfor IPv4, such as10.0.0.11:8080;[ip]:portfor IPv6, such as[2001:db8::11]:8080.
Both forms can be passed to the usual “connect to host:port” functions of
most languages. The name is matched without regard to case and with one
trailing dot ignored. The affinity key matters only for
balance = "sticky"; an empty key is no key.
Call lookup for every new connection, not once at startup: the answer
changes as backends fail and recover, and a lookup costs about a
microsecond.
Errors
Every client fails with one of four codes, named the same in every language:
| Code | Meaning | What to do |
|---|---|---|
UNAVAILABLE | The store cannot be used: it does not exist, cannot be opened, was written by another emfor release’s LMDB, is not initialized, or EMFOR_MAX_STALENESS is invalid. | Check that the daemon runs, EMFOR_STORE, file permissions, and that libemfor comes from the daemon’s release. The message says which. |
INCOMPATIBLE | The store has another format version, or a record cannot be read. | Install the client from the daemon’s release. |
NOT_FOUND | No service of that name. | Check the name against the config. |
ALL_DOWN | No healthy backend, and the service has on_all_down = "closed". | Fail the request, or retry later. |
A stale store is not an error of its own: it makes every backend count as
unhealthy, so an open service still returns an address and a closed
one fails with ALL_DOWN.
Configuration
Clients have no configuration call. Two environment variables are read on
the first lookup in a process (and again in a child after fork):
| Variable | Default | Meaning |
|---|---|---|
EMFOR_STORE | /var/lib/emfor/state.mdb | Path of the store; must match the daemon’s [store] path. |
EMFOR_MAX_STALENESS | 3s | How old the heartbeat may be: a positive whole number with ms or s, such as 1500ms. An invalid value makes every lookup fail with UNAVAILABLE. |
Examples
Rust
match emfor_client::lookup("api.svc.emfor", Some("user-42")) {
Ok(addr) => println!("connect to {addr}"),
Err(e) => eprintln!("{e}"), // "ALL_DOWN: ..."; e.code is an ErrorCode
}
C
#include <emfor.h>
char addr[EMFOR_ADDR_MAX];
emfor_status st = emfor_lookup("api.svc.emfor", "user-42", addr, sizeof addr);
if (st == EMFOR_OK)
printf("connect to %s\n", addr);
else
fprintf(stderr, "%s: %s\n", emfor_status_name(st), emfor_last_error());
Build with cc app.c $(pkg-config --cflags --libs emfor). C also has
EMFOR_INVALID_ARGUMENT for a NULL or non-UTF-8 argument or a buffer that
is too small; EMFOR_ADDR_MAX bytes always fit.
Go
import "github.com/embark-systems/emfor/clients/go"
addr, err := emfor.Lookup("api.svc.emfor", "user-42")
var e *emfor.Error
if errors.As(err, &e) && e.Code == emfor.AllDown {
// no healthy backend
}
Python
import emfor
try:
addr = emfor.lookup("api.svc.emfor", affinity_key="user-42")
except emfor.AllDownError:
...
except emfor.Error as e:
log.warning("%s: %s", e.code, e.message)
Perl
use Emfor;
my $addr = eval { Emfor::lookup('api.svc.emfor', 'user-42') };
warn "$@\n" if $@; # an Emfor::Error with ->code and ->message
PHP
try {
$addr = Emfor\lookup('api.svc.emfor', 'user-42');
} catch (Emfor\LookupException $e) {
error_log($e->getMessage()); // $e->errorCode is "ALL_DOWN" etc.
}
Each client’s README covers installing it and its exact API: Rust, C, Go, Python, Perl, PHP.
From the shell: emfor lookup
emfor lookup answers from the shell what a client library would answer,
for scripts and cron jobs that have no client, and with --explain why.
$ emfor lookup api.svc.emfor
10.0.0.11:8080
$ emfor lookup api.svc.emfor --key user-42
It reads the store at the [store] path of the config (--config, as
emfor status does) and applies EMFOR_MAX_STALENESS like a client. It
never writes the store. --key is the affinity key of a sticky service.
Every run is a new process, so a rotating balance (round-robin,
weighted, locality, fastest, and sticky without a key) gives one of
the candidates, starting at a random point: two runs can print different
backends. A script that needs the same answer every time uses a failover
service, or a sticky one with --key.
--explain says why: whether the store is stale, which tier the candidates
come from (the tiers of chapter 5), every backend
that is out and the reason, and how balance chooses.
$ emfor lookup api.svc.emfor --explain
service: api.svc.emfor
answering: as a client library answers
store: /var/lib/emfor/state.mdb (heartbeat 0.4s ago, fresh)
balance: round-robin on_all_down: open
tier: healthy primary backends whose last check passed
candidates:
10.0.0.11:8080
excluded:
10.0.0.12:8080 last check failed
10.0.0.13:8080 backup held back
10.0.0.14:8080 unhealthy
selection: round-robin: the candidates in turn
the actual choice depends on the rotation start of the process, which is random in a new process
answer: 10.0.0.11:8080
The reasons are stale (a stale store makes no backend healthy or
unknown), unhealthy, unknown (a better tier has candidates), last check failed, backup held back, not in the pool (fastest) and not at this site (locality). A candidate in slow start shows its step and effective
weight. failover and sticky with a key name the backend they choose; the
other methods list the candidates and the rule, because the choice depends
on the rotation of the process. --explain does not advance any rotation.
--json prints one JSON object (service, via, family, store,
balance, on_all_down, result, and with --explain also explain);
later releases only add fields. The text is not a stable interface.
--dns A and --dns AAAA (these imply --explain) explain what the DNS
server of the daemon answers to an A or an AAAA query: only the backends of
that family are looked at, the store counts as fresh, sticky has no key,
and weighted is the daemon’s smooth weighted round-robin. It reads the
store, which gets a new reason up to about one second after the daemon
decides it, so for that second it can differ from the live DNS answer; the
rotation counters are in the daemon’s memory and are not shown. SRV answers
are not explained.
| Exit status | Meaning |
|---|---|
| 0 | An address was printed (with --dns: the service has candidates for the family) |
| 1 | No backend can be chosen: ALL_DOWN, or no backend of the family with --dns |
| 2 | No such service, an invalid command line, or a config that cannot be read |
| 3 | The store cannot be used (UNAVAILABLE) or has a format this command does not read (INCOMPATIBLE), or the output cannot be written (a full disk; a closed pipe is not an error) |
--explain does not change the exit status, and a failed lookup still prints
what is known.
Threads, processes and fork
- Lookups are safe from many threads at once.
- The store is opened once per process and reused. A lookup holds an LMDB reader slot only while it reads. The store has 1024 slots shared by all processes on the host; a process killed in the middle of a lookup leaves its slot taken until the daemon frees it, about once a minute.
- After
fork, the child opens the store again on its first lookup; this works also when the parent looked up before forking (php-fpm, preforking servers), and when other threads of the parent were looking up at the moment of the fork. - On FreeBSD,
forkwaits until no lookup of the process is inside the store, and keeps new lookups out until it has returned in the parent and in the child; without that, a child forked while another thread looked up could wait for ever in its first lookup, on a lock that FreeBSD’s threading library leaves held in the child. The wait is as long as one read, and lookups wait the same way while a fork runs. A thread stopped (SIGSTOP) inside a lookup holds the fork up until it continues.libemforcannot be unloaded on FreeBSD (dlcloseleaves it mapped), because its fork handlers stay registered until the process ends; linklibemfor.astatically into a module that is unloaded, or depend on the Rust crate from your own cdylib, only with-Wl,-z,nodelete. - What the guard does not cover on FreeBSD: a
forkfrom a signal handler that interrupted a lookup of the same thread (it returns, but the child can hang in its first lookup, as without the guard), and other users of process-shared mutexes in the same process (another LMDB environment, say), which can still leave a child waiting in its own lookup. Do not fork from signal handlers; the library cannot fix the rest. - Round-robin counters are per process.
When the store file is replaced or grows
If the store file is replaced while applications run (the daemon moved a
damaged store aside, or someone deleted it and the daemon created a new
one), clients notice within about 100 ms and switch to the new file; no
application restart is needed. The same holds when the daemon is restarted
with a larger [store] map_size and the store grows past the old size.
A store file that exists but cannot be opened (another LMDB format, no
permission, not a store) is tried again at most once a second; lookups in
between fail with the same UNAVAILABLE error. A missing store file is
tried on every lookup, so applications started before the daemon find the
store as soon as it exists.
Do not delete the lock file (<store>-lock) alone. Start the daemon
before applications that run as root or as the daemon’s user: if the lock
file is missing, the first process to open the store creates it, owned by
that process’s user, and the daemon may then fail to start.
Same release as the daemon
The client library contains the daemon’s own LMDB build, and the store’s
lock file format depends on it. Install libemfor (and the wrappers) from
the same emfor release as the daemon, and upgrade them together. A
mismatch shows up as UNAVAILABLE with a message that says so.
Store format version
The daemon writes store format version 2, which adds the backend field
backup. Clients from this release read versions 1 and 2, so they work
with an older daemon too; older clients fail every lookup on a version 2
store with INCOMPATIBLE. When upgrading, upgrade the clients (and
emfor status) on a host before the daemon.
Performance
A lookup takes about 1–1.5 µs on one thread. Throughput grows with threads up to about four, then the shared reader lock of the store sets the pace (about two million lookups a second in total on Linux). See docs/performance.md for the measurements.