Sharding reference

--dir serves a directory of <tenant>.fenec files instead of one file. It is still one process per directory — the single-writer rule is per file, and one process owning all of them is what keeps it. --http is required. The tenant is looked up for every request — so one created a moment ago is there, one closed as idle reopens, one deleted meanwhile answers 404, and one frozen for a move refuses writes while its reads go on.

• Opened on first use. Open cost is proportional to the file, so a node does not spend its start-up on tenants nobody asks for. A tenant that has not been created is 404 — a request never brings one into being, or a typo would make an empty database.
• Closed when idle. --idle-close <s> (default 300) syncs, checkpoints when there is a vector index, and closes a tenant nothing has touched for that long. One with an open request or subscription is never closed.
• --max-memory covers the open tenants together. Opening one more over the ceiling first closes idle tenants, least recently used first. With everything busy the open goes ahead anyway: refusing a read because other tenants are busy would turn a memory warning into an outage. A write that grows a tenant is refused once that tenant alone is past the ceiling, over either listener.
• Shutdown syncs every open tenant and checkpoints the ones with a graph, holding their write locks until exit.

The router drives the node through /_admin/, which answers only to --admin-token and is off (404) without it — a client allowed to read and write a tenant must not be able to delete it or take another tenant's image:

An image is replayed in memory before it is written, so a truncated transfer is refused rather than installed, and the file appears by rename, never half-written. Tenant names are [a-z0-9_-], 1 to 64 characters: they become file names, and that set means the same thing on every file system.

Its only state is the directory: two collections in a fenecdb file of its own, nodes and tenants. Every change is written and synced before it takes effect, and a tenant's node and state change in one statement, so there is no moment on disk where a tenant belongs to both nodes or to neither. The node admin tokens stay in it; GET /_shard/nodes does not return them.

Placement goes by disk rather than memory: memory counts only the open tenants, and which ones are open depends on who asked last.

A page on another origin — an app, the sync layer, FenecHttp.live, the studio served by a node — reaches its tenants through the router once the router has --http-cors <origin>, read as fenec-server reads it (Security): * or one origin. The router answers a browser's preflight itself, before it looks the tenant up, and puts the CORS headers on every answer it gives. Its own — an unknown tenant's 404, a standby's 503, an unreachable node's 502, the 503 past --max-connections or at its thread limit (marked Fenec-Unserved: the request never ran, so the global entry and a relaying router ask the region's next router), the 401 of /_shard/ without its token — reach the page as the refusals they are; without the headers a browser hands the page a network error for each, and the sync layer backs off from what was an answer. On a node's answer the router's headers take the place of the node's, so the page is told the rule the router's preflight told it, and never two Access-Control-Allow-Origins, which a browser refuses whole. The 503s carry their wait as Retry-After and as retry_after in the body, as a quota's refusal does. Without the flag the router sends none of its own and passes a node's through as the node wrote them, and a preflight goes to the node.

Three, each for a different job. Data requests carry whatever the nodes' --http-token expects, and the router passes Authorization through untouched. Each node has its own --admin-token, which only the router holds. The router's --token guards /_shard/; like the nodes, it refuses to listen on a non-loopback address without one unless given --insecure.

With --jwt-secret or --jwt-keys a node also takes JSON Web Tokens held to its --policy, and the policy is the node's, not a tenant's: notes read,write where owner = $jwt.sub matches alice's rows in every tenant's file alike. So on a --dir node a token reaches /t/<t>/ only when its tenant claim names <t>, a text or a list holding it, and is refused with 403 elsewhere: on every route under the prefix, the statements, REST, /batch, a subscription, /_changes, /_schema and the statement counts. The check comes before the tenant is looked up, so a refused token learns nothing of which tenants exist. A token without the tenant claim is refused on a --dir node unless it is started with --jwt-unbound-tenants, which takes such a token for every tenant it serves; one holding the claim as null, "", a number or a list of no tenant's name is refused even then, since its issuer meant to bind it; --jwt-tenant-claim <name> reads another claim than tenant. The node's own --http-token and --admin-token are not bound. The router forwards Authorization as it came and holds no keys, so the node is what enforces it, behind the router or reached directly. What a scoped token writes to a tenant is checked against its rules as on a single file (Scoped access).

Every token is also read from the environment, which keeps it off a command line every user of the machine reads with ps: a node's from FENEC_HTTP_TOKEN, FENEC_ADMIN_TOKEN and FENEC_REPLICATION_TOKEN, a router's from FENEC_SHARD_TOKEN, FENEC_SHARD_REPLICATION_TOKEN and FENEC_SHARD_ENTRY_TOKEN, an archiver's from FENEC_REPLICATION_TOKEN. A variable set and empty, as an environment file leaves a line it does not use, is no token; a token given empty as a flag is refused at the start. A request carrying no Authorization reads as the empty token, so an empty FENEC_SHARD_TOKEN, taken for one, let anyone into /_shard/ -- and past the check that keeps a router with no token off a public address. The kit's systemd units hand every one over from an environment file (Deploying it).

make deploy-check boots systemd in two privileged containers, installs every unit and environment file as the runbook installs them, and runs them: systemd-analyze verify on each, a region, two nodes and a tenant made through the global router, rows written through the router and its standby, no token on any process's command line or in an environment another role's user can read, the limits and the sandbox in force (systemd-analyze security puts each unit at 1.2), a node's stop syncing within its timeout, a sealed archive verified with the key the unit hands its archiver and copied to a bucket by real rclone while its tenant is written, a real node_exporter scraped for the copy's success, a new disk's archive refused over the bucket's and copied once the bucket is copied back, and a standby router promoted and its unit switched as the runbook does it. cloudflared's own matcher is asked where the tunnel sends 28 paths, every spelling of a tenant's replication stream among them. promtool checks the alert rules and runs their tests, and a test of fenec-shard holds every metric and label value they read to a real scrape of every kind of process.

make drill runs each process in a container of its own on one Docker network -- two regions, each a router keeping its own directory and that router's standby, two nodes in two zones under --lease and --replicas, an archiver for each node, and the global router and its standby -- every token in the processes' environments and no port fixed. A writer a tenant puts rows by id, the next only once the one before was answered, while the drill cuts the global routers off the network (each region writes on), cuts the node holding the most primaries off the network and kills it, as a machine lost, and starts it again, kills a region's router and promotes its standby as an operator would, and marks a moment it later restores a tenant to from its node's archive. Then every acknowledged write is read back from each tenant's primary and its replica, and the times are printed. The router fails a killed node over a lease and a tenth after its last renewal and the promotions after: 1.92, 2.56 and 1.90 s after the kill in three runs at a lease of 2 s. Its tenants' writers write again once the request each had in flight to the dead node is let go of, at the router's upstream timeout -- 4 s in the drill, twice its lease, as the kit's routers run at twice theirs: 3.90, 3.97 and 3.93 s. The tenants whose replica it held went 17, 18 and 17 ms between two answered writes at most, the rest 16, 18 and 17; a reviewer's runs on another machine saw 28-30 ms, so these are what three runs saw, not bounds. The runbook says what each alert means, the order to bring a deployment up in, how to rotate each token and upgrade with no downtime, and what is not measured yet.