Tenants and sharding

When one machine is not enough, give each tenant a file of its own, spread the files over nodes, and put fenec-shard in front.

This page runs a small deployment on one machine: two nodes, a router in front of them, a tenant, and the loss of the node that holds it. Each command is shown with what it printed when it was run. Why a tenant is a file of its own is on Sharding: design; the guides at the end are the rest.

Get started

Start two nodes and a router, tell the router about the nodes, create a tenant, and use it through the router.

A node is fenec-server --dir, a server over a directory of tenant files, one for each tenant; the router is fenec-shard. The release archives hold the server but not the router, so build both from a clone (Build from source has what that needs).

Build both, in the clone.

cargo build --release -p fenec-server -p fenec-shard

Make an empty directory to run them from. The nodes and the router make their files in the directory they start in, so every terminal below starts in this one, with the programs on its PATH: do both again in each one you open.
```
mkdir sharding-tutorial && cd sharding-tutorial
export PATH="/path/to/the/clone/target/release:$PATH"
```
In a terminal of its own, start the first node. --dir is its directory of tenant files, --admin-token is what the router drives it with, --replication-token is the one the nodes share for copying tenants from one to another, and --sync always puts every write on disk before it is answered, so that a failover loses none.
terminal 1
```
fenec-server --dir n1 --http 127.0.0.1:8081 --admin-token a1 \
    --replication-token r1 --sync always
```
prints
```
serving tenants from: n1  (0 on disk)
each tenant may hold 25 streams and 25 held writes of the node's 100 connections
fenec-http 0.1.12 listening on: http://127.0.0.1:8081  [no auth]
```

In another, start the second node. --replica-of has it follow the first node's tenants.

terminal 2

fenec-server --dir n2 --http 127.0.0.1:8082 --admin-token a2 \
    --replication-token r1 --replica-of http://127.0.0.1:8081

prints

serving tenants from: n2  (0 on disk)
each tenant may hold 25 streams and 25 held writes of the node's 100 connections
following the tenants of: http://127.0.0.1:8081
fenec-http 0.1.12 listening on: http://127.0.0.1:8082  [no auth]

In a third, start the router. --directory is the file that records which node holds which tenant, and --token guards its /_shard/ routes.
terminal 3
```
fenec-shard --listen 127.0.0.1:8090 --directory shard.fenec --token root
```
prints
```
fenec-shard 0.1.12 listening on: http://127.0.0.1:8090  [0 node(s), 0 tenant(s)]
```
None of the three was given a token for data requests, which is why they answer one without. They listen on loopback; on any other address they ask for one (Tokens).

In a fourth terminal, in the same directory, tell the router about the nodes. The second goes first: the first names it as its standby, and the router refuses a standby it does not know (no node `n2` to be the standby).

curl -X PUT localhost:8090/_shard/nodes/n2 -H 'Authorization: Bearer root' \
     -d '{"addr":"127.0.0.1:8082","token":"a2"}'

prints

{"node":"n2","unreplicated":[]}

curl -X PUT localhost:8090/_shard/nodes/n1 -H 'Authorization: Bearer root' \
     -d '{"addr":"127.0.0.1:8081","token":"a1","standby":"n2"}'

prints

{"node":"n1","unreplicated":[]}

Create a tenant. The router creates it on n1, since nothing is placed on a standby, and makes the file n2 follows it into.
```
curl -X PUT localhost:8090/_shard/tenants/acme -H 'Authorization: Bearer root'
```
prints
```
{"tenant":"acme","node":"n1"}
```

Use it through the router. The path names the tenant, and what follows it is the HTTP endpoint of a database of its own.

curl -X POST localhost:8090/t/acme/query -d '{"query":"create collection notes (title text)"}'

prints

{"message":"collection `notes` created"}

curl -X POST localhost:8090/t/acme/notes -d '{"title":"hello"}'

prints

{"inserted":1}

curl localhost:8090/t/acme/notes

prints

[{"id":1,"title":"hello"}]

A tenant is a file, and the standby has its copy of it. The router knows where the tenant lives and which node is its standby.

ls n1 n2
curl localhost:8090/_shard/tenants/acme -H 'Authorization: Bearer root'

prints

n1:
acme.fenec

n2:
acme.fenec
{"name":"acme","node":"n1","state":"active","standby":"n2"}

Same API. Under /t/<tenant> everything works as on a single database; a client only changes its base URL.

What splitting by tenant changes for a client

A single fenecdb is bounded by one machine's memory and one writer. Past that, the data is split by tenant: each tenant is a file of its own, a node holds many of them, and fenec-shard knows which node holds which.

Below /t/<tenant> the surface is exactly the HTTP endpoint of a single database (HTTP reference) — routes, /query, /batch, subscriptions. A client changes its base URL and nothing else; the sync layer included:

const db = await sync({ url: 'http://127.0.0.1:8090/t/acme', shapes: [...] });

Lose a node

Stop the node that holds acme, and see what the router does and what it leaves to you.

Write one more row, then stop n1 with Ctrl-C in its terminal. A node syncs the tenants it has open as it shuts down.
```
curl -X POST localhost:8090/t/acme/notes -d '{"title":"second"}'
```
prints
```
{"inserted":1}
```
terminal 1 prints
```
shutting down: 1 open tenant(s) synced
```
The second node, which follows the first, notices and goes on trying to reach it, a line each time, for as long as it is down. (The number after os error is the system's: 61 is macOS's for a refused connection, 111 Linux's.)
terminal 2 prints
```
replication: failed to fill whole buffer; connecting again
replication: Connection refused (os error 61); connecting again
```
Write again. The router has no node to send it to, and does not guess where the tenant went: it cannot tell a node that is gone from one it cannot reach, and a guess would make two primaries for one tenant. The answer is a 502.
```
curl -X POST localhost:8090/t/acme/notes -d '{"title":"third"}'
```
prints
```
{"error":"node `n1` (127.0.0.1:8081) did not answer: Connection refused (os error 61)"}
```
Fail the node over. Without --auto-failover, someone who knows the node is gone says so, and the router promotes its tenants on the standby, one at a time, and points the directory at it. The answer lists what was promoted and what was not. (Automatic failover is the router deciding it on a lease.)
```
curl -X POST localhost:8090/_shard/nodes/n1/failover -H 'Authorization: Bearer root'
```
prints
```
{"node":"n1","to":"n2","promoted":["acme"],"failed":[]}
```

Read, write the row that was refused, and read again. Both rows written before n1 stopped are there, and the tenant takes writes.

curl localhost:8090/t/acme/notes

prints

[{"id":1,"title":"hello"},{"id":2,"title":"second"}]

curl -X POST localhost:8090/t/acme/notes -d '{"title":"third"}'

prints

{"inserted":1}

curl localhost:8090/t/acme/notes

prints

[{"id":1,"title":"hello"},{"id":2,"title":"second"},{"id":3,"title":"third"}]

Ask where the tenant lives. It is on n2, with no standby: a failover that moved every tenant ends the pair.

curl localhost:8090/_shard/tenants/acme -H 'Authorization: Bearer root'

prints

{"name":"acme","node":"n2","state":"active"}

n1 can come back as n2's standby, or each tenant can have a replica of its own on another node: both are under Replicas, standbys and failover.

Where next

Working with tenants
A tenant with its schema, quotas, moving one to another node, archiving a node, and a query across tenants.
Replicas, standbys and failover
A replica of each tenant's own, failing a node over on a lease, and a standby router.
Regions and zones
Nodes in regions and zones, and a region that keeps its own directory.
The global entry
One URL for every region, with a Cloudflare Worker.
Read replicas
Reads answered near the client while writes go home.
Reference
The routes of a node and of the router, quotas and refusals, tokens, and the deploy kit.
Design
Why a tenant is a file, how the router forwards, how a lease keeps a tenant to one primary, and what it costs.