How it works

Classic databases cut the disk into pages and copy those pages into a buffer pool in user space. The eviction policy, dirty-page tracking, checkpoints and most of the locking cost all follow from that one structure.

fenecdb skips the layer:

• Segments are immutable. A write only appends to the active one, which is sealed at 8 MiB.
• The byte sequence on disk and the byte sequence in memory are the same format, so no stage copies pages into a cache of its own — a read decodes directly over the mapped bytes.
• The only helper structure is the HashMap<DocId, Loc> offset index.
• Projection and filters decode a single field; the fields in front of it are skipped without allocating.

No eviction policy, no dirty pages, no buffer pool to warm. The system's page cache is cold after a restart, as for any file, and a page's first read comes from the disk. Dead bytes are reclaimed by compact, which fenec-server and the native library run on their own (Compaction): left to grow, a file of updated rows spreads its live records thin over pages of dead ones, and reads wait for the disk. What this means for a cache in front of the server, with the measurements and their conditions, is on Instead of Redis.

Because the format on disk is the format in memory, the file can be the memory: fs::open maps it where the target maps files, and a document is decoded from the pages the operating system brings in — and can drop again. What the process holds is what it derived from the documents: the offset index, the hash, ordered and text indexes, the graph, and the documents written since the open until they amount to 16 MB.

A 10 GB file of 23 million rows opens on an 8 GB machine this way, which reading it cannot: that would want about 20 GB. The trade is the one every mapped file makes: a page the disk cannot read ends the process rather than the query, and a file copied over in place while it is open does too — replace one by rename, as fenecdb's own rewrites do. What it does not change is the open's cost — the records are still walked to build the offset index and the indexes are still built from the documents, so opening is proportional to the file either way.

A rewrite writes the new file and points the stores at it, so the file just written is the one being read and the old one is let go of. The rewrite knows where it put each record, so the stores work their new places out without reading the new file back: that read cost 1.8 to 2.3 s of a 1 GB checkpoint, under the write lock, where working the places out takes 19 ms. A compact over a mapped file therefore copies no record into memory, and rebuilds no index but a graph holding tombstones: the documents are the same ones, and every index is keyed by id. On a server it runs beside the database: the new file is written with no lock held, and the writers wait only while the documents written meanwhile are added to it — 24 ms at most over a 1.1 GB file, where writing it under the lock held them 1.22 s.

What is written after the open reaches the file as each block lands — a /batch, a statement — and the stores hold those documents in memory only until they amount to 16 MB: then they read them from the file, where the records put them, and let the memory go. A block that has not landed yet spills into the file at 16 MB the same way, in a record a load takes only once the block's land names it, so a load of any size holds no more; one of 250 000 768-dim rows peaked at 904 MB where it had at 2 389, of which 820 are its graph. Held until a restart before, 250 000 768-dim rows loaded into a server were held twice, in the file and in memory: the engine counted 1 592 MB after the load, 820 now and 818 for the same file started again. Nothing of the file is read or written for it — where each record went is noted as it lands, and the file is mapped with room past its end, which the appends grow into — and the pause is 1 ms for 16 MB of 768-dim documents, 2 ms of 128-dim ones. Whether the memory goes back to the system is the allocator's call: Linux's give a large block back when it is freed, so the process holds what the engine counts, while macOS's keeps hundreds of megabytes of them in a cache of its own.

fenec-server --no-mmap reads the file into memory instead — a replicated one, and every tenant of a --dir node, as well — over a network file system, where mapping is a bad bet, or to have --max-memory cover the data as well as the indexes.

• HNSW is written in-house. Every vector lives in one contiguous arena.
• Under the cosine metric vectors are normalised inside the index, so the query reduces to a dot product.
• Neighbour selection uses the diversity heuristic (Malkov & Yashunin, Algorithm 4). Taking only the m nearest candidates trapped the graph in local clusters and lowered recall.
• The read-only part of the build — descent, beam search, neighbour selection — runs in parallel over a batch; only writing the links is serial. The result is independent of thread count and deterministic.
• The graph is written to the file, but only the links, not the vectors: the vectors are already in the document records, and the expensive part is the build.

The filter's rows are found first, but only as far as the plan needs them. An index hands them over directly — a @hash equality, a @sorted range. Otherwise they are probed: rows are tested in blocks spread across the collection until more of them match than an ANN walk would measure anyway (ef × m0, 3 200 at the defaults), or until none are left. Then one of two paths runs:

• if the set stayed smaller than that, it is searched directly — cheaper, and exact;
• otherwise the ANN runs and each candidate is tested against the filter. If the result lands under the limit, the probe carries on to the end and the whole set is searched exactly.

The probe changes when the rows are found, never the answer: it is the one finding the whole set first gave, row for row. What it saves is the scan. Over 200 000 × 128, year >= 2020, 23% of the rows, took 16.4 ms with the set found first, 0.14 ms of it the ANN, and takes 1.1 ms probed. A filter under the budget is still read in full unless an index narrows it: 17.9 ms for 1% of the rows, 0.16 ms over @sorted. The blocks are spread out so that a filter for the newest rows, which sit at the end of the id order, is found as fast as one for rows everywhere: 1.7 ms for the newest 23%.

Deleted documents do the same without a filter. Every write that changes or deletes a document's vector leaves a tombstone in the graph until compact, and a beam full of them holds fewer live rows than the page. A short answer is walked again with the beam wider by the tombstones, since no more than all of them can be in it — or, where a walk that wide would cost more than reading every vector, searched exactly. Either way the page is full, and explain says which ran.

Precision is part of the type, not of the index — the same choice pgvector makes with halfvec. That shrinks the record and the arena together; as an index option only the arena would shrink, and in the browser the file image is in memory too.

The runtime representation does not change: values are f32 everywhere and conversion happens only at the storage boundary.

The cost is arithmetic: distance over an f16 arena unpacks inside the loop. Two things were measured and fixed. The expansion has to be branchless — the first version called the exact conversion in codec, whose subnormal branch killed auto-vectorisation and made the build five times slower (52 s); the branchless version is one multiply and is bit-identical over all 65 536 values. And the fixed side of the diversity loop is unpacked once rather than per comparison: 22.6 s down to 12.4 s.

The win is at the memory boundary. One million 768-dimension vectors take about 3 GB as f32 and about 1.5 GB as f16. In WASM32's 4 GB address space that is the difference between fitting and not. Embedding model output sits comfortably inside the f16 range (±65 504, roughly three decimal digits) and the measured recall loss is 0.3 points.

Half precision shrinks the record and the arena together. A quantized index leaves the record alone and shrinks the arena further, by holding codes instead of vectors.

A code only estimates a distance, so it is used for what an estimate is good for: walking the graph to the right neighbourhood. The candidates the beam holds, ef of them, are then put in order by the documents' own vectors, read out of the store as rerank reads them, and the first limit are the answer. The scores are exact; the codes decide only which candidates are found.

Not every candidate has to be read. An int8 code rounds each component to a step of its scale, so it knows how far off its estimate can be: the candidates are read nearest estimate first, and one that can no longer make the page is left unread. A bit code keeps no step, and the whole beam is read. A filtered set too small for the walk to pay — fewer rows than the distances a walk would measure, ef × 2m — is ranked by its codes and read the same way.

A bit code holds the signs of how far a vector lies from its centre, not the signs of the vector. Vectors crowd around the few directions they are about, so most of a vector's own signs are its neighbours' too, and the signs could not tell the neighbours apart. The index learns 128 centres by k-means from its first 2 048 vectors, which it holds whole until then — a smaller index searches exactly — and keeps with each vector the centre it is nearest and two numbers that scale its signs into an estimate. A search takes its query's product with each centre once, and only the query's part away from the centre goes through the signs. The query goes through them cut to six bits a component, a bit plane each, so a code's signs meet it in popcounts: 72 over 768 dimensions where a float was added a component, which took a query at 100 000 × 768 from 0.51 ms to 0.29 at a beam of 200 with the same recall. The centres are kept in the file with the graph, so a graph restored codes its vectors as they were coded.

One million vectors of 768 dimensions, stored as f16, in 64 clusters each spread along 32 directions — embeddings vary along far fewer directions than they have — with a @hash field of 333 groups for the filtered rows, on an 8 GB machine that was swapping, so the latencies are high for all three (make quant-bench):

The bit column was measured again once the query met the codes in popcounts, in a run of bit codes alone: in that run the query whole took 0.80, 1.30 and 2.16 ms at the three beams, and 1.36 ms filtered.

int8 codes follow the vectors closely enough for the default beam of 100, and closely enough that 20 of its hundred candidates are read, with the recall of reading them all. What is left unread is further off than six times the spread of its rounding, which a candidate passes less than 1.5e-8 of the time. Bit codes, taken from the centres, find as many of the true ten at the same beam, but bound nothing, so every candidate is read; a quant=bit index searches 200 unless told otherwise. They are for the arena, a seventh of int8's, and walk the graph fastest of the three. The rest of the heap is the documents, whose vectors the answers are ordered by.

The kernels over codes are scalar on every target, in the eight-strip order of the vector kernels: natively LLVM vectorises them, and in the browser they add in the same order, so a graph over codes is the same graph in both. A bit code's popcounts are whole numbers, exact everywhere, and the few floats they are scaled by are taken in one order.

A learned sparse model weighs a text's vocabulary: SPLADE gives each of BERT's 30 522 word pieces a weight, and all but a couple of hundred come out zero. It learns which words a text is about, including ones it never uses — an abstract on damage to cerebral white matter is weighted for disability as well as the disabilities it says. sparse<N> holds such a vector by its non-zero entries, and @inverted indexes it the way @text indexes words: a list per dimension of the documents with a weight there.

near over it scores by dot product and sums the query's lists a list at a time — or walks them with MaxScore, the pruning match uses, where they hold few documents in a wide span of ids — so what comes back is the exact top ten rather than an estimate of it; only documents sharing a dimension with the query are ranked.

On SciFact, where a claim shares its words with the evidence, SPLADE alone scores what BM25 and the dense vectors do fused, with one index. On FiQA, where a question shares few words with its answer, it lifts BM25 from 0.232 to 0.331 but stays under the dense vectors, and fusing it with BM25 there costs it: the two are both lexical, and the weaker one drags. The index holds 968 928 postings for SciFact's 5 183 documents — 186.9 a document — in 13.1 MB, built in 0.07 s; FiQA's 57 638 hold 8.8 million in 107 MB, built in 1.2 s. Like the text index it is derived from the documents, built by the first query that reads it after an open, and never written to the file.

Writing the index during the load makes every insert a graph mutation. Building it afterwards turns the write path into a pure append and builds the graph in one parallel pass.

This is already the recommended order for SQLite and PostgreSQL. In fenecdb create index exists for exactly this reason, and the comparison in Vector search benchmarks uses that order for all three engines.

The core uses std and nothing else: its own binary codec, its own JSON, its own HNSW, its own lexer and parser, its own SCRAM, even its own decimal-to-f64 — str::parse reaches for a 12 KB table of powers of five, which is a twelfth of the module and incompressible. The reason is the WebAssembly output — it has to stay small and auditable.

The rule covers fenec-core, fenec-ql, fenec-wasm, fenec-http, fenec-server, fenec-wire, fenec-import and fenec-shard. fenec-core does dev-depend on fenec-ql, which Cargo permits through a dev dependency, so tests can be written as real queries.

What it costs. Every bug is ours, and so is every protocol detail: the HTTP server, the JSON reader, the JWT checks and the PostgreSQL client the importer reads with are all hand-written.

fenec-wasm exposes a C ABI and there is no wasm-bindgen. The glue is about 605 lines of JavaScript inside web/fenec.js, alongside persistence and the sync layer; the query builder and the HTTP client are dependency-free modules of their own it imports, which @fenecdb/web/client loads without the rest (Which package).

persist(db) and restore(db) move the database in and out of IndexedDB the way a file holds it: an image, and the writes since as chunks after it. The first persist stores the image; every later one stores only what was written since the last, from a journal the wasm side keeps once asked to. In Chrome 153, over a 32 MB database, persisting one new row takes 0.18 ms where the image took 92 ms; in Safari 26.1, 0.98 ms and 418 ms. Once the chunks outgrow half the image, a new image replaces them, and compact replaces them too. restore reads the image and its chunks back as one file, so the graph restored from the image takes the chunks' writes rather than being rebuilt.

In a worker, openFile(db, name) keeps it in a file of the origin private file system instead, which is the file itself: the bytes fenec-server keeps, so fenec opens a page's file and a page loads a server's. The same journal feeds it, a statement at a time: run appends the writes and flushes the file before it answers, 0.56 ms a row in Chrome and 0.36 ms in Safari, statement included, and the file opens again in half the time the chunks are restored in. A new image is written beside the file and flushed before it is written over it, so a crash leaves one of the two whole.

Reads take a shared lock (Database::query), writes take the exclusive one (execute_with). A /batch is a block: it takes the exclusive lock and holds it until it lands or is put back. One writer is what makes that isolation: nothing reads a block's writes before they land, because nothing else runs. There are no interactive transactions held open across requests: one writer would wait on a client's think time.

A long read holds the shared lock only as long as it takes to pin what it reads: on a server, an aggregate, a facet, an order, a filtered count or a read with no limit over a collection of 10 000 rows or more, and a read-only /batch holding one (Database::pin embedded). Under the lock it copies each collection's id index, 12 bytes a row, and shares its stored documents; the read then runs with no lock held, on the database as it stood at the pin, while writes land beside it. A read an index answers — an equality on a @hash field, a range on a @sorted one, match, near, lookup — is short, and runs under the lock as before: indexes change in place with every write, and copying one would cost as much as the read.

Every write is a block. What a statement writes — every document of a put of many, of a set or a del matching many — is held back from the file and the change feed until it ends, and lands as one record: a crash cuts all of it off or none, a replica applies it at once, and a subscriber is woken once for all of it. A statement that fails half way puts back what it did: each store is cut back to where it stood, and each document it wrote goes out of the indexes, the version before it back in. A /batch and the browser module's run of several statements are one block the same way. A create, a drop and a create index are put back with the rest — the collection made goes, the one dropped comes back, the index goes — and only a compact, which rewrites the file, runs on its own. Held back, a write costs within 2% of what it did: a lone put 841 ns in memory against 829 before blocks.

That is what removes MVCC row headers, the WAL and the visibility map — about 2.8 MB of headers over 100 000 rows. Most of the 2.4× between this file and the same data under pgvector is elsewhere: pgvector's HNSW index keeps its own copy of every vector, while this graph stores only the links (Vector search benchmarks).

Concurrency still exists on the read side. HNSW search buffers are kept thread-locally so the same index can be searched in parallel; eight concurrent full scans on eight cores measured a 4.3× speed-up.

Because the same engine runs in the browser and on the server, the local-copy architecture needed no new layer — only two things: the server being able to say "these changed", and the client being able to choose where to bind a query. bind() already existed, and the first was sitting in the file format, because the file is already a change log.

It is TanStack DB's model with two differences. There is no incremental dataflow: there a collection is a Map, and re-running a filter on every keystroke over a large set is unacceptable — here the local side is indexed and the query is already under a millisecond. And a shape is mandatory: the whole database is in memory, so the client cannot pull all of it. Details in Sync.

Dependencies point one way. Nothing points back up.

fenec-core   storage, indexes, plan executor, plugins, JSON, calendar, changes
     |       std only, zero dependencies
fenec-ql     FenecQL lexer and parser              fenec-wasm   browser ABI
     |                                                          (no wasm-bindgen)
     |
fenec-http   REST/JSON surface, SSE subscriptions, replication, tenants
     |
fenec-wire   the PostgreSQL client the importer reads with
     |
fenec-import SQLite file reader, PostgreSQL COPY source, --follow
     |
fenec-server the server: HTTP, replication, tenants, the --follow mirror
     |
fenec-cli    the `fenec` shell

fenec-bench is a measurement harness and the only crate allowed external dependencies — that is where rusqlite and postgres live, so the comparisons in Benchmarks can run against the real engines.