Embedded Rust

Use fenecdb as a Rust library inside your own program.

Open a file and query it

Open a file with fenec_core, parse a query with fenec_ql, and run it.

use fenec_core::prelude::*;
use fenec_ql::parse;

let mut db = fenec_core::fs::open("data.fenec")?;
for stmt in parse("get docs near embed $1 limit 5")? {
    let r = db.execute_with(&stmt, &[Value::Vector(embedding)])?;
}

What fenec-core depends on

fenec-core is a plain Rust library with no dependencies. Parsing is fenec-ql; everything else is the engine.

Reads and writes take different locks

Reads run side by side; writes run one at a time. Several statements run as one block land together, or none of them does.

let stmts: Vec<Statement> = parse("put orders {total: 12}; set stock {n: 4} where id = 7")?;
let block: Vec<(&Statement, &[Value])> = stmts.iter().map(|s| (s, &[][..])).collect();
match db.execute_block(&block) {
    Ok(answers) => { /* both landed, as one record */ }
    Err((at, e)) => { /* statement `at` failed: neither landed */ }
}

One process per file. Two processes opening the same file and writing it corrupt it. There is no lock file and no advisory locking — the single-writer rule is yours to keep.

How a block is begun, committed and put back

Database::query takes a shared lock and execute_with the exclusive one. Every write is a block: what a statement writes lands in the file as one record, or — a hook's refusal, a document the schema refuses half way through a put of many — none of it does, and the database is as it was. Several statements are one block with execute_block, or between begin and commit, with rollback to put them back; whoever opens one holds the exclusive lock until it ends. A create, a drop or a create index in it is put back with the rest: the collection made goes, the one dropped comes back, the index goes. A compact rewrites the file and runs on its own, outside a block. FenecQL has no BEGIN or COMMIT: over HTTP a /batch is one block (HTTP endpoint).

Compaction is the caller's too

An update leaves the version it replaced dead in the file until a compact. The server and the native library compact on their own; a program embedding the crate starts the same thread over its database, or asks from one of its own:

use fenec_core::engine::{compact_when_due, CompactPolicy, Compactor};

let db = Arc::new(RwLock::new(fenec_core::fs::open("app.fenec")?));
// Looks every 5 s; compacts beside the queries once half the file is dead
// and at least 64 MB, or sooner as the file nears the memory the system
// has for its pages. Stops when dropped.
let _compactor = Compactor::start(&db, CompactPolicy::default())?;
// Or, from a thread you already have:
compact_when_due(&db, &CompactPolicy::default());

Database::garbage says what the file holds against what is live, without reading a document.

Durability is the caller's job

Call sync yourself: until you do, recent writes can be lost in a crash. The library never calls it for you.

What sync and a storage error do

File writes are buffered. If the process crashes before sync is called, the last writes are lost. In embedded use nothing calls it for you — fenec-server has the --sync policy and a shutdown hook precisely because the library does not.

A storage error stops writes. Once the sink refuses an append, a sync or a rewrite, every later write — and every later sync — returns Error::Io until the file is reopened, and Database::failure() says what failed. A failed fsync is not retried: the kernel may already have dropped the pages, so a retry that succeeds would prove nothing. Reads go on, from memory.

The same applies to the HNSW graph. It is derived data and is written only during snapshot, compact and checkpoint, never on the write path. Without a graph in the file it is rebuilt on open: 3.4 s at 100 000 × 128, against 9 ms with it. A crash does not cost the graph: the next open restores the one the last checkpoint wrote and applies the writes since — 0.07 s for 1 000 of them at 100 000 × 128, where the rebuild took 3.4 s. A process that has to answer before those are linked opens with fs::open_serving: the vectors written since go in unlinked, near measures each of them until Database::link_pending(n) has linked them, and the process calls it under its write lock a few at a time, as fenec-server does. Database::unlinked() says how many are left.

Measuring the footprint

Database::memory_bytes() tells you, for free, how much memory the engine's data takes. The process as a whole uses more.

What memory_bytes counts and leaves out

Database::memory_bytes() sums the segment bytes, offset indexes, vector arenas, graph links and every other index from counters they keep as they change, so it costs nothing to ask. It is not RSS — allocator slack, HNSW build buffers, upper-level neighbour allocations and the 2–3× transient peaks all sit outside it. Measured ratios are in Limits.

Features

Two parts of the library are left out of a wasm build.

Feature	What it brings
`std-fs`	Buffered file I/O — `fenec_core::fs`. Off in wasm builds.
threads	The parallel HNSW build path, compiled out of wasm32 entirely.

Why now() errors on wasm32

now() errors on wasm32-unknown-unknown, which has no clock; time is passed in as a parameter there.

Errors

Every call returns the same error type, Error.

The Error type and its HTTP statuses

Error in fenec-core/src/error.rs is the single error type across the whole engine: allocation-free variants, no Box. fenec-server maps it onto HTTP statuses: 404 for a missing collection, 409 for one that exists or a taken id, 400 for a query or type error, 403 for a write refused, 500 for a storage error.

Plugins

Add your own functions and write hooks to the engine by installing a plugin.

use fenec_core::plugin::{Plugin, Registry, ScalarFn};

struct Shout;
impl ScalarFn for Shout {
    fn call(&self, args: &[Value]) -> Result<Value> {
        Ok(Value::Text(format!("{}!", args[0].as_text().unwrap_or(""))))
    }
}

struct MyPlugin;
impl Plugin for MyPlugin {
    fn name(&self) -> &str { "mine" }
    fn init(&self, reg: &mut Registry) -> Result<()> {
        reg.register_fn("shout", Arc::new(Shout))
    }
}

db.install_plugin(&MyPlugin)?;   // get t select shout(name)

How the plugin registry works

The registry takes after PostgreSQL's extension model. Scalar functions and write hooks in the shape of triggers can be registered; a plugin is installed once, and a second install of the same name is refused. fenec-server --follow guards its mirrored collection with such a hook.

Module map

Where each part of the engine lives in fenec-core.

The modules and what each holds

`store`	Segments and the offset index
`engine`	`Database`, `Collection`, replay, snapshot, compact, checkpoint
`vector`	HNSW and the distance kernels
`query`	`Statement` and plan execution
`schema` `value` `codec` `json`	Types, the binary codec, the JSON parser
`time`	Calendar arithmetic, integer only
`changes`	The change ring behind subscriptions
`plugin`	The registry
`fs`	Buffered file I/O, behind `std-fs`