Import

The import derives the schema, the table's integer key becoming each document's id, writes the rows in batches and builds the indexes after the last, rather than keep them up a row at a time. The collection goes into <into>.fenec unless --file names another. A column of embeddings comes across too (Vector columns). Every option is in fenec import --help.

--count is a separate flag because counting means a full scan: SQLite keeps the row count nowhere and count(*) in PostgreSQL reads the table from the start. Silently reading everything twice would be a bad default.

SQLite. The file format is read directly — neither the sqlite3 library nor the binary is needed. INTEGER PRIMARY KEY becomes the document id, --vector field:N turns little-endian f32 BLOBs into vector<N>, and untyped columns are resolved from the first 1000 rows.

PostgreSQL. It connects to the live server and streams COPY ... TO STDOUT — no psql, no pg_dump. Authentication is SCRAM-SHA-256, hand-written in fenec-wire's client over fenec-http's own crypto. pgvector's vector, halfvec and sparsevec columns are recognised along with their dimensions.

Types without an equivalent are not silently rounded: say what you want with --cast <field>=<type>. On PostgreSQL, --cast price=text preserves a decimal exactly. In SQLite a decimal is already stored as an int or a float at the source, so the exact value is gone before the import starts.

--where is a FenecQL expression and behaves identically on both sources: it is applied after the row becomes a document and before it is written. The field names are the target schema's — the same expression you would later write with get.

On PostgreSQL there is also --source-where, which goes into the SELECT inside the COPY so non-matching rows never travel over the wire. That one is SQL, not FenecQL. They are separate flags on purpose: one is a portable expression, the other a win pushed down to the source. Both can be used together.

fenec import postgres://... --table docs --into articles \
    --source-where "year >= 2024" --where 'tags has "rust"'

With --follow the import does not end at the copy: it keeps applying the table's inserts, updates, deletes and truncates as they commit, until it is interrupted. Run the same command again and it picks up where it stopped, without a second copy.

It reads a logical replication slot through PostgreSQL's own pgoutput, so nothing is installed on the server. The publication and the slot are made when missing (--publication, --slot; both default to fenec_<into>). The server needs wal_level = logical, and the table needs an integer key that becomes the document id, since a changed row has to be found again. It must also be in the replica identity: the primary key, or replica identity full.

• A row arrives as the copy would have written it. The stream goes through the same type mapping, --cast, --vector and --where as the copy. A row that stops passing --where is deleted from the collection.
• Nothing is confirmed before it is on disk. The slot only moves past a transaction once an fsync has covered it. A broken connection or a killed follower starts again from there, and the writes it replays are puts and deletes by id, so applying them twice changes nothing.
• An untouched vector is not resent. A vector of 768 floats is 3 KB, past PostgreSQL's TOAST threshold, so an update of any other column arrives without it; the follower takes it from the collection.
• A copy is resumed only when it is whole. A small _follow collection in the file records which collection follows which slot and whether its copy finished. A copy cut short, or one whose slot was dropped in the meantime, is made again rather than streamed on top of, and a collection the follower did not copy is never replaced.

Measured against the Makefile's server, 10 000 rows with a vector(384), the mirror a file (make follow-bench):

The follower is the file's only writer, as every fenecdb file has exactly one: stop it to open the mirror elsewhere, or serve the mirror from the server that follows (below). A column added to the table after the copy is reported and left out. A column the collection needs that is dropped from the table stops the follower. --source-where and --limit do not combine with it: the first narrows the copy but not the stream, and a mirror keeps every row.

A second process over the follower's file corrupts it, so serving the mirror took a copy of it and a process of its own. fenec-server --follow runs the follower inside the server instead, over the file the server serves: the table is served over HTTP and subscriptions as it follows, with no second copy.

Every option of fenec import that says how a row becomes a document — --index, --vector, --cast, --id, --where, --batch — is taken as --follow-index and so on, as are --follow-slot and --follow-publication. The collection takes no write but the follower's: a client's is refused with 403, since the next change from PostgreSQL would write over it. The rest of the file takes writes as ever. A subscriber of the collection (GET /articles/changes) hears each change as the follower applies it, and so does a replica of the server.

The follower stops with the server, its last changes on disk and confirmed, before the shutdown's checkpoint. A server killed outright starts the stream again from what it last confirmed; the changes it applies again are puts and deletes by id, so nothing is lost and nothing comes twice. An error the follower cannot wait out ends the server rather than leave it answering from a mirror that no longer moves.

Against the Makefile's server, 10 000 rows with a vector(384) under an HNSW index, a subscriber reading /changes over HTTP (make mirror-bench):

Commit to visible in the file, with no server between (make follow-bench), was 0.16 ms at the median on the same machine. Killed while the table took a row a transaction, the server was up again in 75 ms and held every one of the table's 14 063 rows 370 ms after it started. The follower adds 178 KB to fenec-server.

The graph is built after the copy, over every vector at once and on every core, which is the order a bulk load wants (Bulk load first, index after). pgvector's vector, halfvec and sparsevec columns become vector<N>, vector<N, f16> and sparse<N> with nothing named; --follow keeps them in step as any column, an untouched vector taken from the collection rather than resent (Following changes).

Import is a default feature of the fenec binary and accounts for 275 KB of it. --no-default-features removes it, and fenec backup, archive, restore and fenec mcp with it (502 KB); with the cli profile as well, the binary drops from 3013 KB to 1880 KB (make small).