Text search benchmarks

100 000 documents of 30 words each over a vocabulary of 5 000, the commoner words more common, a brand under @hash (40 values) and a color with no index (12). Each figure is the median of 15 runs over 200 two-word queries; a row's marks are the difference over the ten rows of match alone. The words a typo reaches are timed over FiQA's own words, 2 000 query words with a typo in each, and the sorted words over 100 000 documents of made-up words, many an edit or two from another, as a language's are (Typos and words begun, Highlights and snippets, Facets).

make search-bench
make fuzzy-bench BEIR=path/to/fiqa

One typo a query, by a generator seeded by the query's place (crates/fenec-bench/src/bin/beir.rs): in one word of four letters or more, a letter dropped, a key beside it on the keyboard typed as well or instead, or two letters swapped — the four kinds Damerau found four misspellings in five to be — and in the first letter 7% of the time, as Pollock and Zamora found it. SciFact's 300 test queries hold 297 such words, FiQA's 648 hold 644. Half typed is the last word cut to half its letters, rounded up. A word of under five letters takes no edit, which is most of what fuzzy leaves unrecovered; with one typo a query, fuzzy 1 does a little better than the two edits nine letters allow, which pay where a long word holds two. The p50s are the whole query's, the words found and their lists merged included.

cd crates/fenec-bench/beir && npm install
node embed.mjs path/to/scifact     # dense: all-MiniLM-L6-v2, 90 MB
node splade.mjs path/to/scifact    # sparse: SPLADE++, 532 MB, optional
make beir BEIR=path/to/scifact

Two BEIR datasets as BEIR ships them — SciFact, 5 183 documents and 300 test queries, and FiQA, 57 638 and 648 — with every document and query embedded by all-MiniLM-L6-v2 the way sentence-transformers does it, and encoded by SPLADE++ (prithivida/Splade_PP_en_v1) into a sparse<30522> field, both cut at 256 tokens with the closing [SEP] kept. transformers.js drops it when it cuts a text, so both scripts cut texts themselves: for SPLADE that was worth 0.46 of SciFact's nDCG@10, for the dense vectors up to 0.003. Each of the engine's ways of ranking ten documents is then scored by nDCG@10, through FenecQL: the answers are the engine's own.

The two datasets pull opposite ways. In SciFact a claim shares its words with the evidence and BM25 alone beats the transformer; in FiQA a question shares few words with its answer and BM25 is far behind. fuse is the one path near the top of both. The rank offset matters less than the depth: at 20 a side, k from 10 to 120 moves either score by at most 0.005.

SPLADE sits between the two. On SciFact it comes within 0.006 of the dense fusion with one index; on FiQA it lifts BM25 by 0.099 but stays under the dense vectors, and fused with BM25 there it loses 0.016 — both sides are lexical. Its queries' 37 to 65 terms reach most of the corpus, and its lists are summed a list at a time rather than walked a document at a time: 0.68 ms, where MaxScore's walk took 2.58 (How it works).