Semantic vector search

Meaning-aware retrieval on top of the full-text index. The engine embeds text with a real bge-small model (via the dl-embed C++ tool), retrieves candidates with multi-index hashing (MIH) over ITQ bit-codes, and re-ranks by exact int8 cosine — all stored in-relation and snapshot-versioned like everything else.

Quickstart

The vector tier is opt-in and adds a vendored ggml submodule (v0.20.2) plus the model:

# One-time: materialize ggml + the bge-small model (git-lfs tracked).
$ git submodule update --init vendor/ggml
$ git lfs pull             # models/bge-small-en-v1.5-f16.gguf (67 MB)
$ make dl-embed           # build ./dl-embed (needs cmake)
$ ./dl-embed self-test   # golden-embedding gate (cosine >= 0.9999 vs reference)

# Embed the corpus into the vector index (18 relations + one publish).
$ ./dl-embed pipeline --db /tmp/db

# Query it.
$ ./dl -d /tmp/db vsearch 'affordable GPU rental' --k 10
$ ./dl -d /tmp/db vhybrid 'gpu rental' 'affordable GPU rental' --k 10

How it works

Each entity is embedded and binarized to a 256-bit ITQ signature. The signature is split into m=16 bands; for band j the store keeps a fixed-arity relation __sig{j}__(band_u32, entity_sym_id) — a postings index from bit-substring to entity sym-id, the same shape as the full-text postings.

__sig0__..__sig15__   (arity 2)  band_value -> entity_sym_id    MIH candidate postings
__vec_q__            (arity 3)  entity_sym_id, chunk_idx, packed_4x_int8_u32   re-rank vectors
__itq_basis__        (arity 3)  dim_i, dim_j, float32_bits_u32   ITQ encode matrix (pinned)

A query embeds the text with the same model, ITQ-encodes it, then for each band enumerates the query’s substring variants within the pigeonhole budget ⌊r/m⌋, probes each via dl_prefix over __sig{j}__, unions the candidates across bands, filters to live entities, and re-ranks by exact integer int8 cosine. All reads are snapshot-versioned like any other relation, so dl_vector_search_version and dl search --version give you time-travelling semantic search for free.

The CLI

vsearch

Semantic vector search over the MIH postings + int8 re-rank.

dl [-d <dir>] vsearch '<query>' [--k N] [--radius R] [--version V]
                                  [--sig <hex64>] [--ivec <hex768>]

--version V queries as-of a published snapshot (0 = live). For a programmatic path with no model, pass the pre-encoded query as --sig (8 u32 = 64 hex) and --ivec (96 u32 = 768 hex).

vhybrid

Lexical ∩ semantic hybrid: intersect search results with vsearch candidates, then re-rank the intersection.

dl [-d <dir>] vhybrid '<terms>' '<query>' [--k N] [--radius R] [--version V]

search (full-text)

The symbolic half: AND-intersect tokenized terms over __postings__ and rank by co-occurrence. --version N queries as-of a snapshot.

dl [-d <dir>] search '<terms>' [--top N] [--version N]

dl-embed

The C++ embedding tool (ggml-based). It runs the real bge-small-en-v1.5 model in-process — the weights are the actual pretrained model (git-lfs tracked under models/), not a re-implementation. The C++ provides the inference runtime: GGUF load + BERT forward pass + CLS pooling + L2 normalization, a WordPiece tokenizer that auto-detects the llama.cpp WPM vocab convention, and the ITQ fit/encode + int8 quantization.

dl-embed pipeline --db DIR     embed corpus + emit vector relations + publish
dl-embed encode --db DIR QRY   print 'sig_hex ivec_hex' for a query (CLI consumes this)
dl-embed embed QRY             print the raw 384-float embedding
dl-embed tokenize QRY          print token ids + strings (debug)
dl-embed self-test             math/tokenizer checks (+ golden-embedding gate if model present)
dl-embed dump-tensors [PATH]   list GGUF tensors
dl-embed fetch-model           download the bge-small GGUF to models/

The dl vsearch/vhybrid commands fork/execve dl-embed encode to embed the query (no shell, no Python). The golden gate (./dl-embed self-test) embeds reference strings and asserts cosine ≥ 0.9999 against the reference model — the numeric proof that the C++ forward pass is correct.

Design

The full design of record lives in the design docs: datalog-dafsa-vector-search.md (MIH over ITQ, the integration seam, honest ceiling) and the int8-in-store note. In short: in-store MIH is competitive with HNSW at ~1e5–1e6 entities and wins the consolidation story — one identity space, one snapshot, one WAL, one crash-recovery story.