Build a lexical index
Add an on-device BM25 code index to a collection with lexical=True + build_lexical, then query it with search_text(mode="lexical") — keyword/code search, no model, scoped by partition.
Alongside vector search, a collection can carry an on-device lexical index — a code-aware BM25 index over the exact tokens of your text. It needs no model and no network: it matches on identifiers, keywords, and symbols (camelCase and snake_case are split, so getUserName also matches get, user, name). Use it for code search and keyword recall, on its own or beside the vector path. Turn it on with lexical=True at creation, build it with build_lexical, and query it with search_text(mode="lexical").
Enable it, ingest, build
from telys import Telys, Eq
from telys.embedding import AlgentaBigramEmbedder
telys = Telys("./telys-data")
col = telys.create_collection(
"repo_symbols",
dim=64,
partition_by="scope_key",
embedder=AlgentaBigramEmbedder(), # ingest still embeds a vector; a lightweight one is fine
filter_columns=("path",),
lexical=True, # retain code tokens at add time
)
col.add_texts(
["class RefundService:\n def reconcile_ledger(self, order): ..."],
ids=["refund.py:RefundService"],
metadata=[{"scope_key": "acme/payments", "path": "payments/refund.py"}],
)
col.build_lexical() # fit the BM25 index over the retained tokens
col.save()lexical=True tells the collection to keep each record's code tokens as it ingests (add_texts / upsert_texts). build_lexical() then fits the BM25 index from those tokens.
Query with mode="lexical"
Pass mode="lexical" to search_text. It runs BM25 over the query's tokens instead of embedding — no embedder is needed at query time — and honours the same where filter:
where=Eq(partition_key, …) scopes BM25 to one partition (fast); an Eq on a filter column or no filter is supported too. Results are {"ids": [...], "scores": [...]}, best-match first — the same shape as dense search. BM25 scores are not comparable to cosine scores; rank by position, not absolute value.
Parameter
Meaning for mode="lexical"
text
The query string; tokenized (not embedded).
where
Eq(partition_key, v) scopes to one partition; Eq(column, v) / None also work.
top_k, explain, with_metadata
As in dense search.
Rebuild after writes; deletes are honoured live
Like build_ivf, the lexical index is fitted over the rows present at build time. Inserts and in-place updates made after the last build_lexical() are folded in at query time for a filtered partition query, but a build_lexical() after a batch of writes (and after compact()) keeps the index fully current. Deletes and updates are always honoured — a removed or superseded record never surfaces, rebuild or not.
Tuning
build_lexical(k1=1.8, b=1.0) accepts the two BM25 knobs — k1 (term-frequency saturation) and b (document length normalization). The defaults are tuned for code retrieval; override per collection if your corpus differs.
Lexical vs dense
A collection can hold both: mode="dense" (the default) uses the embedder's vectors for semantic recall; mode="lexical" uses BM25 over tokens for keyword/code recall. See Lexical vs dense for when to reach for each — and pair them (dense + lexical) when you want both.
Next steps
Last updated
Was this helpful?