Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Build a lexical index

Add an on-device BM25 code index to a collection with lexical=True + build_lexical, then query it with search_text(mode="lexical") — keyword/code search, no model, scoped by partition.

Alongside vector search, a collection can carry an on-device lexical index — a code-aware BM25 index over the exact tokens of your text. It needs no model and no network: it matches on identifiers, keywords, and symbols (camelCase and snake_case are split, so getUserName also matches get, user, name). Use it for code search and keyword recall, on its own or beside the vector path. Turn it on with lexical=True at creation, build it with build_lexical, and query it with search_text(mode="lexical").

The lexical index is a fitted index (like BM25) — it learns corpus statistics at build time, so it is built explicitly (like build_ivf), not maintained on every write. It is local-only — not exposed over a RemoteTelys connection.

Enable it, ingest, build

from telys import Telys, Eq
from telys.embedding import AlgentaBigramEmbedder

telys = Telys("./telys-data")
col = telys.create_collection(
    "repo_symbols",
    dim=64,
    partition_by="scope_key",
    embedder=AlgentaBigramEmbedder(),   # ingest still embeds a vector; a lightweight one is fine
    filter_columns=("path",),
    lexical=True,                        # retain code tokens at add time
)

col.add_texts(
    ["class RefundService:\n    def reconcile_ledger(self, order): ..."],
    ids=["refund.py:RefundService"],
    metadata=[{"scope_key": "acme/payments", "path": "payments/refund.py"}],
)

col.build_lexical()                      # fit the BM25 index over the retained tokens
col.save()

lexical=True tells the collection to keep each record's code tokens as it ingests (add_texts / upsert_texts). build_lexical() then fits the BM25 index from those tokens.

Query with mode="lexical"

Pass mode="lexical" to search_text. It runs BM25 over the query's tokens instead of embedding — no embedder is needed at query time — and honours the same where filter:

where=Eq(partition_key, …) scopes BM25 to one partition (fast); an Eq on a filter column or no filter is supported too. Results are {"ids": [...], "scores": [...]}, best-match first — the same shape as dense search. BM25 scores are not comparable to cosine scores; rank by position, not absolute value.

Parameter

Meaning for mode="lexical"

text

The query string; tokenized (not embedded).

where

Eq(partition_key, v) scopes to one partition; Eq(column, v) / None also work.

top_k, explain, with_metadata

Rebuild after writes; deletes are honoured live

Like build_ivf, the lexical index is fitted over the rows present at build time. Inserts and in-place updates made after the last build_lexical() are folded in at query time for a filtered partition query, but a build_lexical() after a batch of writes (and after compact()) keeps the index fully current. Deletes and updates are always honoured — a removed or superseded record never surfaces, rebuild or not.

Tuning

build_lexical(k1=1.8, b=1.0) accepts the two BM25 knobs — k1 (term-frequency saturation) and b (document length normalization). The defaults are tuned for code retrieval; override per collection if your corpus differs.

Lexical vs dense

A collection can hold both: mode="dense" (the default) uses the embedder's vectors for semantic recall; mode="lexical" uses BM25 over tokens for keyword/code recall. See Lexical vs dense for when to reach for each — and pair them (dense + lexical) when you want both.

Next steps

Last updated

Was this helpful?