> For the complete documentation index, see [llms.txt](https://docs.telys.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.telys.ai/working-with-data/build-lexical-index.md).

# Build a lexical index

Add an on-device BM25 code index to a collection with lexical=True + build\_lexical, then query it with search\_text(mode="lexical") — keyword/code search, no model, scoped by partition.

Alongside vector search, a collection can carry an **on-device lexical index** — a code-aware BM25 index over the exact tokens of your text. It needs no model and no network: it matches on identifiers, keywords, and symbols (camelCase and snake\_case are split, so `getUserName` also matches `get`, `user`, `name`). Use it for code search and keyword recall, on its own or beside the vector path. Turn it on with `lexical=True` at [creation](https://docs.telys.ai/working-with-data/create-a-collection.md), build it with `build_lexical`, and query it with `search_text(mode="lexical")`.

{% hint style="info" %}
The lexical index is a **fitted** index (like BM25) — it learns corpus statistics at build time, so it is built explicitly (like [`build_ivf`](https://docs.telys.ai/working-with-data/build-ivf-index.md)), not maintained on every write. It is **local-only** — not exposed over a [RemoteTelys](https://docs.telys.ai/reference/remote-api.md) connection.
{% endhint %}

## Enable it, ingest, build

```python
from telys import Telys, Eq
from telys.embedding import AlgentaBigramEmbedder

telys = Telys("./telys-data")
col = telys.create_collection(
    "repo_symbols",
    dim=64,
    partition_by="scope_key",
    embedder=AlgentaBigramEmbedder(),   # ingest still embeds a vector; a lightweight one is fine
    filter_columns=("path",),
    lexical=True,                        # retain code tokens at add time
)

col.add_texts(
    ["class RefundService:\n    def reconcile_ledger(self, order): ..."],
    ids=["refund.py:RefundService"],
    metadata=[{"scope_key": "acme/payments", "path": "payments/refund.py"}],
)

col.build_lexical()                      # fit the BM25 index over the retained tokens
col.save()
```

`lexical=True` tells the collection to keep each record's code tokens as it ingests (`add_texts` / `upsert_texts`). `build_lexical()` then fits the BM25 index from those tokens.

{% hint style="warning" %}
`lexical=True` retains tokens on the **text** ingest path, so you still attach an embedder for `add_texts`/`upsert_texts` (it produces the stored vector). If you only ever query lexically, use a lightweight embedder like `AlgentaBigramEmbedder` — the lexical index ignores the vector and matches on tokens.
{% endhint %}

## Query with `mode="lexical"`

Pass `mode="lexical"` to [`search_text`](https://docs.telys.ai/working-with-data/search-by-text.md). It runs BM25 over the query's tokens instead of embedding — no embedder is needed at query time — and honours the same `where` filter:

```python
hits = col.search_text(
    "reconcile ledger entries",
    top_k=5,
    where=Eq("scope_key", "acme/payments"),   # the single-key wedge: BM25 over just this partition
    mode="lexical",
    explain=True,
)
# {"ids": ["refund.py:RefundService"], "scores": [...], "explain": {"plan": "PartitionLexicalBM25", ...}}
```

`where=Eq(partition_key, …)` scopes BM25 to one partition (fast); an `Eq` on a filter column or no filter is supported too. Results are `{"ids": [...], "scores": [...]}`, best-match first — the same shape as dense search. BM25 scores are not comparable to cosine scores; rank by position, not absolute value.

| Parameter                           | Meaning for `mode="lexical"`                                                        |
| ----------------------------------- | ----------------------------------------------------------------------------------- |
| `text`                              | The query string; tokenized (not embedded).                                         |
| `where`                             | `Eq(partition_key, v)` scopes to one partition; `Eq(column, v)` / `None` also work. |
| `top_k`, `explain`, `with_metadata` | As in [dense search](https://docs.telys.ai/working-with-data/search-by-text.md).    |

## Rebuild after writes; deletes are honoured live

Like `build_ivf`, the lexical index is fitted over the rows present at build time. Inserts and in-place updates made **after** the last `build_lexical()` are folded in at query time for a filtered partition query, but a `build_lexical()` after a batch of writes (and after [`compact()`](https://docs.telys.ai/working-with-data/delete-and-compact.md)) keeps the index fully current. **Deletes and updates are always honoured** — a removed or superseded record never surfaces, rebuild or not.

```python
col.upsert_texts([...], ids=[...], metadata=[...])
col.build_lexical()      # re-fit so new rows are fully indexed
col.save()               # persists; the index is rebuilt from retained tokens on open
```

## Tuning

`build_lexical(k1=1.8, b=1.0)` accepts the two BM25 knobs — `k1` (term-frequency saturation) and `b` (document length normalization). The defaults are tuned for code retrieval; override per collection if your corpus differs.

## Lexical vs dense

A collection can hold both: `mode="dense"` (the default) uses the embedder's vectors for semantic recall; `mode="lexical"` uses BM25 over tokens for keyword/code recall. See [Lexical vs dense](https://docs.telys.ai/understand-telys/lexical-vs-dense.md) for when to reach for each — and pair them (dense + lexical) when you want both.

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Search by text</strong></td><td>The full search_text signature, including mode.</td><td><a href="https://docs.telys.ai/working-with-data/search-by-text.md">Search by text</a></td></tr><tr><td><strong>Lexical vs dense</strong></td><td>When to use lexical, dense, or both.</td><td><a href="https://docs.telys.ai/understand-telys/lexical-vs-dense.md">Lexical vs dense</a></td></tr><tr><td><strong>Filter results</strong></td><td>Scope a query with Eq to one partition or column.</td><td><a href="https://docs.telys.ai/working-with-data/filter-results.md">Filter results</a></td></tr></tbody></table>
