> For the complete documentation index, see [llms.txt](https://docs.telys.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.telys.ai/understand-telys/lexical-vs-dense.md).

# Lexical vs dense

When to use Telys's built-in on-device lexical embedders versus bring-your-own dense embeddings, and the privacy, latency, and recall tradeoffs of each.

Telys supports two families of embedder, and the choice shapes your privacy, latency, and — most of all — *what kinds of matches you get back*. The built-in **lexical** embedders run entirely on-device with no model download and excel at keyword and natural-language-to-code recall. **Dense** embeddings — from OpenAI, BGE, or any model you wrap in a `CallableEmbedder` — capture semantic paraphrase. This page covers the difference, the tradeoffs, and how to decide.

## Two ways to turn text into vectors

A **lexical** embedder derives its vector from the surface form of the text — its characters and token n-grams. Similar spellings and shared tokens land near each other. It has no learned notion of meaning, but it is deterministic, fast, and needs only the local kernel.

A **dense** embedder is a trained model that maps text into a learned semantic space, so paraphrases with no words in common (`"how do I cancel"` vs `"stop my subscription"`) still land close. That power comes from running — or calling — a model.

Both produce the same thing Telys stores: a `(N, D)` float32 array, one vector per text.

## Built-in lexical embedders

Two lexical embedders ship in `telys.embedding`. They compute vectors in the native kernel, so there is no model to download and results are deterministic for the same input:

{% tabs %}
{% tab title="Multigram (default)" icon="layer-group" %}
`AlgentaMultigramEmbedder` concatenates L2-normalized unigram, bigram, and trigram hash blocks. It is the internal default for the MCP server (dimension hint: 384) and the strongest on-device choice for natural-language-to-code and keyword recall.

```python
from telys.embedding import AlgentaMultigramEmbedder

emb = AlgentaMultigramEmbedder()             # no download, deterministic
vecs = emb.embed_documents(["def add(a, b): return a + b"])
print(vecs.shape)                            # (1, D) — D from the kernel (hint: 384)
```

{% endtab %}

{% tab title="Bigram" icon="grip" %}
`AlgentaBigramEmbedder` is a leaner bigram-hash embedder (dimension hint: 64). Use it when you want the smallest footprint and near-exact token overlap is enough.

```python
from telys.embedding import AlgentaBigramEmbedder

emb = AlgentaBigramEmbedder()                # smaller, faster, still on-device
vecs = emb.embed_documents(["SELECT * FROM users"])
print(vecs.shape)                            # (1, D) — D from the kernel (hint: 64)
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
The `Algenta*` embedders require the native kernel and raise `ImportError` without it. Their dimension is read from the kernel — the 64 and 384 figures are hints, not guaranteed constants.
{% endhint %}

## Lexical embedder vs the lexical BM25 index

There are **two** on-device lexical paths, and they are different tools:

* A lexical **embedder** (bigram/multigram, above) turns text into a fixed vector you search like any other — it is a `mode="dense"` vector path that happens to encode surface form.
* The lexical **BM25 index** is a fitted keyword index over your exact code tokens (camelCase/snake\_case split, IDF-weighted). You enable it with `lexical=True`, build it with `build_lexical()`, and query it with `search_text(mode="lexical")`. It is code-aware term matching — the closest thing to classic BM25/grep, scoped to a partition.

Reach for the **BM25 index** when you want precise identifier/keyword/symbol recall over code (the strongest on-device option for that); reach for a lexical **embedder** when you want a single vector space (e.g. to mix with dense) or typo-adjacent character-n-gram matching. A collection can carry both — query `mode="dense"` or `mode="lexical"` per request. See [Build a lexical index](https://docs.telys.ai/working-with-data/build-lexical-index.md).

## Bring-your-own dense embeddings

Any dense model becomes a Telys embedder by wrapping a function in `CallableEmbedder`. Supply a callable `list[str] -> (N, D) float32` and a matching `EmbeddingProfile`; Telys asserts the array is 2-D and its width equals `profile.dimension`:

```python
import numpy as np
from telys.embedding import CallableEmbedder, EmbeddingProfile

def embed(texts: list[str]) -> np.ndarray:
    # Call your model (OpenAI, a local BGE model, anything).
    # Must return shape (len(texts), 1536), dtype float32.
    ...

profile = EmbeddingProfile(
    provider="openai",
    model_id="text-embedding-3-small",
    model_version="1",
    dimension=1536,
    normalization="l2",
    distance="ip",
)
emb = CallableEmbedder(embed, profile)       # dense, BYO
```

`CallableEmbedder` embeds whatever your function returns, dense or lexical. It needs no native kernel (your function does the work), though every collection operation still needs the signed runtime. See [Recipe: OpenAI embeddings](https://docs.telys.ai/embeddings/recipe-openai.md) and [Recipe: BGE / sentence-transformers](https://docs.telys.ai/embeddings/recipe-bge.md) for runnable functions.

## The tradeoffs

|                   | Built-in lexical (bigram / multigram)         | Dense (BYO via `CallableEmbedder`)                           |
| ----------------- | --------------------------------------------- | ------------------------------------------------------------ |
| Where it runs     | On-device, in the native kernel               | Wherever your model runs (API or local)                      |
| Model download    | None                                          | Model weights, or a network API call                         |
| Determinism       | Deterministic for the same input              | Depends on the provider                                      |
| Network           | None required                                 | Remote APIs (e.g. OpenAI) send text out; local models do not |
| Best recall shape | Keyword / NL→code / near-exact tokens         | Paraphrase / semantic similarity                             |
| Dimension         | Kernel-sourced (bigram \~64, multigram \~384) | Whatever your model emits (e.g. 1536)                        |
| Latency           | Very low, no I/O                              | Model inference or a network round-trip                      |
| Privacy           | Text never leaves the machine                 | Text leaves if the provider is remote                        |

The dimension difference is not cosmetic: it determines the collection's [space](https://docs.telys.ai/understand-telys/embeddings-and-spaces.md). A collection built with a multigram embedder cannot be reopened with an OpenAI embedder — they are different spaces.

## Which one should you choose

* **Default to multigram** for on-device work, code search, and keyword-heavy corpora. It ships in the box, needs no download or network, and is deterministic — ideal for offline and privacy-sensitive deployments, and the strongest on-device option for natural-language-to-code recall.
* **Reach for dense** when your queries and documents share meaning but not words — support tickets, chat logs, prose paraphrase — and you accept a model download (local) or an outbound call (hosted API).
* **Keep both**, in separate collections — a codebase collection can be lexical while a conversation collection is dense.

{% hint style="warning" %}
A remote dense provider means your document and query text leaves the machine on every embed. If that is unacceptable, use a local dense model or stay lexical. See [What leaves the machine](https://docs.telys.ai/security/what-leaves-the-machine.md).
{% endhint %}

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Choosing an embedder</strong></td><td>A decision guide mapping data and constraints to an embedder.</td><td><a href="https://docs.telys.ai/embeddings/choosing-an-embedder.md">Choosing an embedder</a></td></tr><tr><td><strong>Recipe: OpenAI embeddings</strong></td><td>A complete CallableEmbedder for OpenAI dense vectors.</td><td><a href="https://docs.telys.ai/embeddings/recipe-openai.md">OpenAI recipe</a></td></tr><tr><td><strong>Recipe: BGE / sentence-transformers</strong></td><td>Local dense embeddings with no network calls.</td><td><a href="https://docs.telys.ai/embeddings/recipe-bge.md">BGE recipe</a></td></tr></tbody></table>

Next: see [Embeddings & vector spaces](https://docs.telys.ai/understand-telys/embeddings-and-spaces.md) for how your choice fixes the collection's space.
