> For the complete documentation index, see [llms.txt](https://docs.telys.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.telys.ai/understand-telys/lexical-vs-dense.md).

# Lexical vs dense

Telys supports two families of embedder, and the choice shapes your privacy, latency, and — most of all — *what kinds of matches you get back*. The built-in **lexical** embedders run entirely on-device with no model download and excel at keyword and natural-language-to-code recall. **Dense** embeddings — from OpenAI, BGE, or any model you wrap in a `CallableEmbedder` — capture semantic paraphrase. This page covers the difference, the tradeoffs, and how to decide.

## Two ways to turn text into vectors

A **lexical** embedder derives its vector from the surface form of the text — its characters and token n-grams. Similar spellings and shared tokens land near each other. It has no learned notion of meaning, but it is deterministic, fast, and needs only the local kernel.

A **dense** embedder is a trained model that maps text into a learned semantic space, so paraphrases with no words in common (`"how do I cancel"` vs `"stop my subscription"`) still land close. That power comes from running — or calling — a model.

Both produce the same thing Telys stores: a `(N, D)` float32 array, one vector per text.

## Built-in lexical embedders

Two lexical embedders ship in `telys.embedding`. They compute vectors in the native kernel, so there is no model to download and results are deterministic for the same input:

{% tabs %}
{% tab title="Multigram (default)" icon="layer-group" %}
`AlgentaMultigramEmbedder` concatenates L2-normalized unigram, bigram, and trigram hash blocks. It is the internal default for the MCP server (dimension hint: 384) and the strongest on-device choice for natural-language-to-code and keyword recall.

```python
from telys.embedding import AlgentaMultigramEmbedder

emb = AlgentaMultigramEmbedder()             # no download, deterministic
vecs = emb.embed_documents(["def add(a, b): return a + b"])
print(vecs.shape)                            # (1, D) — D from the kernel (hint: 384)
```

{% endtab %}

{% tab title="Bigram" icon="grip" %}
`AlgentaBigramEmbedder` is a leaner bigram-hash embedder (dimension hint: 64). Use it when you want the smallest footprint and near-exact token overlap is enough.

```python
from telys.embedding import AlgentaBigramEmbedder

emb = AlgentaBigramEmbedder()                # smaller, faster, still on-device
vecs = emb.embed_documents(["SELECT * FROM users"])
print(vecs.shape)                            # (1, D) — D from the kernel (hint: 64)
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
The `Algenta*` embedders require the native kernel and raise `ImportError` without it. Their dimension is read from the kernel — the 64 and 384 figures are hints, not guaranteed constants.
{% endhint %}

## Bring-your-own dense embeddings

Any dense model becomes a Telys embedder by wrapping a function in `CallableEmbedder`. Supply a callable `list[str] -> (N, D) float32` and a matching `EmbeddingProfile`; Telys asserts the array is 2-D and its width equals `profile.dimension`:

```python
import numpy as np
from telys.embedding import CallableEmbedder, EmbeddingProfile

def embed(texts: list[str]) -> np.ndarray:
    # Call your model (OpenAI, a local BGE model, anything).
    # Must return shape (len(texts), 1536), dtype float32.
    ...

profile = EmbeddingProfile(
    provider="openai",
    model_id="text-embedding-3-small",
    model_version="1",
    dimension=1536,
    normalization="l2",
    distance="ip",
)
emb = CallableEmbedder(embed, profile)       # dense, BYO
```

`CallableEmbedder` embeds whatever your function returns, dense or lexical. It needs no native kernel (your function does the work), though every collection operation still needs the signed runtime. See [Recipe: OpenAI embeddings](/embeddings/recipe-openai.md) and [Recipe: BGE / sentence-transformers](/embeddings/recipe-bge.md) for runnable functions.

## The tradeoffs

|                   | Built-in lexical (bigram / multigram)         | Dense (BYO via `CallableEmbedder`)                           |
| ----------------- | --------------------------------------------- | ------------------------------------------------------------ |
| Where it runs     | On-device, in the native kernel               | Wherever your model runs (API or local)                      |
| Model download    | None                                          | Model weights, or a network API call                         |
| Determinism       | Deterministic for the same input              | Depends on the provider                                      |
| Network           | None required                                 | Remote APIs (e.g. OpenAI) send text out; local models do not |
| Best recall shape | Keyword / NL→code / near-exact tokens         | Paraphrase / semantic similarity                             |
| Dimension         | Kernel-sourced (bigram \~64, multigram \~384) | Whatever your model emits (e.g. 1536)                        |
| Latency           | Very low, no I/O                              | Model inference or a network round-trip                      |
| Privacy           | Text never leaves the machine                 | Text leaves if the provider is remote                        |

The dimension difference is not cosmetic: it determines the collection's [space](/understand-telys/embeddings-and-spaces.md). A collection built with a multigram embedder cannot be reopened with an OpenAI embedder — they are different spaces.

## Which one should you choose

* **Default to multigram** for on-device work, code search, and keyword-heavy corpora. It ships in the box, needs no download or network, and is deterministic — ideal for offline and privacy-sensitive deployments, and the strongest on-device option for natural-language-to-code recall.
* **Reach for dense** when your queries and documents share meaning but not words — support tickets, chat logs, prose paraphrase — and you accept a model download (local) or an outbound call (hosted API).
* **Keep both**, in separate collections — a codebase collection can be lexical while a conversation collection is dense.

{% hint style="warning" %}
A remote dense provider means your document and query text leaves the machine on every embed. If that is unacceptable, use a local dense model or stay lexical. See [What leaves the machine](/security/what-leaves-the-machine.md).
{% endhint %}

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Choosing an embedder</strong></td><td>A decision guide mapping data and constraints to an embedder.</td><td><a href="/pages/XuNwRcXB5hZbgTGaTAau">/pages/XuNwRcXB5hZbgTGaTAau</a></td></tr><tr><td><strong>Recipe: OpenAI embeddings</strong></td><td>A complete CallableEmbedder for OpenAI dense vectors.</td><td><a href="/pages/osxpAbHUV1iJmzTnmdX3">/pages/osxpAbHUV1iJmzTnmdX3</a></td></tr><tr><td><strong>Recipe: BGE / sentence-transformers</strong></td><td>Local dense embeddings with no network calls.</td><td><a href="/pages/ygPCwb0JT95fwoRUVC1g">/pages/ygPCwb0JT95fwoRUVC1g</a></td></tr></tbody></table>

Next: see [Embeddings & vector spaces](/understand-telys/embeddings-and-spaces.md) for how your choice fixes the collection's space.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.telys.ai/understand-telys/lexical-vs-dense.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.