Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Lexical vs dense

When to use Telys's built-in on-device lexical embedders versus bring-your-own dense embeddings, and the privacy, latency, and recall tradeoffs of each.

Telys supports two families of embedder, and the choice shapes your privacy, latency, and — most of all — what kinds of matches you get back. The built-in lexical embedders run entirely on-device with no model download and excel at keyword and natural-language-to-code recall. Dense embeddings — from OpenAI, BGE, or any model you wrap in a CallableEmbedder — capture semantic paraphrase. This page covers the difference, the tradeoffs, and how to decide.

Two ways to turn text into vectors

A lexical embedder derives its vector from the surface form of the text — its characters and token n-grams. Similar spellings and shared tokens land near each other. It has no learned notion of meaning, but it is deterministic, fast, and needs only the local kernel.

A dense embedder is a trained model that maps text into a learned semantic space, so paraphrases with no words in common ("how do I cancel" vs "stop my subscription") still land close. That power comes from running — or calling — a model.

Both produce the same thing Telys stores: a (N, D) float32 array, one vector per text.

Built-in lexical embedders

Two lexical embedders ship in telys.embedding. They compute vectors in the native kernel, so there is no model to download and results are deterministic for the same input:

AlgentaMultigramEmbedder concatenates L2-normalized unigram, bigram, and trigram hash blocks. It is the internal default for the MCP server (dimension hint: 384) and the strongest on-device choice for natural-language-to-code and keyword recall.

from telys.embedding import AlgentaMultigramEmbedder

emb = AlgentaMultigramEmbedder()             # no download, deterministic
vecs = emb.embed_documents(["def add(a, b): return a + b"])
print(vecs.shape)                            # (1, D) — D from the kernel (hint: 384)

AlgentaBigramEmbedder is a leaner bigram-hash embedder (dimension hint: 64). Use it when you want the smallest footprint and near-exact token overlap is enough.

from telys.embedding import AlgentaBigramEmbedder

emb = AlgentaBigramEmbedder()                # smaller, faster, still on-device
vecs = emb.embed_documents(["SELECT * FROM users"])
print(vecs.shape)                            # (1, D) — D from the kernel (hint: 64)

The Algenta* embedders require the native kernel and raise ImportError without it. Their dimension is read from the kernel — the 64 and 384 figures are hints, not guaranteed constants.

Lexical embedder vs the lexical BM25 index

There are two on-device lexical paths, and they are different tools:

  • A lexical embedder (bigram/multigram, above) turns text into a fixed vector you search like any other — it is a mode="dense" vector path that happens to encode surface form.

  • The lexical BM25 index is a fitted keyword index over your exact code tokens (camelCase/snake_case split, IDF-weighted). You enable it with lexical=True, build it with build_lexical(), and query it with search_text(mode="lexical"). It is code-aware term matching — the closest thing to classic BM25/grep, scoped to a partition.

Reach for the BM25 index when you want precise identifier/keyword/symbol recall over code (the strongest on-device option for that); reach for a lexical embedder when you want a single vector space (e.g. to mix with dense) or typo-adjacent character-n-gram matching. A collection can carry both — query mode="dense" or mode="lexical" per request. See Build a lexical index.

Bring-your-own dense embeddings

Any dense model becomes a Telys embedder by wrapping a function in CallableEmbedder. Supply a callable list[str] -> (N, D) float32 and a matching EmbeddingProfile; Telys asserts the array is 2-D and its width equals profile.dimension:

CallableEmbedder embeds whatever your function returns, dense or lexical. It needs no native kernel (your function does the work), though every collection operation still needs the signed runtime. See Recipe: OpenAI embeddings and Recipe: BGE / sentence-transformers for runnable functions.

The tradeoffs

Built-in lexical (bigram / multigram)

Dense (BYO via CallableEmbedder)

Where it runs

On-device, in the native kernel

Wherever your model runs (API or local)

Model download

None

Model weights, or a network API call

Determinism

Deterministic for the same input

Depends on the provider

Network

None required

Remote APIs (e.g. OpenAI) send text out; local models do not

Best recall shape

Keyword / NL→code / near-exact tokens

Paraphrase / semantic similarity

Dimension

Kernel-sourced (bigram ~64, multigram ~384)

Whatever your model emits (e.g. 1536)

Latency

Very low, no I/O

Model inference or a network round-trip

Privacy

Text never leaves the machine

Text leaves if the provider is remote

The dimension difference is not cosmetic: it determines the collection's space. A collection built with a multigram embedder cannot be reopened with an OpenAI embedder — they are different spaces.

Which one should you choose

  • Default to multigram for on-device work, code search, and keyword-heavy corpora. It ships in the box, needs no download or network, and is deterministic — ideal for offline and privacy-sensitive deployments, and the strongest on-device option for natural-language-to-code recall.

  • Reach for dense when your queries and documents share meaning but not words — support tickets, chat logs, prose paraphrase — and you accept a model download (local) or an outbound call (hosted API).

  • Keep both, in separate collections — a codebase collection can be lexical while a conversation collection is dense.

Next steps

Next: see Embeddings & vector spaces for how your choice fixes the collection's space.

Last updated

Was this helpful?