Lexical vs dense
When to use Telys's built-in on-device lexical embedders versus bring-your-own dense embeddings, and the privacy, latency, and recall tradeoffs of each.
Telys supports two families of embedder, and the choice shapes your privacy, latency, and — most of all — what kinds of matches you get back. The built-in lexical embedders run entirely on-device with no model download and excel at keyword and natural-language-to-code recall. Dense embeddings — from OpenAI, BGE, or any model you wrap in a CallableEmbedder — capture semantic paraphrase. This page covers the difference, the tradeoffs, and how to decide.
Two ways to turn text into vectors
A lexical embedder derives its vector from the surface form of the text — its characters and token n-grams. Similar spellings and shared tokens land near each other. It has no learned notion of meaning, but it is deterministic, fast, and needs only the local kernel.
A dense embedder is a trained model that maps text into a learned semantic space, so paraphrases with no words in common ("how do I cancel" vs "stop my subscription") still land close. That power comes from running — or calling — a model.
Both produce the same thing Telys stores: a (N, D) float32 array, one vector per text.
Built-in lexical embedders
Two lexical embedders ship in telys.embedding. They compute vectors in the native kernel, so there is no model to download and results are deterministic for the same input:
AlgentaMultigramEmbedder concatenates L2-normalized unigram, bigram, and trigram hash blocks. It is the internal default for the MCP server (dimension hint: 384) and the strongest on-device choice for natural-language-to-code and keyword recall.
from telys.embedding import AlgentaMultigramEmbedder
emb = AlgentaMultigramEmbedder() # no download, deterministic
vecs = emb.embed_documents(["def add(a, b): return a + b"])
print(vecs.shape) # (1, D) — D from the kernel (hint: 384)AlgentaBigramEmbedder is a leaner bigram-hash embedder (dimension hint: 64). Use it when you want the smallest footprint and near-exact token overlap is enough.
from telys.embedding import AlgentaBigramEmbedder
emb = AlgentaBigramEmbedder() # smaller, faster, still on-device
vecs = emb.embed_documents(["SELECT * FROM users"])
print(vecs.shape) # (1, D) — D from the kernel (hint: 64)Lexical embedder vs the lexical BM25 index
There are two on-device lexical paths, and they are different tools:
A lexical embedder (bigram/multigram, above) turns text into a fixed vector you search like any other — it is a
mode="dense"vector path that happens to encode surface form.The lexical BM25 index is a fitted keyword index over your exact code tokens (camelCase/snake_case split, IDF-weighted). You enable it with
lexical=True, build it withbuild_lexical(), and query it withsearch_text(mode="lexical"). It is code-aware term matching — the closest thing to classic BM25/grep, scoped to a partition.
Reach for the BM25 index when you want precise identifier/keyword/symbol recall over code (the strongest on-device option for that); reach for a lexical embedder when you want a single vector space (e.g. to mix with dense) or typo-adjacent character-n-gram matching. A collection can carry both — query mode="dense" or mode="lexical" per request. See Build a lexical index.
Bring-your-own dense embeddings
Any dense model becomes a Telys embedder by wrapping a function in CallableEmbedder. Supply a callable list[str] -> (N, D) float32 and a matching EmbeddingProfile; Telys asserts the array is 2-D and its width equals profile.dimension:
CallableEmbedder embeds whatever your function returns, dense or lexical. It needs no native kernel (your function does the work), though every collection operation still needs the signed runtime. See Recipe: OpenAI embeddings and Recipe: BGE / sentence-transformers for runnable functions.
The tradeoffs
Built-in lexical (bigram / multigram)
Dense (BYO via CallableEmbedder)
Where it runs
On-device, in the native kernel
Wherever your model runs (API or local)
Model download
None
Model weights, or a network API call
Determinism
Deterministic for the same input
Depends on the provider
Network
None required
Remote APIs (e.g. OpenAI) send text out; local models do not
Best recall shape
Keyword / NL→code / near-exact tokens
Paraphrase / semantic similarity
Dimension
Kernel-sourced (bigram ~64, multigram ~384)
Whatever your model emits (e.g. 1536)
Latency
Very low, no I/O
Model inference or a network round-trip
Privacy
Text never leaves the machine
Text leaves if the provider is remote
The dimension difference is not cosmetic: it determines the collection's space. A collection built with a multigram embedder cannot be reopened with an OpenAI embedder — they are different spaces.
Which one should you choose
Default to multigram for on-device work, code search, and keyword-heavy corpora. It ships in the box, needs no download or network, and is deterministic — ideal for offline and privacy-sensitive deployments, and the strongest on-device option for natural-language-to-code recall.
Reach for dense when your queries and documents share meaning but not words — support tickets, chat logs, prose paraphrase — and you accept a model download (local) or an outbound call (hosted API).
Keep both, in separate collections — a codebase collection can be lexical while a conversation collection is dense.
Next steps
Next: see Embeddings & vector spaces for how your choice fixes the collection's space.
Last updated
Was this helpful?