> For the complete documentation index, see [llms.txt](https://docs.telys.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.telys.ai/recipes/batch-ingestion.md).

# Batch ingestion

Load large corpora into Telys efficiently — batch add\_texts, precompute vectors with add for BYO models, build the IVF index after load, and save once.

Loading a large corpus is throughput work. A few habits make it fast:

1. **Batch** `add_texts` — insert a few hundred to a few thousand rows per call.
2. **Precompute** vectors and use `add` when you bring your own embedding model.
3. **Build IVF once, after the load** — not before, not per batch.
4. **Save once, at the end** — `save()` is the commit point; call it after ingestion, not in the loop.

{% hint style="info" %}
Install with `pipx install telys`, sign in with `telys login`, then verify with `telys runtime verify`. Text ingestion (`add_texts`) needs an embedder with the native kernel; the raw-vector path (`add`) needs only the runtime. See [Install Telys](https://docs.telys.ai/start-here/install.md).
{% endhint %}

## Batch text ingestion

Feed documents in fixed-size batches. `add_texts` embeds and inserts new rows in one call, so larger batches amortize per-call overhead. Commit once at the end.

{% code title="ingest\_texts.py" lineNumbers="true" %}

```python
from telys import Telys
from telys.embedding import AlgentaMultigramEmbedder


def batched(iterable, size):
    batch = []
    for item in iterable:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch


def ingest(docs, store="./corpus", name="docs", batch_size=1000):
    emb = AlgentaMultigramEmbedder()
    col = Telys(store).create_collection(
        name,
        dim=emb.profile.dimension,
        partition_by="source",
        embedder=emb,
    )

    total = 0
    for batch in batched(docs, batch_size):
        texts = [d["text"] for d in batch]
        ids = [d["id"] for d in batch]
        metas = [{"source": d["source"], "text": d["text"]} for d in batch]
        col.add_texts(texts, ids, metas)   # embeds + inserts new rows
        total += len(batch)
        print(f"ingested {total} docs")

    col.build_ivf(min_rows=20000, target_recall=0.98)  # once, after the load
    col.save()                                          # single commit point
    print(f"done: {len(col)} rows in {name}")
    return col


if __name__ == "__main__":
    corpus = (
        {"id": f"doc-{i}", "source": "wiki", "text": f"Document number {i} about vector search."}
        for i in range(50_000)
    )
    ingest(corpus)
```

{% endcode %}

## Bring-your-own vectors with `add`

If you already have embeddings — computed offline or exported from another pipeline — skip embedding and insert raw vectors with `add`. It takes an `(n, dim)` float32 array plus aligned ids and metadata. The collection still needs a `dim`, but the raw-vector path needs no embedder.

{% code title="ingest\_vectors.py" lineNumbers="true" %}

```python
import numpy as np

from telys import Telys

DIM = 768  # your model's output dimension


def ingest_vectors(rows, store="./corpus", name="byo", batch_size=2048):
    # No embedder needed — we insert precomputed vectors directly.
    col = Telys(store).create_collection(name, dim=DIM, partition_by="source")

    buf_vecs, buf_ids, buf_meta = [], [], []

    def flush():
        if not buf_ids:
            return
        vecs = np.asarray(buf_vecs, dtype="float32")   # shape (n, DIM)
        col.add(vecs, buf_ids, buf_meta)
        buf_vecs.clear(); buf_ids.clear(); buf_meta.clear()

    for r in rows:
        buf_vecs.append(r["vector"])
        buf_ids.append(r["id"])
        buf_meta.append({"source": r["source"], "text": r["text"]})
        if len(buf_ids) == batch_size:
            flush()
    flush()

    col.build_ivf(min_rows=20000, target_recall=0.98)
    col.save()
    return col
```

{% endcode %}

{% hint style="warning" %}
`add` and `add_texts` insert **new** rows only and raise `KeyError` if an id exists. When a load may replay ids (a resumed job, an updated snapshot), use `upsert` / `upsert_texts`, which insert-or-replace. See [Add & upsert records](https://docs.telys.ai/working-with-data/add-and-upsert.md).
{% endhint %}

## Throughput tips

* **Save once.** `save()` writes the `id_map.npy` sidecar, then `collection.json` as the final commit — not per batch. Save after the whole load (and periodically for very long jobs if you want checkpoints).
* **Build IVF after loading.** `build_ivf` partitions the loaded vectors; running it before data exists or per batch wastes work. It is a no-op below `min_rows`, so one call at the end is always safe. See [Build the IVF index](https://docs.telys.ai/working-with-data/build-ivf-index.md).
* **Prefer `add` over `upsert` when ids are unique.** The insert-only path skips existence-and-replace bookkeeping.
* **Right-size batches.** A few hundred to a few thousand rows per call balances call overhead against memory; tune to row size and available RAM.
* **Group by partition.** Rows sharing a `partition_by` value land together; ingesting a partition contiguously keeps writes localized.

## Expected result

The text example streams progress and reports the final row count once the commit lands:

```
ingested 1000 docs
ingested 2000 docs
...
done: 50000 rows in docs
```

IVF is built and the collection saved only at the end, so the on-disk collection under `./corpus/docs/` is written exactly once, at `save()`.

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Build the IVF index</strong></td><td>When and how to build IVF for fast, high-recall search.</td><td><a href="https://docs.telys.ai/working-with-data/build-ivf-index.md">Build IVF index</a></td></tr><tr><td><strong>Add text vs vectors</strong></td><td>Choose between embedding and bring-your-own vectors.</td><td><a href="https://docs.telys.ai/working-with-data/add-text-vs-vectors.md">Text vs vectors</a></td></tr><tr><td><strong>Snapshots &#x26; persistence</strong></td><td>How save, snapshots, and the on-disk format work.</td><td><a href="https://docs.telys.ai/working-with-data/snapshots-and-persistence.md">Save &amp; snapshots</a></td></tr></tbody></table>
