> For the complete documentation index, see [llms.txt](https://docs.telys.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.telys.ai/recipes/batch-ingestion.md).

# Batch ingestion

Loading a large corpus is throughput work. A few habits make it fast:

1. **Batch** `add_texts` — insert a few hundred to a few thousand rows per call.
2. **Precompute** vectors and use `add` when you bring your own embedding model.
3. **Build IVF once, after the load** — not before, not per batch.
4. **Save once, at the end** — `save()` is the commit point; call it after ingestion, not in the loop.

{% hint style="info" %}
Install with `pipx install telys`, sign in with `telys login`, then verify with `telys runtime verify`. Text ingestion (`add_texts`) needs an embedder with the native kernel; the raw-vector path (`add`) needs only the runtime. See [Install Telys](/start-here/install.md).
{% endhint %}

## Batch text ingestion

Feed documents in fixed-size batches. `add_texts` embeds and inserts new rows in one call, so larger batches amortize per-call overhead. Commit once at the end.

{% code title="ingest\_texts.py" lineNumbers="true" %}

```python
from telys import Telys
from telys.embedding import AlgentaMultigramEmbedder


def batched(iterable, size):
    batch = []
    for item in iterable:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch


def ingest(docs, store="./corpus", name="docs", batch_size=1000):
    emb = AlgentaMultigramEmbedder()
    col = Telys(store).create_collection(
        name,
        dim=emb.profile.dimension,
        partition_by="source",
        embedder=emb,
    )

    total = 0
    for batch in batched(docs, batch_size):
        texts = [d["text"] for d in batch]
        ids = [d["id"] for d in batch]
        metas = [{"source": d["source"], "text": d["text"]} for d in batch]
        col.add_texts(texts, ids, metas)   # embeds + inserts new rows
        total += len(batch)
        print(f"ingested {total} docs")

    col.build_ivf(min_rows=20000, target_recall=0.98)  # once, after the load
    col.save()                                          # single commit point
    print(f"done: {len(col)} rows in {name}")
    return col


if __name__ == "__main__":
    corpus = (
        {"id": f"doc-{i}", "source": "wiki", "text": f"Document number {i} about vector search."}
        for i in range(50_000)
    )
    ingest(corpus)
```

{% endcode %}

## Bring-your-own vectors with `add`

If you already have embeddings — computed offline or exported from another pipeline — skip embedding and insert raw vectors with `add`. It takes an `(n, dim)` float32 array plus aligned ids and metadata. The collection still needs a `dim`, but the raw-vector path needs no embedder.

{% code title="ingest\_vectors.py" lineNumbers="true" %}

```python
import numpy as np

from telys import Telys

DIM = 768  # your model's output dimension


def ingest_vectors(rows, store="./corpus", name="byo", batch_size=2048):
    # No embedder needed — we insert precomputed vectors directly.
    col = Telys(store).create_collection(name, dim=DIM, partition_by="source")

    buf_vecs, buf_ids, buf_meta = [], [], []

    def flush():
        if not buf_ids:
            return
        vecs = np.asarray(buf_vecs, dtype="float32")   # shape (n, DIM)
        col.add(vecs, buf_ids, buf_meta)
        buf_vecs.clear(); buf_ids.clear(); buf_meta.clear()

    for r in rows:
        buf_vecs.append(r["vector"])
        buf_ids.append(r["id"])
        buf_meta.append({"source": r["source"], "text": r["text"]})
        if len(buf_ids) == batch_size:
            flush()
    flush()

    col.build_ivf(min_rows=20000, target_recall=0.98)
    col.save()
    return col
```

{% endcode %}

{% hint style="warning" %}
`add` and `add_texts` insert **new** rows only and raise `KeyError` if an id exists. When a load may replay ids (a resumed job, an updated snapshot), use `upsert` / `upsert_texts`, which insert-or-replace. See [Add & upsert records](/working-with-data/add-and-upsert.md).
{% endhint %}

## Throughput tips

* **Save once.** `save()` writes the `id_map.npy` sidecar, then `collection.json` as the final commit — not per batch. Save after the whole load (and periodically for very long jobs if you want checkpoints).
* **Build IVF after loading.** `build_ivf` partitions the loaded vectors; running it before data exists or per batch wastes work. It is a no-op below `min_rows`, so one call at the end is always safe. See [Build the IVF index](/working-with-data/build-ivf-index.md).
* **Prefer `add` over `upsert` when ids are unique.** The insert-only path skips existence-and-replace bookkeeping.
* **Right-size batches.** A few hundred to a few thousand rows per call balances call overhead against memory; tune to row size and available RAM.
* **Group by partition.** Rows sharing a `partition_by` value land together; ingesting a partition contiguously keeps writes localized.

## Expected result

The text example streams progress and reports the final row count once the commit lands:

```
ingested 1000 docs
ingested 2000 docs
...
done: 50000 rows in docs
```

IVF is built and the collection saved only at the end, so the on-disk collection under `./corpus/docs/` is written exactly once, at `save()`.

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Build the IVF index</strong></td><td>When and how to build IVF for fast, high-recall search.</td><td><a href="/pages/vAtnT9NtTA0el7skR3vX">/pages/vAtnT9NtTA0el7skR3vX</a></td></tr><tr><td><strong>Add text vs vectors</strong></td><td>Choose between embedding and bring-your-own vectors.</td><td><a href="/pages/9Xcdtx1OrsC118fTHJL0">/pages/9Xcdtx1OrsC118fTHJL0</a></td></tr><tr><td><strong>Snapshots &#x26; persistence</strong></td><td>How save, snapshots, and the on-disk format work.</td><td><a href="/pages/3IabEB7rmGWO2uCD2t8X">/pages/3IabEB7rmGWO2uCD2t8X</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.telys.ai/recipes/batch-ingestion.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.