> For the complete documentation index, see [llms.txt](https://docs.telys.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.telys.ai/understand-telys/versioning-soft-delete-compaction.md).

# Versioning & compaction

A Telys collection is not overwritten in place on every write. Your first batch seals a **base image**; every later batch layers on top as changes — brand-new rows and versioned replacements. Deletes are logical until `compact()` reclaims their space, and nothing touches disk until `save()`. Understand this model and the results of `search`, `ids`, and `len(col)` are never a surprise.

{% hint style="warning" %}
This is a **provenance and space-reclamation** model, not time travel. Telys has no `as_of()`, validity-window, or historical query API — `search`, `search_text`, and `ids` always return the current live state. Querying past versions is on the [roadmap](/resources/roadmap.md), not shipped.
{% endhint %}

## The layered model

Think of a collection as a sealed base plus an ordered stack of deltas:

* The **base** is the image written the first time you ingest and `save()`. It is the baseline every later change is measured against.
* Each later batch is layered as **deltas**. A row whose external id has never been seen is a **delta-insert**; a row whose id already exists is a **versioned-update** that supersedes the base (or an earlier delta) for that id.
* A **delete** marks an id as removed from the live set without physically rewriting the base.
* **Compaction** folds the deltas back into a fresh base and drops everything superseded or deleted.

The live set — what queries see — is always "the base, with versioned-updates applied and deleted ids removed."

## Which write does what

Every ingest call returns the list of external ids it touched. What happens to an id depends on whether it already exists:

| Call                      | Brand-new id                     | Id that already exists              | Needs an embedder               |
| ------------------------- | -------------------------------- | ----------------------------------- | ------------------------------- |
| `add_texts` / `add`       | inserts a new row (delta-insert) | raises `KeyError`                   | `add_texts` yes, `add` no       |
| `upsert_texts` / `upsert` | inserts a new row                | replaces the row (versioned-update) | `upsert_texts` yes, `upsert` no |
| `update_texts`            | — targets existing rows          | replaces the row (versioned-update) | yes                             |
| `delete`                  | —                                | removes the id from the live set    | no                              |

`add*` is strict: it refuses to clobber an existing id, so double-ingestion fails loudly. `upsert*` is insert-or-replace. `update_texts` refreshes the vector and payload of an id you know exists.

```python
from telys import Telys
from telys.embedding import AlgentaBigramEmbedder

db = Telys("./data")
emb = AlgentaBigramEmbedder()
notes = db.create_collection(
    "notes",
    dim=emb.profile.dimension,    # kernel-sourced vector dimension — read it, never hard-code
    partition_by="scope",
    embedder=emb,
)

# First batch — this seals the base image of the collection.
notes.add_texts(
    ["kickoff notes", "design review"],
    ids=["n1", "n2"],
    metadata=[{"scope": "team"}, {"scope": "team"}],
)

# A later batch layers deltas on top of the base:
#   "n3" is new         -> delta-insert
#   "n1" already exists  -> versioned-update (supersedes the base row for n1)
notes.upsert_texts(
    ["retro notes", "kickoff notes (revised)"],
    ids=["n3", "n1"],
    metadata=[{"scope": "team"}, {"scope": "team"}],
)

notes.save()
print(len(notes))                # 3 — n1 (revised), n2, n3
```

## Soft-delete

`delete(ids)` removes only the ids present and ignores unknown ones — passing a missing id is not an error. The effect is immediate: a deleted id disappears from `ids()`, `search`/`search_text` results, `len(col)`, and the `in` operator.

```python
notes.delete(["n2"])

print("n2" in notes)             # False
print(len(notes))                # 2
print(notes.ids(where={"scope": "team"}))   # ['n1', 'n3']
```

The delete is **logical**: the row leaves the live set at once, but its bytes in the base slab are not reclaimed until you compact. No query returns it, yet the physical footprint stays until a rewrite.

## Compaction

`compact()` rewrites the collection into a fresh base: versioned-updates are folded in, superseded and deleted rows are dropped, and their space is reclaimed. It does **not** change what queries return — the live set is identical before and after — it only shrinks and tidies the on-disk representation.

```python
notes.compact()                  # merge deltas, drop superseded/deleted rows
notes.save()
print(notes.stats())             # runtime stats + name, key_name, external_ids
```

Run `compact()` after a large batch of upserts or deletes, or on a schedule for long-lived collections. Between compactions, high churn just stacks more deltas over the base — correctness is unaffected, only physical size and scan cost.

{% hint style="info" %}
Reach for `stats()` and `len(col)` to see the effect. `len(col)` is the count of live external ids; `stats()` reports the underlying runtime counters alongside `name`, `key_name`, and `external_ids`.
{% endhint %}

## Persistence: save() is the commit point

None of the above reaches disk until you call `save()`. `create_collection` builds and caches the collection in memory; `save()` writes the `id_map.npy` sidecar first and then `collection.json` **last** — that final write is the atomic commit point, so a crash mid-save never leaves a half-written `collection.json`. Only saved collections appear in `db.collections()`.

`save()` persists the collection's identity and durable state: `name`, `dim`, `dtype`, `key_name`, `filter_columns`, the id-map sidecar, `telys_version`, `default_target_recall`, `applied_tuner`, and the `embedder_profile`.

{% hint style="warning" %}
Over a server connection, `RemoteCollection` is a subset of the local API: it exposes `add`, `upsert`, `add_texts`, `upsert_texts`, `delete`, `compact`, and `save`, but **not** `update_texts`. To version an existing row remotely, use `upsert_texts`.
{% endhint %}

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Update records</strong></td><td>Refresh vectors and payloads with upsert and update_texts.</td><td><a href="/pages/IEihttmP8r8zxpFDyjgd">/pages/IEihttmP8r8zxpFDyjgd</a></td></tr><tr><td><strong>Delete &#x26; compact</strong></td><td>Remove ids and reclaim their space, step by step.</td><td><a href="/pages/RllSb4oFIVsEKxs8s4a3">/pages/RllSb4oFIVsEKxs8s4a3</a></td></tr><tr><td><strong>Snapshots &#x26; persistence</strong></td><td>What save() commits and how to snapshot a collection.</td><td><a href="/pages/3IabEB7rmGWO2uCD2t8X">/pages/3IabEB7rmGWO2uCD2t8X</a></td></tr><tr><td><strong>Roadmap</strong></td><td>Where temporal and validity-window queries stand.</td><td><a href="/pages/8U9vULgHgKKQYoaFcNte">/pages/8U9vULgHgKKQYoaFcNte</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.telys.ai/understand-telys/versioning-soft-delete-compaction.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.