> ## Documentation Index
> Fetch the complete documentation index at: https://toradb.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# On-disk layout

> Database directory, segments, and index sidecars

A ToraDB database is a **directory** you pass to `toradb.local("./my_db")`. Inside, each table has Parquet **segments** and optional **index sidecars**.

## Inspect with DESCRIBE

```bash theme={null}
toradb sql ./examples/_demo_db "DESCRIBE articles"
```

Typical output includes:

* Row count and vector dimension (if any)
* Segment count and paths
* Sidecars: `bm25`, `vectors`, `hnsw`, `diskann`, `segment_*`

## Components

| Component | Purpose |
| - | - |
| Parquet segments | Columnar document storage per flush (`id`, `text`, optional typed metadata columns, `metadata_json`, `embedding`) |
| WAL | Replay safety for recent writes |
| `bm25` sidecar | Sparse lexical index (`TBM3` mmap-friendly `*.bm25.bin`) |
| `{seg}.bm25.lex.bin` | Per-segment term lexicon for query pruning |
| `bm25.route.bin` | Table-level term → segment routing index |
| `hnsw` sidecar | In-memory-style dense graph index |
| `diskann` sidecar | On-disk ANN graph for large embedding tables |
| Materialized views | Cached search/analytics results (see [SQL materialized views](/api-reference/sql/materialized-views)) |

## Persistence model

1. **Ingest** — documents append to the active segment; indexes update incrementally where supported.
2. **Flush** — segments are written as Parquet under the table directory.
3. **Reindex** — rebuild a specific index type (`BM25`, `HNSW`, `DISKANN`) after bulk loads or schema changes.
4. **Compact** — merge small segments to reduce file count (`COMPACT TABLE`; see [Reindex and compact](/guides/reindex-compact)). `COMPACT TABLE … FULL` also migrates legacy JSON-only segments to [native typed metadata columns](/guides/typed-columns).

## Segment-only bulk ingest

Large tables use `index_mode: segment_only` in `manifest.json`: per-segment `*.bm25.bin` (TBM3) and `*.bm25.lex.bin` lexicons are built at finish; optional `query_mode: routed` uses `bm25.route.bin` to skip irrelevant segments at query time.

Environment variables for the demo and CLI:

| Variable | Purpose |
| - | - |
| `TORADB_CACHE_INDEX_BYTES` | Fixed LRU budget for mmap segment BM25 blobs |
| `TORADB_CACHE_AUTO` | Set `1` to size cache from RAM fraction (serving profile) |
| `TORADB_CACHE_INDEX_FRACTION` | RAM fraction for auto cache (default `0.20`) |
| `TORADB_CACHE_AUTO_CAP_BYTES` | Max auto cache bytes (default 8GB) |
| `TORADB_WARMUP_ON_START` | Background search warmup after API start (`1` in serving demo) |
| `TORADB_INGEST_THREADS` | Parallelism for index build |
| `TORADB_PROFILE` | Demo preset: `ingest` or `serving` (`demo/run.sh`) |

See [Production serving profiles](/guides/production-serving) for ingest vs serving presets.

<Tip>
  Run `examples/full_example.py` once to populate `examples/_demo_db`, then use `DESCRIBE` on each table to see real sidecar names.
</Tip>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.