rony-chat-bot/docs/vps-context-sizing.md

# Context window sizing — reference

Quick reference for picking `context_size` and `max_tokens` in
`configs/portfolio-bot.yaml` based on the host's RAM budget. Math,
recommendations, and tips.

---

## 1. `context_size` vs `max_tokens`

Two different budgets in the provider config:

```yaml
context_size: 4096   # total window (input + output)
max_tokens: 2048     # generation cap per response
```

```
context_size (4096) = system prompt + RAG + history + respuesta
                      └────────── input ──────────┘  └output┘
                                                   max_tokens
```

- **`context_size`** → total tokens the model can see + produce. Maps to
  llama-server's `--ctx-size`.
- **`max_tokens`** → cap on **generation** per response. Doesn't affect
  how much input fits, only how long the answer can be.

Rule of thumb for a Q&A bot: 512–1024 `max_tokens` is plenty. Bigger just
steals budget from the input side, where the auto-compactor then has to
fire sooner.

---

## 2. KV cache math

The constraint on context size is **KV cache RAM**, not the model's
advertised window. KV cache grows linearly with context and is held in
RAM per active stream:

```
KV cache (bytes) ≈ 2 × num_layers × num_kv_heads × head_dim × bytes × context_size
```

Reference values for the models the bot is usually paired with:

| Model             | num_layers | num_kv_heads | head_dim | KB/token |
|-------------------|-----------:|-------------:|---------:|---------:|
| qwen2.5-1.5b      |         28 |            2 |      128 |       ~18 |
| qwen2.5-3b        |         36 |            4 |      128 |       ~72 |
| gemma-3-1b        |         18 |            1 |      256 |       ~36 |
| gemma-3-4b        |         34 |            4 |      256 |      ~272 |

Q4_K_M model weights (also RAM-resident):

| Model             | Size    |
|-------------------|--------:|
| qwen2.5-1.5b      | ~1.0 GB |
| qwen2.5-3b        | ~2.0 GB |
| gemma-3-1b        | ~0.8 GB |
| gemma-3-4b        | ~2.5 GB |

---

## 3. RAM budget on a single-model VPS

Fixed cost before we pick a context size:

```
Sistema + Go binary + SQLite     ~0.5 GB
Modelo Q4_K_M weights            (ver tabla arriba)
Buffer para picos y tmpfs        ~0.5 GB
```

Available for **KV cache + headroom** = `RAM_total − 0.5 GB − modelo`.

---

## 4. Recommendations by RAM

### 4 GB VPS (bare-bones dev)

Solo viable con modelo chico y contexto bajo.

| Model             | context_size | KV cache | RAM usada |
|-------------------|-------------:|---------:|----------:|
| gemma-3-1b Q4     |         8192 |  ~290 MB |    ~1.8 GB |
| qwen2.5-1.5b Q4   |         4096 |   ~72 MB |    ~1.6 GB |

### 8 GB VPS (típico)

> The estimates in this section predate the measurements below and were made
> on a machine with a GPU. They understate CPU-only RSS by 1–3 GB and their
> "recommended" contexts are far larger than this bot ever uses. Treat the
> **Measured RSS** section as authoritative and this one as the KV-cache
> arithmetic only.

| Model             | context_size | KV cache | Verdict                        |
|-------------------|-------------:|---------:|--------------------------------|
| qwen2.5-3b Q4     |     **4096** |  ~290 MB | **the default** — 3.64 GB measured |
| qwen2.5-3b Q4     |         8192 |  ~580 MB | +212 MB for headroom never used |
| gemma-3-1b Q4     |         4096 |  ~145 MB | cheap, but see the accuracy note |
| gemma-3-4b Q4     |         4096 | ~1.1 GB  | untested here                  |

### Measured RSS (not estimated)

**Measure with `--device none`, or the numbers lie.** llama.cpp initialises a
compiled-in GPU backend even with `-ngl 0`. If the build machine has a GPU it
quietly holds the compute buffers, and the RSS you measure is the RSS you will
*not* get on a GPU-less VPS. The gap is not a rounding error:

| Model                        | with a GPU present | `--device none` |
|------------------------------|-------------------:|----------------:|
| qwen2.5-3b Q4_K_M            |            2.54 GB |     **3.66 GB** |
| granite-4.0-h-tiny (7B-A1B)  |            4.48 GB |     **7.29 GB** |
| gemma-3-1b Q4_K_M            |            1.29 GB |     **1.05 GB** |

Granite looked like it fit an 8 GB box and does not. gemma goes the other way —
its compute buffers are tiny either way, so dropping the GPU runtime is a net
saving.

Real `RSS` on CPU only, `--parallel 1 --threads 2 --mlock`, in steady state
after serving requests:

| Process                                   | ctx  |      RSS |
|-------------------------------------------|-----:|---------:|
| qwen2.5-3b Q4_K_M                         | 8192 |  3.85 GB |
| qwen2.5-3b Q4_K_M                         | 4096 |  3.64 GB |
| gemma-3-1b Q4_K_M                         | 8192 |  1.05 GB |
| granite-4.0-h-tiny Q4_K_M                 | any  |  7.25 GB |
| nomic-embed-v2-moe Q5_K_M (`--embedding`) | 2048 |  0.91 GB |
| the Go bot + SQLite                       |    — |  0.02 GB |

Two things worth noting from that table:

- **Cold RSS understates it.** qwen at 4096 loads at 3.51 GB and settles at
  3.64 GB after ten requests. Budget from the steady figure.
- **Granite ignores `--ctx-size` entirely** (7.24 GB at 2048, 7.29 GB at 8192).
  It is a hybrid Mamba model: the recurrent state is fixed-size, so a 1M-token
  window is nearly free — and there is no context lever to pull when it doesn't
  fit.

**Sizing on a shared VPS.** If the box also serves other sites, budget backwards
from what they need. On an 8 GB VPS the full hybrid stack (qwen2.5-3b at 4096 +
embedder + bot) is **4.6 GB**, leaving ~3.4 GB.

KV quantization buys less than people expect — 130 MB on qwen2.5-3b at 8192 —
because the weights dominate. Reach for a smaller model, or a smaller context,
before reaching for `--cache-type-*`.

### 16 GB VPS

Estimated, not measured — and the same GPU caveat applies, so add 1–2 GB for a
CPU-only host. Contexts this large are also well past anything this bot builds
(1255 tokens measured); they only matter if you repurpose it for long documents.

| Model             | context_size | KV cache | RAM usada (est.) |
|-------------------|-------------:|---------:|-----------------:|
| qwen2.5-3b Q4     |        32768 | ~2.3 GB  |          ~5.0 GB |
| gemma-3-4b Q4     |        16384 | ~4.5 GB  |          ~7.5 GB |

---

## 5. Worked examples

### qwen2.5-3b on a shared 8 GB VPS — the shipped default

```yaml
# configs/portfolio-bot.yaml
providers:
  - name: llamacpp-local
    type: llamacpp
    model: qwen2.5-3b-instruct
    endpoint: http://localhost:9100/v1
    context_size: 4096      # 3.64 GB measured; largest real prompt was 1255 tokens
    max_tokens: 640
    temperature: 0.7
    top_k: 20
    top_p: 0.8
    repeat_penalty: 1.05
```

```bash
llama-server \
  -m qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
```

Plus the embedder, which has to stay resident because every visitor question
must be embedded before it can be compared:

```bash
llama-server \
  -m nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2
```

Total: 3.64 + 0.91 + 0.02 = **4.57 GB**.

### gemma-3-1b — cheaper, and why it isn't the default

```bash
llama-server \
  -m gemma-3-1b-it-Q4_K_M.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.15
```

1.05 GB instead of 3.64 — 2.8 GB cheaper and ~35% faster. On a 20-question
bilingual battery against the real corpus, with identical hybrid retrieval, it
scored **3/10** on grounded content against qwen's **9/10**, and among its
failures it echoed the system prompt's own instructions back to the visitor.
The sampling flags above matter: Google's published config is `temp 1.0 /
top_k 64 / top_p 0.95 / min_p 0.0`, and `--repeat-penalty 1.15` is deliberately
off-spec because at Google's recommended 1.0 the model looped on the repetitive
shape of the retrieved-chunk headers.

### Gemma con chat template custom (sin system role líder)

Gemma 3 rechaza mensajes `system` antes del primer `user`. Dos opciones:

**Opción A** — template custom en `~/.llama/gemma3.jinja`:

```jinja
{% if messages[0]['role'] != 'system' and messages[0]['role'] != 'user' %}
{{ raise_exception('First message must be system or user') }}
{% endif %}
{% for message in messages %}
{% if message['role'] == 'system' %}
{{ message['content'] | trim + '\n\n' -}}
{% elif message['role'] == 'user' %}
{{- '<start_of_turn>user\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% elif message['role'] == 'assistant' or message['role'] == 'model' %}
{{- '<start_of_turn>model\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
{{- '<start_of_turn>model\n' -}}
{% endif %}
```

```bash
llama-server \
  -m gemma-3-1b-it-Q4_K_M.gguf \
  --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --chat-template-file ~/.llama/gemma3.jinja
```

**Opción B** — usar Ollama, que mapea system → prefix del primer user
automáticamente.

---

## 6. Tuning with auto-compaction

The bot has built-in auto-compaction (`configs/portfolio-bot.yaml` →
`compaction:` block). When the previous turn's input tokens exceed
`threshold_ratio × MaxContextWindow`, the older portion of the chat gets
summarized into a single system note. This means:

- A **smaller `context_size`** still works for long conversations — the
  compactor frees up room by folding old turns.
- **Bigger `max_tokens`** means the compactor fires sooner (less budget
  left for input).
- Default `threshold_ratio: 0.75` triggers compaction at ~75% of the
  window. Lower it (e.g. `0.5`) for headroom on slow CPU where each
  request is expensive; raise it (e.g. `0.9`) when you want to keep
  more verbatim history.

For this bot at `context_size: 4096` and `max_tokens: 640`, compaction fires
when input exceeds ~3070 tokens. Measured over 20 real requests the largest
prompt was 1255 tokens, so in practice it never fires on a single-question
visit — it exists for the chatty visitor who keeps a thread going.

---

## 7. Tips

1. **`--mlock`** es oro en VPS. Bloquea el modelo en RAM y evita swaps
   cuando hay picos de memoria. Cuesta ~modelo_size de locked RAM.
2. **Monitoreá con `htop`** o `free -h` la primera semana. Si ves swap,
   bajá el context.
3. **Más contexto ≠ más rápido.** El prefill (procesar el input) escala
   lineal con la cantidad de tokens. Generación (output) no se ve
   afectada. Con `--ctx-size 4096` y un input de 500 tokens, el TTFT apenas
   cambia; con 20k tokens de input sí. Medido: bajar de 8192 a 4096 dejó la
   generación exactamente igual (21,0 tok/s), porque el prefill escala con los
   tokens que procesás, no con los que reservás.
4. **Streams concurrentes.** Cada stream activo reserva su propio KV
   cache. En 8 GB no hagas más de 1-2 streams simultáneos — el rate
   limiter del bot (default 30 req/min) ya te protege.
5. **`max_tokens` bajo ayuda.** 512 es suficiente para Q&A. Bajarlo
   deja más presupuesto para input y retrasa la compactación.

---

## 8. Quick-pick table

Start from what the prompt actually costs, not from what the model can hold.
Measured on this bot over 20 real requests: the largest prompt ever built was
**1255 tokens** (system prompt + project catalogue + 5 retrieved chunks + the
question), median 1069. Compaction begins at `threshold_ratio` × the window, so
4096 leaves 2.4x headroom before it even engages.

| Setup                        | `context_size` | `max_tokens` | Note |
|------------------------------|---------------:|-------------:|------|
| 8 GB shared + qwen2.5-3b     |       **4096** |      **640** | the default; 4.6 GB total stack |
| 8 GB dedicated + qwen2.5-3b  |           8192 |          640 | +212 MB, no measured benefit |
| 8 GB + gemma-3-1b            |           4096 |          640 | 2.8 GB cheaper, and 3/10 vs 9/10 on grounded answers — see the provider comment in the config |
| 16 GB + qwen2.5-3b           |           8192 |         1024 | room for longer threads |

Going above 8192 for a portfolio bot is reserving memory you will not use. A
bigger window does not make answers better; it makes the KV cache bigger and
delays compaction that was never going to trigger.

`max_tokens: 640` is sized for the answers this persona is asked to give (2–4
sentences plus a short list). Raising it takes budget from the input side and
makes compaction fire sooner.