# Context window sizing — reference

Quick reference for picking `context_size` and `max_tokens` in
`configs/portfolio-bot.yaml` based on the host's RAM budget. Math,
recommendations, and tips.

---

## 1. `context_size` vs `max_tokens`

Two different budgets in the provider config:

```yaml
context_size: 4096   # total window (input + output)
max_tokens: 2048     # generation cap per response
```

```
context_size (4096) = system prompt + RAG + history + respuesta
                      └────────── input ──────────┘  └output┘
                                                   max_tokens
```

- **`context_size`** → total tokens the model can see + produce. Maps to
  llama-server's `--ctx-size`.
- **`max_tokens`** → cap on **generation** per response. Doesn't affect
  how much input fits, only how long the answer can be.

Rule of thumb for a Q&A bot: 512–1024 `max_tokens` is plenty. Bigger just
steals budget from the input side, where the auto-compactor then has to
fire sooner.

---

## 2. KV cache math

The constraint on context size is **KV cache RAM**, not the model's
advertised window. KV cache grows linearly with context and is held in
RAM per active stream:

```
KV cache (bytes) ≈ 2 × num_layers × num_kv_heads × head_dim × bytes × context_size
```

Reference values for the models the bot is usually paired with:

| Model             | num_layers | num_kv_heads | head_dim | KB/token |
|-------------------|-----------:|-------------:|---------:|---------:|
| qwen2.5-1.5b      |         28 |            2 |      128 |       ~18 |
| qwen2.5-3b        |         36 |            4 |      128 |       ~72 |
| gemma-3-1b        |         18 |            1 |      256 |       ~36 |
| gemma-3-4b        |         34 |            4 |      256 |      ~272 |

Q4_K_M model weights (also RAM-resident):

| Model             | Size    |
|-------------------|--------:|
| qwen2.5-1.5b      | ~1.0 GB |
| qwen2.5-3b        | ~2.0 GB |
| gemma-3-1b        | ~0.8 GB |
| gemma-3-4b        | ~2.5 GB |

---

## 3. RAM budget on a single-model VPS

Fixed cost before we pick a context size:

```
Sistema + Go binary + SQLite     ~0.5 GB
Modelo Q4_K_M weights            (ver tabla arriba)
Buffer para picos y tmpfs        ~0.5 GB
```

Available for **KV cache + headroom** = `RAM_total − 0.5 GB − modelo`.

---

## 4. Recommendations by RAM

### 4 GB VPS (bare-bones dev)

Solo viable con modelo chico y contexto bajo.

| Model             | context_size | KV cache | RAM usada |
|-------------------|-------------:|---------:|----------:|
| gemma-3-1b Q4     |         8192 |  ~290 MB |    ~1.8 GB |
| qwen2.5-1.5b Q4   |         4096 |   ~72 MB |    ~1.6 GB |

### 8 GB VPS (típico)

| Model             | context_size | KV cache | RAM usada | Veredicto           |
|-------------------|-------------:|---------:|----------:|---------------------|
| qwen2.5-1.5b Q4   |        16384 |  ~290 MB |    ~2.0 GB | muy cómodo          |
| qwen2.5-3b Q4     |         8192 |  ~580 MB |    ~3.0 GB | **sweet spot**      |
| qwen2.5-3b Q4     |        16384 | ~1.1 GB  |    ~3.6 GB | **recomendado**     |
| qwen2.5-3b Q4     |        32768 | ~2.3 GB  |    ~4.8 GB | máximo útil         |
| gemma-3-1b Q4     |        32768 | ~1.1 GB  |    ~2.4 GB | **recomendado**     |
| gemma-3-4b Q4     |         8192 | ~2.2 GB  |    ~5.2 GB | ajustado            |

### 16 GB VPS

| Model             | context_size | KV cache | RAM usada |
|-------------------|-------------:|---------:|----------:|
| qwen2.5-3b Q4     |        32768 | ~2.3 GB  |    ~5.0 GB |
| gemma-3-4b Q4     |        16384 | ~4.5 GB  |    ~7.5 GB |

---

## 5. Worked examples

### qwen2.5-3b en 8 GB

```yaml
# configs/portfolio-bot.yaml
providers:
  - name: llamacpp-local
    type: llamacpp
    model: qwen2.5-3b-instruct
    endpoint: http://localhost:9100/v1
    context_size: 16384     # ~1.1 GB KV, deja 4 GB libres
    max_tokens: 1024        # respuestas moderadas
```

```bash
llama-server \
  -m qwen2.5-3b-instruct-q4_k_m.gguf \
  --ctx-size 16384 \
  -ngl 0 -t 2 \
  --mlock
```

### gemma-3-1b en 8 GB

```yaml
providers:
  - name: llamacpp-local
    type: llamacpp
    model: gemma-3-1b-it
    endpoint: http://localhost:9100/v1
    context_size: 32768     # sobra RAM, contexto largo
    max_tokens: 1024
```

```bash
llama-server \
  -m gemma-3-1b-it-Q4_K_M.gguf \
  --ctx-size 32768 \
  -ngl 0 -t 2 \
  --mlock
```

### Gemma con chat template custom (sin system role líder)

Gemma 3 rechaza mensajes `system` antes del primer `user`. Dos opciones:

**Opción A** — template custom en `~/.llama/gemma3.jinja`:

```jinja
{% if messages[0]['role'] != 'system' and messages[0]['role'] != 'user' %}
{{ raise_exception('First message must be system or user') }}
{% endif %}
{% for message in messages %}
{% if message['role'] == 'system' %}
{{ message['content'] | trim + '\n\n' -}}
{% elif message['role'] == 'user' %}
{{- '<start_of_turn>user\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% elif message['role'] == 'assistant' or message['role'] == 'model' %}
{{- '<start_of_turn>model\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
{{- '<start_of_turn>model\n' -}}
{% endif %}
```

```bash
llama-server \
  -m gemma-3-1b-it-Q4_K_M.gguf \
  --ctx-size 32768 \
  -ngl 0 -t 2 \
  --mlock \
  --chat-template-file ~/.llama/gemma3.jinja
```

**Opción B** — usar Ollama, que mapea system → prefix del primer user
automáticamente.

---

## 6. Tuning with auto-compaction

The bot has built-in auto-compaction (`configs/portfolio-bot.yaml` →
`compaction:` block). When the previous turn's input tokens exceed
`threshold_ratio × MaxContextWindow`, the older portion of the chat gets
summarized into a single system note. This means:

- A **smaller `context_size`** still works for long conversations — the
  compactor frees up room by folding old turns.
- **Bigger `max_tokens`** means the compactor fires sooner (less budget
  left for input).
- Default `threshold_ratio: 0.75` triggers compaction at ~75% of the
  window. Lower it (e.g. `0.5`) for headroom on slow CPU where each
  request is expensive; raise it (e.g. `0.9`) when you want to keep
  more verbatim history.

For a portfolio bot with `context_size: 16384` and `max_tokens: 1024`,
compaction fires when input exceeds ~12k tokens — leaving ~5k for the
fresh history, which is ~10-15 recent user turns. More than enough.

---

## 7. Tips

1. **`--mlock`** es oro en VPS. Bloquea el modelo en RAM y evita swaps
   cuando hay picos de memoria. Cuesta ~modelo_size de locked RAM.
2. **Monitoreá con `htop`** o `free -h` la primera semana. Si ves swap,
   bajá el context.
3. **Más contexto ≠ más rápido.** El prefill (procesar el input) escala
   lineal con la cantidad de tokens. Generación (output) no se ve
   afectada. Con `--ctx-size 32768` y un input de 500 tokens, el TTFT
   apenas cambia; con 20k tokens de input sí.
4. **Streams concurrentes.** Cada stream activo reserva su propio KV
   cache. En 8 GB no hagas más de 1-2 streams simultáneos — el rate
   limiter del bot (default 30 req/min) ya te protege.
5. **`max_tokens` bajo ayuda.** 512 es suficiente para Q&A. Bajarlo
   deja más presupuesto para input y retrasa la compactación.

---

## 8. Quick-pick table

Copy-paste según tu setup:

| Setup                       | `context_size` | `max_tokens` |
|-----------------------------|---------------:|-------------:|
| 4 GB + gemma-3-1b           |           8192 |          512 |
| 4 GB + qwen2.5-1.5b         |           4096 |          512 |
| 8 GB + qwen2.5-1.5b         |          16384 |          768 |
| 8 GB + qwen2.5-3b           |          16384 |         1024 |
| 8 GB + gemma-3-1b           |          32768 |         1024 |
| 16 GB + qwen2.5-3b          |          32768 |         1024 |
| 16 GB + gemma-3-4b          |          16384 |         1024 |