victorvargas/rony-chat-bot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

chore: add documents for context sizing

Browse source
This commit is contained in:
Victor Hugo Vargas 2026-07-18 00:32:16 -07:00
parent 71f8fea0b5
commit 672859c9b8
1 changed files with 243 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

243
docs/vps-context-sizing.md Normal file
View file

@ -0,0 +1,243 @@
# Context window sizing — reference
Quick reference for picking `context_size` and `max_tokens` in
`configs/portfolio-bot.yaml` based on the host's RAM budget. Math,
recommendations, and tips.
---
## 1. `context_size` vs `max_tokens`
Two different budgets in the provider config:
```yaml
context_size: 4096 # total window (input + output)
max_tokens: 2048 # generation cap per response
```
```
context_size (4096) = system prompt + RAG + history + respuesta
└────────── input ──────────┘ └output┘
max_tokens
```
- **`context_size`** → total tokens the model can see + produce. Maps to
llama-server's `--ctx-size`.
- **`max_tokens`** → cap on **generation** per response. Doesn't affect
how much input fits, only how long the answer can be.
Rule of thumb for a Q&A bot: 512–1024 `max_tokens` is plenty. Bigger just
steals budget from the input side, where the auto-compactor then has to
fire sooner.
---
## 2. KV cache math
The constraint on context size is **KV cache RAM**, not the model's
advertised window. KV cache grows linearly with context and is held in
RAM per active stream:
```
KV cache (bytes) ≈ 2 × num_layers × num_kv_heads × head_dim × bytes × context_size
```
Reference values for the models the bot is usually paired with:
| Model | num_layers | num_kv_heads | head_dim | KB/token |
|-------------------|-----------:|-------------:|---------:|---------:|
| qwen2.5-1.5b | 28 | 2 | 128 | ~18 |
| qwen2.5-3b | 36 | 4 | 128 | ~72 |
| gemma-3-1b | 18 | 1 | 256 | ~36 |
| gemma-3-4b | 34 | 4 | 256 | ~272 |
Q4_K_M model weights (also RAM-resident):
| Model | Size |
|-------------------|--------:|
| qwen2.5-1.5b | ~1.0 GB |
| qwen2.5-3b | ~2.0 GB |
| gemma-3-1b | ~0.8 GB |
| gemma-3-4b | ~2.5 GB |
---
## 3. RAM budget on a single-model VPS
Fixed cost before we pick a context size:
```
Sistema + Go binary + SQLite ~0.5 GB
Modelo Q4_K_M weights (ver tabla arriba)
Buffer para picos y tmpfs ~0.5 GB
```
Available for **KV cache + headroom** = `RAM_total − 0.5 GB − modelo`.
---
## 4. Recommendations by RAM
### 4 GB VPS (bare-bones dev)
Solo viable con modelo chico y contexto bajo.
| Model | context_size | KV cache | RAM usada |
|-------------------|-------------:|---------:|----------:|
| gemma-3-1b Q4 | 8192 | ~290 MB | ~1.8 GB |
| qwen2.5-1.5b Q4 | 4096 | ~72 MB | ~1.6 GB |
### 8 GB VPS (típico)
| Model | context_size | KV cache | RAM usada | Veredicto |
|-------------------|-------------:|---------:|----------:|---------------------|
| qwen2.5-1.5b Q4 | 16384 | ~290 MB | ~2.0 GB | muy cómodo |
| qwen2.5-3b Q4 | 8192 | ~580 MB | ~3.0 GB | **sweet spot** |
| qwen2.5-3b Q4 | 16384 | ~1.1 GB | ~3.6 GB | **recomendado** |
| qwen2.5-3b Q4 | 32768 | ~2.3 GB | ~4.8 GB | máximo útil |
| gemma-3-1b Q4 | 32768 | ~1.1 GB | ~2.4 GB | **recomendado** |
| gemma-3-4b Q4 | 8192 | ~2.2 GB | ~5.2 GB | ajustado |
### 16 GB VPS
| Model | context_size | KV cache | RAM usada |
|-------------------|-------------:|---------:|----------:|
| qwen2.5-3b Q4 | 32768 | ~2.3 GB | ~5.0 GB |
| gemma-3-4b Q4 | 16384 | ~4.5 GB | ~7.5 GB |
---
## 5. Worked examples
### qwen2.5-3b en 8 GB
```yaml
# configs/portfolio-bot.yaml
providers:
- name: llamacpp-local
type: llamacpp
model: qwen2.5-3b-instruct
endpoint: http://localhost:9100/v1
context_size: 16384 # ~1.1 GB KV, deja 4 GB libres
max_tokens: 1024 # respuestas moderadas
```
```bash
llama-server \
-m qwen2.5-3b-instruct-q4_k_m.gguf \
--ctx-size 16384 \
-ngl 0 -t 2 \
--mlock
```
### gemma-3-1b en 8 GB
```yaml
providers:
- name: llamacpp-local
type: llamacpp
model: gemma-3-1b-it
endpoint: http://localhost:9100/v1
context_size: 32768 # sobra RAM, contexto largo
max_tokens: 1024
```
```bash
llama-server \
-m gemma-3-1b-it-Q4_K_M.gguf \
--ctx-size 32768 \
-ngl 0 -t 2 \
--mlock
```
### Gemma con chat template custom (sin system role líder)
Gemma 3 rechaza mensajes `system` antes del primer `user`. Dos opciones:
**Opción A** — template custom en `~/.llama/gemma3.jinja`:
```jinja
{% if messages[0]['role'] != 'system' and messages[0]['role'] != 'user' %}
{{ raise_exception('First message must be system or user') }}
{% endif %}
{% for message in messages %}
{% if message['role'] == 'system' %}
{{ message['content'] | trim + '\n\n' -}}
{% elif message['role'] == 'user' %}
{{- '<start_of_turn>user\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% elif message['role'] == 'assistant' or message['role'] == 'model' %}
{{- '<start_of_turn>model\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
{{- '<start_of_turn>model\n' -}}
{% endif %}
```
```bash
llama-server \
-m gemma-3-1b-it-Q4_K_M.gguf \
--ctx-size 32768 \
-ngl 0 -t 2 \
--mlock \
--chat-template-file ~/.llama/gemma3.jinja
```
**Opción B** — usar Ollama, que mapea system → prefix del primer user
automáticamente.
---
## 6. Tuning with auto-compaction
The bot has built-in auto-compaction (`configs/portfolio-bot.yaml` →
`compaction:` block). When the previous turn's input tokens exceed
`threshold_ratio × MaxContextWindow`, the older portion of the chat gets
summarized into a single system note. This means:
- A **smaller `context_size`** still works for long conversations — the
compactor frees up room by folding old turns.
- **Bigger `max_tokens`** means the compactor fires sooner (less budget
left for input).
- Default `threshold_ratio: 0.75` triggers compaction at ~75% of the
window. Lower it (e.g. `0.5`) for headroom on slow CPU where each
request is expensive; raise it (e.g. `0.9`) when you want to keep
more verbatim history.
For a portfolio bot with `context_size: 16384` and `max_tokens: 1024`,
compaction fires when input exceeds ~12k tokens — leaving ~5k for the
fresh history, which is ~10-15 recent user turns. More than enough.
---
## 7. Tips
1. **`--mlock`** es oro en VPS. Bloquea el modelo en RAM y evita swaps
cuando hay picos de memoria. Cuesta ~modelo_size de locked RAM.
2. **Monitoreá con `htop`** o `free -h` la primera semana. Si ves swap,
bajá el context.
3. **Más contexto ≠ más rápido.** El prefill (procesar el input) escala
lineal con la cantidad de tokens. Generación (output) no se ve
afectada. Con `--ctx-size 32768` y un input de 500 tokens, el TTFT
apenas cambia; con 20k tokens de input sí.
4. **Streams concurrentes.** Cada stream activo reserva su propio KV
cache. En 8 GB no hagas más de 1-2 streams simultáneos — el rate
limiter del bot (default 30 req/min) ya te protege.
5. **`max_tokens` bajo ayuda.** 512 es suficiente para Q&A. Bajarlo
deja más presupuesto para input y retrasa la compactación.
---
## 8. Quick-pick table
Copy-paste según tu setup:
| Setup | `context_size` | `max_tokens` |
|-----------------------------|---------------:|-------------:|
| 4 GB + gemma-3-1b | 8192 | 512 |
| 4 GB + qwen2.5-1.5b | 4096 | 512 |
| 8 GB + qwen2.5-1.5b | 16384 | 768 |
| 8 GB + qwen2.5-3b | 16384 | 1024 |
| 8 GB + gemma-3-1b | 32768 | 1024 |
| 16 GB + qwen2.5-3b | 32768 | 1024 |
| 16 GB + gemma-3-4b | 16384 | 1024 |