# Context window sizing — reference Quick reference for picking `context_size` and `max_tokens` in `configs/portfolio-bot.yaml` based on the host's RAM budget. Math, recommendations, and tips. --- ## 1. `context_size` vs `max_tokens` Two different budgets in the provider config: ```yaml context_size: 4096 # total window (input + output) max_tokens: 2048 # generation cap per response ``` ``` context_size (4096) = system prompt + RAG + history + respuesta └────────── input ──────────┘ └output┘ max_tokens ``` - **`context_size`** → total tokens the model can see + produce. Maps to llama-server's `--ctx-size`. - **`max_tokens`** → cap on **generation** per response. Doesn't affect how much input fits, only how long the answer can be. Rule of thumb for a Q&A bot: 512–1024 `max_tokens` is plenty. Bigger just steals budget from the input side, where the auto-compactor then has to fire sooner. --- ## 2. KV cache math The constraint on context size is **KV cache RAM**, not the model's advertised window. KV cache grows linearly with context and is held in RAM per active stream: ``` KV cache (bytes) ≈ 2 × num_layers × num_kv_heads × head_dim × bytes × context_size ``` Reference values for the models the bot is usually paired with: | Model | num_layers | num_kv_heads | head_dim | KB/token | |-------------------|-----------:|-------------:|---------:|---------:| | qwen2.5-1.5b | 28 | 2 | 128 | ~18 | | qwen2.5-3b | 36 | 4 | 128 | ~72 | | gemma-3-1b | 18 | 1 | 256 | ~36 | | gemma-3-4b | 34 | 4 | 256 | ~272 | Q4_K_M model weights (also RAM-resident): | Model | Size | |-------------------|--------:| | qwen2.5-1.5b | ~1.0 GB | | qwen2.5-3b | ~2.0 GB | | gemma-3-1b | ~0.8 GB | | gemma-3-4b | ~2.5 GB | --- ## 3. RAM budget on a single-model VPS Fixed cost before we pick a context size: ``` Sistema + Go binary + SQLite ~0.5 GB Modelo Q4_K_M weights (ver tabla arriba) Buffer para picos y tmpfs ~0.5 GB ``` Available for **KV cache + headroom** = `RAM_total − 0.5 GB − modelo`. --- ## 4. Recommendations by RAM ### 4 GB VPS (bare-bones dev) Solo viable con modelo chico y contexto bajo. | Model | context_size | KV cache | RAM usada | |-------------------|-------------:|---------:|----------:| | gemma-3-1b Q4 | 8192 | ~290 MB | ~1.8 GB | | qwen2.5-1.5b Q4 | 4096 | ~72 MB | ~1.6 GB | ### 8 GB VPS (típico) > The estimates in this section predate the measurements below and were made > on a machine with a GPU. They understate CPU-only RSS by 1–3 GB and their > "recommended" contexts are far larger than this bot ever uses. Treat the > **Measured RSS** section as authoritative and this one as the KV-cache > arithmetic only. | Model | context_size | KV cache | Verdict | |-------------------|-------------:|---------:|--------------------------------| | qwen2.5-3b Q4 | **4096** | ~290 MB | **the default** — 3.64 GB measured | | qwen2.5-3b Q4 | 8192 | ~580 MB | +212 MB for headroom never used | | gemma-3-1b Q4 | 4096 | ~145 MB | cheap, but see the accuracy note | | gemma-3-4b Q4 | 4096 | ~1.1 GB | untested here | ### Measured RSS (not estimated) **Measure with `--device none`, or the numbers lie.** llama.cpp initialises a compiled-in GPU backend even with `-ngl 0`. If the build machine has a GPU it quietly holds the compute buffers, and the RSS you measure is the RSS you will *not* get on a GPU-less VPS. The gap is not a rounding error: | Model | with a GPU present | `--device none` | |------------------------------|-------------------:|----------------:| | qwen2.5-3b Q4_K_M | 2.54 GB | **3.66 GB** | | granite-4.0-h-tiny (7B-A1B) | 4.48 GB | **7.29 GB** | | gemma-3-1b Q4_K_M | 1.29 GB | **1.05 GB** | Granite looked like it fit an 8 GB box and does not. gemma goes the other way — its compute buffers are tiny either way, so dropping the GPU runtime is a net saving. Real `RSS` on CPU only, `--parallel 1 --threads 2 --mlock`, in steady state after serving requests: | Process | ctx | RSS | |-------------------------------------------|-----:|---------:| | qwen2.5-3b Q4_K_M | 8192 | 3.85 GB | | qwen2.5-3b Q4_K_M | 4096 | 3.64 GB | | gemma-3-1b Q4_K_M | 8192 | 1.05 GB | | granite-4.0-h-tiny Q4_K_M | any | 7.25 GB | | nomic-embed-v2-moe Q5_K_M (`--embedding`) | 2048 | 0.91 GB | | the Go bot + SQLite | — | 0.02 GB | Two things worth noting from that table: - **Cold RSS understates it.** qwen at 4096 loads at 3.51 GB and settles at 3.64 GB after ten requests. Budget from the steady figure. - **Granite ignores `--ctx-size` entirely** (7.24 GB at 2048, 7.29 GB at 8192). It is a hybrid Mamba model: the recurrent state is fixed-size, so a 1M-token window is nearly free — and there is no context lever to pull when it doesn't fit. **Sizing on a shared VPS.** If the box also serves other sites, budget backwards from what they need. On an 8 GB VPS the full hybrid stack (qwen2.5-3b at 4096 + embedder + bot) is **4.6 GB**, leaving ~3.4 GB. KV quantization buys less than people expect — 130 MB on qwen2.5-3b at 8192 — because the weights dominate. Reach for a smaller model, or a smaller context, before reaching for `--cache-type-*`. ### 16 GB VPS Estimated, not measured — and the same GPU caveat applies, so add 1–2 GB for a CPU-only host. Contexts this large are also well past anything this bot builds (1255 tokens measured); they only matter if you repurpose it for long documents. | Model | context_size | KV cache | RAM usada (est.) | |-------------------|-------------:|---------:|-----------------:| | qwen2.5-3b Q4 | 32768 | ~2.3 GB | ~5.0 GB | | gemma-3-4b Q4 | 16384 | ~4.5 GB | ~7.5 GB | --- ## 5. Worked examples ### qwen2.5-3b on a shared 8 GB VPS — the shipped default ```yaml # configs/portfolio-bot.yaml providers: - name: llamacpp-local type: llamacpp model: qwen2.5-3b-instruct endpoint: http://localhost:9100/v1 context_size: 4096 # 3.64 GB measured; largest real prompt was 1255 tokens max_tokens: 640 temperature: 0.7 top_k: 20 top_p: 0.8 repeat_penalty: 1.05 ``` ```bash llama-server \ -m qwen2.5-3b-instruct-q4_k_m.gguf \ --port 9100 --ctx-size 4096 --parallel 1 \ --device none --threads 2 --mlock \ --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05 ``` Plus the embedder, which has to stay resident because every visitor question must be embedded before it can be compared: ```bash llama-server \ -m nomic-embed-v2-moe.Q5_K_M.gguf \ --port 9200 --embedding --pooling mean \ --ctx-size 2048 --parallel 1 --device none --threads 2 ``` Total: 3.64 + 0.91 + 0.02 = **4.57 GB**. ### gemma-3-1b — cheaper, and why it isn't the default ```bash llama-server \ -m gemma-3-1b-it-Q4_K_M.gguf \ --port 9100 --ctx-size 4096 --parallel 1 \ --device none --threads 2 --mlock \ --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.15 ``` 1.05 GB instead of 3.64 — 2.8 GB cheaper and ~35% faster. On a 20-question bilingual battery against the real corpus, with identical hybrid retrieval, it scored **3/10** on grounded content against qwen's **9/10**, and among its failures it echoed the system prompt's own instructions back to the visitor. The sampling flags above matter: Google's published config is `temp 1.0 / top_k 64 / top_p 0.95 / min_p 0.0`, and `--repeat-penalty 1.15` is deliberately off-spec because at Google's recommended 1.0 the model looped on the repetitive shape of the retrieved-chunk headers. ### Gemma con chat template custom (sin system role líder) Gemma 3 rechaza mensajes `system` antes del primer `user`. Dos opciones: **Opción A** — template custom en `~/.llama/gemma3.jinja`: ```jinja {% if messages[0]['role'] != 'system' and messages[0]['role'] != 'user' %} {{ raise_exception('First message must be system or user') }} {% endif %} {% for message in messages %} {% if message['role'] == 'system' %} {{ message['content'] | trim + '\n\n' -}} {% elif message['role'] == 'user' %} {{- 'user\n' + message['content'] | trim + '\n' -}} {% elif message['role'] == 'assistant' or message['role'] == 'model' %} {{- 'model\n' + message['content'] | trim + '\n' -}} {% endif %} {% endfor %} {% if add_generation_prompt %} {{- 'model\n' -}} {% endif %} ``` ```bash llama-server \ -m gemma-3-1b-it-Q4_K_M.gguf \ --ctx-size 4096 --parallel 1 \ --device none --threads 2 --mlock \ --chat-template-file ~/.llama/gemma3.jinja ``` **Opción B** — usar Ollama, que mapea system → prefix del primer user automáticamente. --- ## 6. Tuning with auto-compaction The bot has built-in auto-compaction (`configs/portfolio-bot.yaml` → `compaction:` block). When the previous turn's input tokens exceed `threshold_ratio × MaxContextWindow`, the older portion of the chat gets summarized into a single system note. This means: - A **smaller `context_size`** still works for long conversations — the compactor frees up room by folding old turns. - **Bigger `max_tokens`** means the compactor fires sooner (less budget left for input). - Default `threshold_ratio: 0.75` triggers compaction at ~75% of the window. Lower it (e.g. `0.5`) for headroom on slow CPU where each request is expensive; raise it (e.g. `0.9`) when you want to keep more verbatim history. For this bot at `context_size: 4096` and `max_tokens: 640`, compaction fires when input exceeds ~3070 tokens. Measured over 20 real requests the largest prompt was 1255 tokens, so in practice it never fires on a single-question visit — it exists for the chatty visitor who keeps a thread going. --- ## 7. Tips 1. **`--mlock`** es oro en VPS. Bloquea el modelo en RAM y evita swaps cuando hay picos de memoria. Cuesta ~modelo_size de locked RAM. 2. **Monitoreá con `htop`** o `free -h` la primera semana. Si ves swap, bajá el context. 3. **Más contexto ≠ más rápido.** El prefill (procesar el input) escala lineal con la cantidad de tokens. Generación (output) no se ve afectada. Con `--ctx-size 4096` y un input de 500 tokens, el TTFT apenas cambia; con 20k tokens de input sí. Medido: bajar de 8192 a 4096 dejó la generación exactamente igual (21,0 tok/s), porque el prefill escala con los tokens que procesás, no con los que reservás. 4. **Streams concurrentes.** Cada stream activo reserva su propio KV cache. En 8 GB no hagas más de 1-2 streams simultáneos — el rate limiter del bot (default 30 req/min) ya te protege. 5. **`max_tokens` bajo ayuda.** 512 es suficiente para Q&A. Bajarlo deja más presupuesto para input y retrasa la compactación. --- ## 8. Quick-pick table Start from what the prompt actually costs, not from what the model can hold. Measured on this bot over 20 real requests: the largest prompt ever built was **1255 tokens** (system prompt + project catalogue + 5 retrieved chunks + the question), median 1069. Compaction begins at `threshold_ratio` × the window, so 4096 leaves 2.4x headroom before it even engages. | Setup | `context_size` | `max_tokens` | Note | |------------------------------|---------------:|-------------:|------| | 8 GB shared + qwen2.5-3b | **4096** | **640** | the default; 4.6 GB total stack | | 8 GB dedicated + qwen2.5-3b | 8192 | 640 | +212 MB, no measured benefit | | 8 GB + gemma-3-1b | 4096 | 640 | 2.8 GB cheaper, and 3/10 vs 9/10 on grounded answers — see the provider comment in the config | | 16 GB + qwen2.5-3b | 8192 | 1024 | room for longer threads | Going above 8192 for a portfolio bot is reserving memory you will not use. A bigger window does not make answers better; it makes the KV cache bigger and delays compaction that was never going to trigger. `max_tokens: 640` is sized for the answers this persona is asked to give (2–4 sentences plus a short list). Raising it takes budget from the input side and makes compaction fire sooner.