rony-chat-bot/docs/vps-context-sizing.md

Context window sizing — reference

Quick reference for picking context_size and max_tokens in configs/portfolio-bot.yaml based on the host's RAM budget. Math, recommendations, and tips.

---

1. context_size vs max_tokens

Two different budgets in the provider config:

context_size: 4096   # total window (input + output)
max_tokens: 2048     # generation cap per response

context_size (4096) = system prompt + RAG + history + respuesta
                      └────────── input ──────────┘  └output┘
                                                   max_tokens

• context_size → total tokens the model can see + produce. Maps to llama-server's --ctx-size.
• max_tokens → cap on generation per response. Doesn't affect how much input fits, only how long the answer can be.

Rule of thumb for a Q&A bot: 512–1024 max_tokens is plenty. Bigger just steals budget from the input side, where the auto-compactor then has to fire sooner.

---

2. KV cache math

The constraint on context size is KV cache RAM, not the model's advertised window. KV cache grows linearly with context and is held in RAM per active stream:

KV cache (bytes) ≈ 2 × num_layers × num_kv_heads × head_dim × bytes × context_size

Reference values for the models the bot is usually paired with:

Model	num_layers	num_kv_heads	head_dim	KB/token
qwen2.5-1.5b	28	2	128	~18
qwen2.5-3b	36	4	128	~72
gemma-3-1b	18	1	256	~36
gemma-3-4b	34	4	256	~272

Q4_K_M model weights (also RAM-resident):

Model	Size
qwen2.5-1.5b	~1.0 GB
qwen2.5-3b	~2.0 GB
gemma-3-1b	~0.8 GB
gemma-3-4b	~2.5 GB

---

3. RAM budget on a single-model VPS

Fixed cost before we pick a context size:

Sistema + Go binary + SQLite     ~0.5 GB
Modelo Q4_K_M weights            (ver tabla arriba)
Buffer para picos y tmpfs        ~0.5 GB

Available for KV cache + headroom = RAM_total − 0.5 GB − modelo.

---

4. Recommendations by RAM

4 GB VPS (bare-bones dev)

Solo viable con modelo chico y contexto bajo.

Model	context_size	KV cache	RAM usada
gemma-3-1b Q4	8192	~290 MB	~1.8 GB
qwen2.5-1.5b Q4	4096	~72 MB	~1.6 GB

8 GB VPS (típico)

The estimates in this section predate the measurements below and were made on a machine with a GPU. They understate CPU-only RSS by 1–3 GB and their "recommended" contexts are far larger than this bot ever uses. Treat the Measured RSS section as authoritative and this one as the KV-cache arithmetic only.

Model	context_size	KV cache	Verdict
qwen2.5-3b Q4	4096	~290 MB	the default — 3.64 GB measured
qwen2.5-3b Q4	8192	~580 MB	+212 MB for headroom never used
gemma-3-1b Q4	4096	~145 MB	cheap, but see the accuracy note
gemma-3-4b Q4	4096	~1.1 GB	untested here

Measured RSS (not estimated)

Measure with --device none, or the numbers lie. llama.cpp initialises a compiled-in GPU backend even with -ngl 0. If the build machine has a GPU it quietly holds the compute buffers, and the RSS you measure is the RSS you will not get on a GPU-less VPS. The gap is not a rounding error:

Model	with a GPU present	--device none
qwen2.5-3b Q4_K_M	2.54 GB	3.66 GB
granite-4.0-h-tiny (7B-A1B)	4.48 GB	7.29 GB
gemma-3-1b Q4_K_M	1.29 GB	1.05 GB

Granite looked like it fit an 8 GB box and does not. gemma goes the other way — its compute buffers are tiny either way, so dropping the GPU runtime is a net saving.

Real RSS on CPU only, --parallel 1 --threads 2 --mlock, in steady state after serving requests:

Process	ctx	RSS
qwen2.5-3b Q4_K_M	8192	3.85 GB
qwen2.5-3b Q4_K_M	4096	3.64 GB
gemma-3-1b Q4_K_M	8192	1.05 GB
granite-4.0-h-tiny Q4_K_M	any	7.25 GB
nomic-embed-v2-moe Q5_K_M (--embedding)	2048	0.91 GB
the Go bot + SQLite	—	0.02 GB

Two things worth noting from that table:

• Cold RSS understates it. qwen at 4096 loads at 3.51 GB and settles at 3.64 GB after ten requests. Budget from the steady figure.
• Granite ignores --ctx-size entirely (7.24 GB at 2048, 7.29 GB at 8192). It is a hybrid Mamba model: the recurrent state is fixed-size, so a 1M-token window is nearly free — and there is no context lever to pull when it doesn't fit.

Sizing on a shared VPS. If the box also serves other sites, budget backwards from what they need. On an 8 GB VPS the full hybrid stack (qwen2.5-3b at 4096 + embedder + bot) is 4.6 GB, leaving ~3.4 GB.

KV quantization buys less than people expect — 130 MB on qwen2.5-3b at 8192 — because the weights dominate. Reach for a smaller model, or a smaller context, before reaching for --cache-type-*.

16 GB VPS

Estimated, not measured — and the same GPU caveat applies, so add 1–2 GB for a CPU-only host. Contexts this large are also well past anything this bot builds (1255 tokens measured); they only matter if you repurpose it for long documents.

Model	context_size	KV cache	RAM usada (est.)
qwen2.5-3b Q4	32768	~2.3 GB	~5.0 GB
gemma-3-4b Q4	16384	~4.5 GB	~7.5 GB

---

5. Worked examples

qwen2.5-3b on a shared 8 GB VPS — the shipped default

# configs/portfolio-bot.yaml
providers:
  - name: llamacpp-local
    type: llamacpp
    model: qwen2.5-3b-instruct
    endpoint: http://localhost:9100/v1
    context_size: 4096      # 3.64 GB measured; largest real prompt was 1255 tokens
    max_tokens: 640
    temperature: 0.7
    top_k: 20
    top_p: 0.8
    repeat_penalty: 1.05

llama-server \
  -m qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05

Plus the embedder, which has to stay resident because every visitor question must be embedded before it can be compared:

llama-server \
  -m nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2

Total: 3.64 + 0.91 + 0.02 = 4.57 GB.

gemma-3-1b — cheaper, and why it isn't the default

llama-server \
  -m gemma-3-1b-it-Q4_K_M.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.15

1.05 GB instead of 3.64 — 2.8 GB cheaper and ~35% faster. On a 20-question bilingual battery against the real corpus, with identical hybrid retrieval, it scored 3/10 on grounded content against qwen's 9/10, and among its failures it echoed the system prompt's own instructions back to the visitor. The sampling flags above matter: Google's published config is temp 1.0 / top_k 64 / top_p 0.95 / min_p 0.0, and --repeat-penalty 1.15 is deliberately off-spec because at Google's recommended 1.0 the model looped on the repetitive shape of the retrieved-chunk headers.

Gemma con chat template custom (sin system role líder)

Gemma 3 rechaza mensajes system antes del primer user. Dos opciones:

Opción A — template custom en ~/.llama/gemma3.jinja:

{% if messages[0]['role'] != 'system' and messages[0]['role'] != 'user' %}
{{ raise_exception('First message must be system or user') }}
{% endif %}
{% for message in messages %}
{% if message['role'] == 'system' %}
{{ message['content'] | trim + '\n\n' -}}
{% elif message['role'] == 'user' %}
{{- '<start_of_turn>user\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% elif message['role'] == 'assistant' or message['role'] == 'model' %}
{{- '<start_of_turn>model\n' + message['content'] | trim + '<end_of_turn>\n' -}}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
{{- '<start_of_turn>model\n' -}}
{% endif %}

llama-server \
  -m gemma-3-1b-it-Q4_K_M.gguf \
  --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --chat-template-file ~/.llama/gemma3.jinja

Opción B — usar Ollama, que mapea system → prefix del primer user automáticamente.

---

6. Tuning with auto-compaction

The bot has built-in auto-compaction (configs/portfolio-bot.yaml → compaction: block). When the previous turn's input tokens exceed threshold_ratio × MaxContextWindow, the older portion of the chat gets summarized into a single system note. This means:

• A smaller context_size still works for long conversations — the compactor frees up room by folding old turns.
• Bigger max_tokens means the compactor fires sooner (less budget left for input).
• Default threshold_ratio: 0.75 triggers compaction at ~75% of the window. Lower it (e.g. 0.5) for headroom on slow CPU where each request is expensive; raise it (e.g. 0.9) when you want to keep more verbatim history.

For this bot at context_size: 4096 and max_tokens: 640, compaction fires when input exceeds ~3070 tokens. Measured over 20 real requests the largest prompt was 1255 tokens, so in practice it never fires on a single-question visit — it exists for the chatty visitor who keeps a thread going.

---

7. Tips

1. --mlock es oro en VPS. Bloquea el modelo en RAM y evita swaps cuando hay picos de memoria. Cuesta ~modelo_size de locked RAM.
2. Monitoreá con htop o free -h la primera semana. Si ves swap, bajá el context.
3. Más contexto ≠ más rápido. El prefill (procesar el input) escala lineal con la cantidad de tokens. Generación (output) no se ve afectada. Con --ctx-size 4096 y un input de 500 tokens, el TTFT apenas cambia; con 20k tokens de input sí. Medido: bajar de 8192 a 4096 dejó la generación exactamente igual (21,0 tok/s), porque el prefill escala con los tokens que procesás, no con los que reservás.
4. Streams concurrentes. Cada stream activo reserva su propio KV cache. En 8 GB no hagas más de 1-2 streams simultáneos — el rate limiter del bot (default 30 req/min) ya te protege.
5. max_tokens bajo ayuda. 512 es suficiente para Q&A. Bajarlo deja más presupuesto para input y retrasa la compactación.

---

8. Quick-pick table

Start from what the prompt actually costs, not from what the model can hold. Measured on this bot over 20 real requests: the largest prompt ever built was 1255 tokens (system prompt + project catalogue + 5 retrieved chunks + the question), median 1069. Compaction begins at threshold_ratio × the window, so 4096 leaves 2.4x headroom before it even engages.

Setup	context_size	max_tokens	Note
8 GB shared + qwen2.5-3b	4096	640	the default; 4.6 GB total stack
8 GB dedicated + qwen2.5-3b	8192	640	+212 MB, no measured benefit
8 GB + gemma-3-1b	4096	640	2.8 GB cheaper, and 3/10 vs 9/10 on grounded answers — see the provider comment in the config
16 GB + qwen2.5-3b	8192	1024	room for longer threads

Going above 8192 for a portfolio bot is reserving memory you will not use. A bigger window does not make answers better; it makes the KV cache bigger and delays compaction that was never going to trigger.

max_tokens: 640 is sized for the answers this persona is asked to give (2–4 sentences plus a short list). Raising it takes budget from the input side and makes compaction fire sooner.