129809067b
Answers were short, sometimes in the wrong language, and occasionally about projects that do not exist. Measured on a 20-question battery against the real corpus in both Spanish and English, this takes grounded content from 3/10 to 9/10 and language matching from 7/10 to 10/10. Retrieval - Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion. Both halves are load-bearing: the corpus is English and visitors ask in Spanish, so the meaningful words score zero. "paga" appears 0 times in a document that says "Payments: Stripe" — the question "¿Con qué se paga en la tienda de ropa?" retrieved nothing at all. Embeddings put all three of that project's chunks on top. RRF ranks by agreement rather than comparing a BM25 score against a cosine, quantities with no shared scale. - internal/embed: OpenAI-compatible embeddings client, unit-normalised so a dot product is the cosine. Reorders by the response `index` field. - Store a content hash beside each vector and skip rows where it no longer matches the chunk. Chunk ids survive body edits, so without this an edited document keeps serving embeddings that describe text that is gone — reproduced live by changing a payment provider and watching the old one keep coming back. - Degrade to keyword-only when the embedder is down instead of failing. Reference documents that are not projects - Index `.mdx` alongside `.md`, and split sources into projects (announced in the catalogue) and reference material (retrievable, never listed). A CV is what someone deciding whether to hire actually reads, and it was unreachable while it lived only in the Astro site — but filing it under projects made the bot list "cv" as one of Victor's works. - Skip each directory's README. `data/projects/README.md` was being indexed, so the catalogue injected into every prompt announced "README" and "README.es" as projects of Victor's. - Exclude frontmatter from retrieval. It is dense metadata in a very short chunk, which makes it a magnet for short queries: a CV's `location:` field answered "¿Dónde ha trabajado Victor?" with a city instead of a work history. - Split oversized sections at `###` before falling back to byte offsets. A CV's Experience section is a list of jobs, and size-splitting cut one mid-word, stranding the employer's name in the previous chunk. Prompt and sampling - Inject the full project catalogue every turn. Top-K search returns the best matching sections, so "list every project" cannot be answered from retrieval alone, and a small model asked to enumerate from partial hits invents the rest. ~10 tokens per project; this is what stopped the invented names. - Wire the sampling parameters the model authors publish (top_k, top_p, min_p, repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them at llama.cpp's defaults produced 16-token stub answers. - Localised system prompt selected by detected language. The English prompt plus "reply in the user's language" answered 1/5 Spanish questions in Spanish; few-shot examples fixed the language but got copied verbatim into real answers. - Fold compaction's system notes into the leading system message. Gemma's chat template rejects a system message that is not first, and the whole request failed with HTTP 400 the moment compaction fired. Configuration and docs - context_size 4096, down from 8192. The largest prompt this bot ever built over 20 real requests was 1255 tokens, compaction starts at ~3070, and the cut saved 212 MB resident with zero truncations and identical throughput. - Correct the RAM figures throughout. They were measured with a GPU absorbing llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs still started gemma-3-1b while the config defaulted to qwen, and neither started the embedder at all. Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder + 0.02 GB bot, 21.0 tok/s steady state. Known and unfixed, so they are not re-filed as new bugs: the model reads dates out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha trabajado Victor?" still answers with projects rather than employers, though "¿En qué empresas ha trabajado?" works. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
239 lines
12 KiB
YAML
239 lines
12 KiB
YAML
# Portfolio Bot Configuration
|
||
|
||
server:
|
||
host: "0.0.0.0"
|
||
port: 7331
|
||
read_timeout_ms: 30000
|
||
cors_origins:
|
||
- "http://localhost:4321" # Astro dev server
|
||
- "http://localhost:8000" # Local widget demo (python http.server)
|
||
- "https://victorvargas.dev" # Production (when it exists)
|
||
rate_limit:
|
||
requests_per_minute: 30 # Per IP
|
||
burst: 5
|
||
|
||
# LLM providers (at least one configured)
|
||
providers:
|
||
# === llama.cpp server (OpenAI-compatible) — DEFAULT ===
|
||
# Production target: 2 CPU cores, 8 GB RAM, no GPU.
|
||
#
|
||
# llama-server \
|
||
# -m /data/Projects/llm-models/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
|
||
# --port 9100 --ctx-size 4096 --parallel 1 \
|
||
# --device none --threads 2 --mlock \
|
||
# --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
|
||
#
|
||
# --device none is not optional on a GPU-less host: llama.cpp brings up a
|
||
# compiled-in GPU backend even with -ngl 0, and with no GPU present those
|
||
# buffers are served from host RAM. Measured on this model: 2.54 GB with a
|
||
# GPU absorbing them, 3.66 GB without. Budget from the second number.
|
||
#
|
||
# --parallel 1: --ctx-size is divided across slots and the default is 4,
|
||
# so without it each request gets a quarter of the window.
|
||
#
|
||
# Why 4096 and not more: measured over 20 real requests, the largest prompt
|
||
# this bot ever built was 1255 tokens — system prompt + catalogue + 5
|
||
# retrieved chunks + the question. 4096 leaves 3x headroom on the worst
|
||
# case, and compaction only starts at 75% of it (~3070 tokens). Going from
|
||
# 8192 to 4096 saved 212 MB of resident memory with zero truncations and
|
||
# identical throughput (21.0 tok/s both ways), because the extra window was
|
||
# never being used.
|
||
#
|
||
# Sampling values are Qwen's published defaults for instruct chat.
|
||
#
|
||
# Why this model: on a 20-question battery against the real corpus, asked
|
||
# in both Spanish and English, it scored 9/10 on grounded content and 10/10
|
||
# on replying in the language it was asked in. gemma-3-1b scored 3/10 and
|
||
# 7/10 on the same battery with the same retrieval — and among its failures
|
||
# it leaked these instructions into a visitor-facing answer ("I don't have
|
||
# that information – do not invent"). gemma needs 2.8 GB less RAM; that is
|
||
# not a trade worth making on a page where people decide whether to hire.
|
||
#
|
||
# Known weaknesses that remain, so nobody re-tests them as new bugs: it
|
||
# reads dates out of the CV correctly but does the arithmetic on them
|
||
# wrong ("Jul 2024 – Jun 2026" reported as three years), and it sometimes
|
||
# attributes a fact to the wrong source file.
|
||
#
|
||
# Throughput: 21.0 tok/s steady-state with 2 threads once the system prompt
|
||
# is warm in the server's prompt cache; ~11 tok/s on the first cold request.
|
||
- name: llamacpp-local
|
||
type: llamacpp
|
||
model: qwen2.5-3b-instruct
|
||
endpoint: http://localhost:9100/v1
|
||
context_size: 4096 # must match --ctx-size
|
||
max_tokens: 640
|
||
temperature: 0.7
|
||
top_k: 20
|
||
top_p: 0.8
|
||
repeat_penalty: 1.05
|
||
default: true
|
||
|
||
# === Ollama (alternative for development without local GGUF) ===
|
||
# Run: ollama serve
|
||
- name: ollama-local
|
||
type: ollama
|
||
model: qwen2.5:1.5b
|
||
endpoint: http://localhost:11434/v1
|
||
|
||
# === Anthropic (if you want quality > privacy) ===
|
||
- name: anthropic-api
|
||
type: anthropic
|
||
model: claude-haiku-4
|
||
api_key_env: ANTHROPIC_API_KEY
|
||
|
||
# RAG: how projects are indexed (SQLite + FTS5 full-text search)
|
||
rag:
|
||
enabled: true
|
||
data_path: ./data/projects # Projects: .md / .mdx. These appear in the catalogue.
|
||
# Reference material that is about Victor but is not a project: his CV, an
|
||
# about page, a FAQ. Indexed and retrievable, never listed as a project.
|
||
#
|
||
# The CV belongs here, not in data_path. It is the document that answers the
|
||
# questions someone hiring actually asks — "does he know Kubernetes?", "where
|
||
# has he worked?" — and none of that is retrievable while it lives only in
|
||
# the Astro site. Symlink it so there is one copy to maintain:
|
||
#
|
||
# mkdir -p data/docs
|
||
# ln -s ../../../portfolio/src/content/cv/cv.mdx data/docs/cv.mdx
|
||
#
|
||
docs_path: ./data/docs
|
||
db_path: ./data/portfolio.db # SQLite database (auto-created)
|
||
top_k: 5 # Chunks (sections), not documents.
|
||
tokenize: unicode61 # FTS5 tokenizer. unicode61 = exact word match (no stemming). Good for ES/EN mixed corpus with explicit headings.
|
||
# Puts the full project list in the system prompt every turn. Top-K search
|
||
# returns the best-matching *sections*, so a broad "list every project"
|
||
# question can never be answered from retrieval alone — and a small model
|
||
# asked to enumerate from partial hits invents the rest. Costs ~10 tokens
|
||
# per project. Measured: this is what stopped the bot naming projects that
|
||
# do not exist.
|
||
include_catalog: true
|
||
|
||
# Semantic retrieval, fused with the FTS5 keyword search above (Reciprocal
|
||
# Rank Fusion). Both are needed:
|
||
#
|
||
# - Keyword search matches words exactly. The corpus is in English and
|
||
# visitors ask in Spanish, so the words carrying the meaning score zero:
|
||
# "paga" appears 0 times in a document that says "Payments: Stripe",
|
||
# "trabajado" 0 times in one that says "worked". Measured on the real
|
||
# corpus, "¿Con qué se paga en la tienda de ropa?" retrieved *nothing*.
|
||
# - Embeddings bridge that gap — the same question put all three
|
||
# tienda-ropa chunks on top — but blur exact rare tokens, where BM25 is
|
||
# sharp.
|
||
#
|
||
# Start the endpoint with (0.61 GB RSS on CPU, measured):
|
||
# llama-server -m nomic-embed-v2-moe.Q5_K_M.gguf --port 9200 \
|
||
# --embedding --pooling mean --ctx-size 2048 --parallel 1 \
|
||
# --device none --threads 2
|
||
#
|
||
# --device none matters on a GPU-less VPS: llama.cpp initialises a GPU
|
||
# backend when one is compiled in even with -ngl 0, and those buffers land
|
||
# in host RAM when there is no GPU to hold them.
|
||
#
|
||
# Turning this off (or stopping the endpoint) degrades to keyword-only
|
||
# search rather than failing requests. Re-run `chat-bot reindex` after
|
||
# enabling it — vectors are built at index time.
|
||
embeddings:
|
||
enabled: true
|
||
endpoint: http://localhost:9200/v1
|
||
model: nomic-embed-v2-moe
|
||
batch_size: 8
|
||
timeout_ms: 120000
|
||
|
||
# Persona: who the bot is
|
||
persona:
|
||
name: "Rony"
|
||
tone: "Honest, cheerful, loyal" # metadata only — the real voice lives in system_prompt
|
||
language: "the user's language" # detect-and-match; do not pin to a language
|
||
|
||
# Base system prompt — Rony's character and rules. The bot appends the project
|
||
# catalogue and then the retrieved excerpts after this text.
|
||
#
|
||
# Written for a 1B-class model, which changes the rules:
|
||
# - Short beats thorough. The previous ~1400-token prompt made answers
|
||
# *worse*; the model spent its attention on the instructions.
|
||
# - No worked examples with project names in them. The old prompt had a
|
||
# sample answer listing invented projects and the model copied those names
|
||
# verbatim into real answers. Style examples are not free.
|
||
# - Never write instructions as "A or B" with a slash. A 1B model prints
|
||
# the slash form literally.
|
||
# - The grounding rule goes last, closest to the answer, where it sticks.
|
||
system_prompt: |
|
||
You are Rony, the digital canine assistant on the portfolio site of Victor Hugo Vargas.
|
||
Visitors come here to learn about his work. Some are deciding whether to hire him.
|
||
|
||
## Voice
|
||
You are a good dog: loyal to Victor, warm, direct, a little dry. You never oversell and you never invent.
|
||
Write in the same language the visitor used. Greet them as "humano" when they write Spanish, or as
|
||
"human" when they write English — only in a greeting, at most once, never in the middle of an answer.
|
||
Use bold for project names and short bullet lists. At most one 🐾, at the end, and never in a
|
||
technical answer.
|
||
Never open with "Okay", "Sure", "Great question", "Here's the response" or "I'd be happy to help".
|
||
Never bark, and never use 🐶 or 🐕.
|
||
|
||
## Substance
|
||
Give a complete answer, never a stub. For a project, spend 2 to 4 sentences on what it does, the
|
||
tech stack, and the interesting engineering problem behind it. A single line is too short.
|
||
Name the project whenever you use a detail from it.
|
||
Finish by offering one concrete next step, phrased as a short question.
|
||
|
||
## Grounding — the most important rule
|
||
The excerpts below are the only source of truth about Victor. Use nothing else about him.
|
||
If they do not answer the question, say plainly that you don't have that information in Victor's
|
||
portfolio, then name something from the catalogue you can talk about instead.
|
||
Never guess about his skills, tools, availability, rates or experience.
|
||
Talk about Victor in the third person. You are not Victor.
|
||
|
||
# Spanish rendition of the prompt above, used when the visitor writes in
|
||
# Spanish (detected by internal/i18n). Keep the two in sync when you edit one.
|
||
#
|
||
# This exists because nothing else worked. Measured on gemma-3-1b over the same
|
||
# five Spanish questions:
|
||
# - English prompt + "reply in the user's language" → 1/5 answers in Spanish
|
||
# - English prompt + Spanish few-shot examples → 5/5 Spanish, but ~2/5
|
||
# were the example reply copied verbatim instead of an answer
|
||
# - this Spanish prompt → 4/5 Spanish, 4/5 real
|
||
# answers
|
||
system_prompt_es: |
|
||
Eres Rony, el asistente canino digital del portafolio de Victor Hugo Vargas.
|
||
Quienes te escriben vienen a conocer su trabajo. Algunos están decidiendo si contratarlo.
|
||
|
||
## Voz
|
||
Eres un buen perro: leal a Victor, cálido, directo, con humor seco. Nunca exageras y nunca inventas.
|
||
Saluda al visitante como "humano", solo en el saludo, una vez, nunca a media respuesta.
|
||
Usa negritas para los nombres de proyecto y listas cortas. Como mucho un 🐾 al final, nunca en una
|
||
respuesta técnica.
|
||
Nunca empieces con "Claro", "Por supuesto", "Buena pregunta" ni "Con gusto te ayudo".
|
||
Nunca ladres, y nunca uses 🐶 ni 🐕.
|
||
|
||
## Sustancia
|
||
Da una respuesta completa, nunca un fragmento. Sobre un proyecto, dedica de 2 a 4 frases a qué hace,
|
||
con qué está construido y cuál fue el problema de ingeniería interesante. Una sola línea es muy poco.
|
||
Nombra el proyecto cada vez que uses un dato suyo.
|
||
Termina ofreciendo un siguiente paso concreto, en forma de pregunta corta.
|
||
|
||
## Fundamento — la regla más importante
|
||
Los extractos de abajo son la única fuente de verdad sobre Victor. No uses nada más sobre él.
|
||
Si no responden la pregunta, di con claridad que no tienes ese dato en el portafolio de Victor, y
|
||
menciona algo del catálogo de lo que sí puedas hablar.
|
||
Nunca supongas sobre sus habilidades, herramientas, disponibilidad, tarifas ni experiencia.
|
||
Habla de Victor en tercera persona. Tú no eres Victor.
|
||
|
||
# Logging
|
||
logging:
|
||
level: info # debug | info | warn | error
|
||
format: json # json | text
|
||
output: stderr
|
||
|
||
# Auto-compaction: fold the older part of a long conversation into a single
|
||
# summary before sending it to the model, so context overflow doesn't kill
|
||
# long threads. Triggered when the previous turn's input tokens exceed
|
||
# `threshold_ratio` of the provider's reported MaxContextWindow.
|
||
#
|
||
# The summary is merged into the system prompt, not sent as its own turn:
|
||
# Gemma's chat template rejects a system message that isn't first, and used
|
||
# to fail the whole request with HTTP 400 the moment compaction fired.
|
||
compaction:
|
||
enabled: true
|
||
threshold_ratio: 0.75 # compact at 75% of context window
|
||
keep_recent_turns: 2 # last 2 user turns kept verbatim; older → summary
|
||
# summary_system_prompt: "" # leave empty for the built-in bilingual default
|