rony-chat-bot/docs/architecture.md

📋 Rony Chat Bot — Technical Design Document

Version: 1.0 Author: Victor Hugo Vargas Date: 2026-06-28 Status: Complete specification for implementation Path: rony-chat-bot/docs/architecture.md

🌐 Language: English | Español

📚 Workspace: This project is part of the Rony/ workspace. See ../README.md.

🔑 Depends on: rony-llm-agent — core library that provides agent loop, LLM clients, RAG, persona system.

📐 Methodology: This project follows the SDD + DDD + Hexagonal Architecture approach. Functional Requirements are numbered as CRF-XXX. See ../../METHODOLOGY.md.

---

🎯 1. Project Vision

1.1 What is Chat-Bot?

An HTTP chatbot that answers questions about Victor Hugo Vargas and his projects. Uses RAG (Retrieval-Augmented Generation) over markdown files describing each project, and a local LLM (or cloud) to generate responses.

1.2 Primary use case

Victor has a portfolio website (Astro + React). On the site there's a chat widget where visitors can ask:

• "What projects has Victor done?"
• "What's his experience with Go?"
• "How does Rony Harness work?"
• "Has Victor worked with PostgreSQL?"

The bot responds with accurate information extracted from the projects' markdown files + bio + skills.

1.3 Secondary use cases (future)

• Client adaptation: The same bot, with other data and another persona, serves car dealerships, restaurants, etc.
• Standalone CLI: ./chat-bot ask "what do you know about X?" for terminal use.
• Slack/Discord bot: Wrapper that consumes the HTTP API.

1.4 Philosophy

• Self-hosted by default — works 100% local with Ollama + 1-3B models
• Cloud optional — if you need more quality, swap to Anthropic API
• Portable — easy to fork/customize for other contexts
• Streaming — token-by-token responses with SSE (no waiting for complete response)
• Reuses rony-llm-agent — doesn't reinvent the agent loop

---

🏗️ 2. Architecture

2.1 Overview

┌─────────────────────────────────────────────────────────────────┐
│  Browser (Astro site)                                            │
│      ↓ HTTP POST /api/chat                                       │
│  Astro SSR (proxy)  ←────────── Serves portfolio + proxy chat    │
│      ↓ HTTP POST /api/chat                                       │
│  Chat-Bot HTTP server (:7331)                                    │
│      ↓                                                           │
│  Agent loop (rony-llm-agent)                                       │
│      ↓                                                           │
│  Hybrid RAG → SQLite FTS5 (BM25) ⊕ vectors, fused with RRF        │
│      ↓                    over data/projects/ + data/docs/        │
│      ├─→ embeddings server (:9200, nomic-embed-v2-moe)            │
│      ↓                                                           │
│  LLM (llama.cpp local default / Ollama or Anthropic optional)   │
└─────────────────────────────────────────────────────────────────┘

Two local model servers, not one. Both are plain llama-server processes the bot talks to over HTTP; neither is linked into the binary.

2.2 Main components

Component	Path	Responsibility
HTTP server	internal/server/	chi handlers, SSE streaming
Agent runner	internal/agent/	Wrapper over rony-llm-agent: prompt assembly, retrieval, language selection, compaction
Portfolio loader	internal/portfolio/	Reads data/projects/ and data/docs/ (.md + .mdx), indexes into SQLite FTS5 + vectors, hybrid search
Embeddings client	internal/embed/	OpenAI-compatible embeddings, normalisation, float32 codec
Persona	internal/persona/	Loads persona from configs/portfolio-bot.yaml
CLI	cmd/chat-bot/	Commands: serve, reindex, ask, version

2.3 Tech stack

Layer	Technology	Reason
Language	Go 1.26+	Same as rony-harness, leverage os.Root, iter.Seq
HTTP router	net/http + chi	Stdlib + chi for middleware (CORS, logging)
SSE	net/http Flusher	Stdlib is enough, no external library needed
Config	gopkg.in/yaml.v3	Same as harness
RAG backend	SQLite + FTS5 (BM25) ⊕ dense vectors	Zero external deps, single file. No ANN index: a portfolio is hundreds of chunks, so a full scan is microseconds
LLM	llama.cpp (qwen2.5-3b-instruct Q4_K_M) — default; Ollama as alt	Self-hosted by default. 3B, not 1.5B: see the benchmark in configs/portfolio-bot.yaml
Embeddings	nomic-embed-v2-moe Q5_K_M, 768-dim	Multilingual — the whole point is matching Spanish questions to English documents
Tests	stdlib + testify	Consistency with the rest

---

🔌 3. HTTP API

3.1 Endpoints

POST /api/chat — Chat with SSE streaming

Request:

{
  "messages": [
    {"role": "user", "content": "What projects does Victor have?"}
  ],
  "stream": true,
  "conversation_id": "57f4aa3c7fab466bc4de9c43b296903e"
}

Field	Required	Notes
messages	yes	At least one user message; alternation is not enforced.
stream	no, default true	false returns a single JSON body instead of SSE.
conversation_id	no	Hex string. If omitted, the server mints a new one and returns it (see below). Pass an existing ID to keep the thread.

Response (SSE):

data: {"type":"start","conversation_id":"57f4aa3c7fab466bc4de9c43b296903e"}

data: {"type":"chunk","content":"Victor"}
data: {"type":"chunk","content":" has"}
data: {"type":"chunk","content":" several"}
data: {"type":"chunk","content":" projects"}

data: {"type":"sources","documents":["rony-harness.md","rony-llm-agent.md"]}

data: {"type":"done","usage":{"input_tokens":245,"output_tokens":38}}

The conversation_id in the start event is what the client should store (see §3.4 — Conversation persistence). When the client passed an existing ID the server echoes it back; otherwise it's freshly minted.

Without streaming ("stream": false):

{
  "conversation_id": "57f4aa3c7fab466bc4de9c43b296903e",
  "content": "Victor has several projects...",
  "sources": ["rony-harness.md", "rony-llm-agent.md"],
  "usage": {"input_tokens": 245, "output_tokens": 38}
}

GET /api/conversations — List recent conversations

Returns the most recent conversation summaries, newest first. Useful for a "show my chats" sidebar in a custom UI.

Query params:

• limit (1–200, default 50)

Response:

{
  "count": 2,
  "conversations": [
    {
      "id": "57f4aa3c7fab466bc4de9c43b296903e",
      "created_at": "2026-07-17T05:02:07Z",
      "updated_at": "2026-07-17T05:04:31Z",
      "preview": "What projects does Victor have?"
    }
  ]
}

GET /api/conversations/{id} — Fetch one conversation

Returns the full history of a conversation with all messages in chronological order.

Response (200):

{
  "id": "57f4aa3c7fab466bc4de9c43b296903e",
  "created_at": "2026-07-17T05:02:07Z",
  "updated_at": "2026-07-17T05:04:31Z",
  "messages": [
    {"id": 1, "role": "user",      "content": "What projects does Victor have?", "created_at": "..."},
    {"id": 2, "role": "assistant", "content": "Victor has several projects...", "sources": ["..."], "created_at": "..."}
  ]
}

Response (404): when the ID is unknown (e.g. server DB was wiped or the client lost sync). The widget treats this as "start fresh".

⚠️ Auth note: the conversation ID is the only access token. For a public bot this is fine; for private contexts add auth at the proxy layer (e.g. require a session cookie before forwarding to this endpoint).

DELETE /api/conversations/{id} — Delete a conversation

Removes the conversation and all its messages (cascade). Returns 204 on success, 404 if the ID doesn't exist.

POST /api/reindex — Re-index portfolio

Useful when files in data/projects/ are modified.

Request: empty Response:

{
  "indexed_files": 12,
  "total_chunks": 87,
  "duration_ms": 4321
}

GET /api/health — Health check (real)

Probes the LLM provider and the SQLite store in parallel and returns their states. Designed for monitoring/load balancers. Returns 200 when healthy or degraded, 503 when unhealthy.

• ?deep=true adds a chunk count to the store probe (same latency budget).

Status taxonomy:

status	HTTP	Meaning
healthy	200	LLM up, store up
degraded	200	LLM up, store down — bot still answers, just without RAG
unhealthy	503	LLM down — bot cannot answer, no point routing traffic here

Probe details:

Component	Probe	Latency
llm	GET {provider}/health (llamacpp, ollama) or /models (openai)	~1ms for local llama-server
store	SELECT 1 on the SQLite handle	~100µs

Each probe has a 2s timeout; the whole call returns within ~2.5s even if a dependency hangs.

Response shape (healthy):

{
  "status": "healthy",
  "version": "0.2.0-dev",
  "checked_at": "2026-07-17T05:02:07Z",
  "components": {
    "llm": {
      "status": "up",
      "latency": "1.028ms",
      "details": {"provider": "llamacpp", "model": "qwen2.5-3b-instruct", "url": "http://localhost:9100/health"}
    },
    "store": {
      "status": "up",
      "latency": "107µs"
    }
  }
}

Response shape (degraded, with ?deep=true):

{
  "status": "degraded",
  "version": "0.2.0-dev",
  "checked_at": "2026-07-17T05:02:07Z",
  "components": {
    "llm": {"status": "up", "latency": "0.8ms", "details": {...}},
    "store": {"status": "up", "latency": "70µs", "details": {"chunks": 28}}
  }
}

Response shape (unhealthy): HTTP 503, same JSON with "status": "unhealthy" and the failed component reporting "status": "down" plus an error field.

GET /api/info — Bot metadata

{
  "name": "Rony Chat Bot",
  "model": "qwen2.5-3b-instruct",
  "persona": "...",
  "topics": ["projects", "experience", "technical skills"]
}

3.2 SSE Implementation

// internal/server/chat.go
package server

import (
    "encoding/json"
    "fmt"
    "net/http"
    "github.com/VictorVargas/rony-llm-agent/pkg/agent"
)

func (s *Server) handleChat(w http.ResponseWriter, r *http.Request) {
    // SSE headers
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("X-Accel-Buffering", "no")
    
    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "SSE not supported", http.StatusInternalServerError)
        return
    }
    
    // Parse request
    var req ChatRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        writeError(w, flusher, "invalid request", err)
        return
    }
    
    // Start event
    writeSSE(w, flusher, "start", map[string]string{
        "conversation_id": generateConvID(),
    })
    
    // Run agent with streaming
    sources := []string{}
    for chunk, err := range s.agent.RunStream(r.Context(), req.Messages) {
        if err != nil {
            writeSSE(w, flusher, "error", map[string]string{"message": err.Error()})
            return
        }
        if chunk.Type == "source" {
            sources = append(sources, chunk.Source)
        }
        writeSSE(w, flusher, chunk.Type, chunk.Data)
    }
    
    // Done event
    writeSSE(w, flusher, "done", map[string]any{
        "usage": map[string]int{
            "input_tokens":  245,
            "output_tokens": 38,
        },
    })
}

func writeSSE(w http.ResponseWriter, flusher http.Flusher, eventType string, data any) {
    payload, _ := json.Marshal(data)
    fmt.Fprintf(w, "data: {\"type\":%q,\"data\":%s}\n\n", eventType, payload)
    flusher.Flush()
}

3.3 Middleware

// internal/server/middleware.go
package server

func (s *Server) loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        // Wrap response writer to capture status
        rw := &statusRecorder{ResponseWriter: w, status: 200}
        next.ServeHTTP(rw, r)
        
        slog.Info("http.request",
            "method", r.Method,
            "path", r.URL.Path,
            "status", rw.status,
            "duration_ms", time.Since(start).Milliseconds(),
            "ip", r.RemoteAddr,
        )
    })
}

func (s *Server) corsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        origin := r.Header.Get("Origin")
        for _, allowed := range s.config.Server.CORSOrigins {
            if origin == allowed {
                w.Header().Set("Access-Control-Allow-Origin", origin)
                w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS")
                w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
                break
            }
        }
        if r.Method == "OPTIONS" {
            w.WriteHeader(204)
            return
        }
        next.ServeHTTP(w, r)
    })
}

func (s *Server) rateLimitMiddleware(next http.Handler) http.Handler {
    limiter := rate.NewLimiter(rate.Every(time.Minute/time.Duration(s.config.Server.RateLimit.RequestsPerMinute)), s.config.Server.RateLimit.Burst)
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !limiter.Allow() {
            http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
            return
        }
        next.ServeHTTP(w, r)
    })
}

3.4 Conversation persistence

The bot persists conversation threads in the same SQLite database as the RAG index (./data/portfolio.db). Schema lives in internal/portfolio/conversations.go.

CREATE TABLE conversations (
    id         TEXT PRIMARY KEY,         -- 16-byte random hex (32 chars)
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);

CREATE TABLE messages (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    conversation_id TEXT    NOT NULL,
    role            TEXT    NOT NULL,    -- user | assistant | system
    content         TEXT    NOT NULL,
    sources         TEXT,                 -- JSON array, nullable
    created_at      INTEGER NOT NULL,
    FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE CASCADE
);
CREATE INDEX idx_messages_conv ON messages(conversation_id, id);

Lifecycle:

When	What
POST /api/chat (no conversation_id)	Server mints a new hex ID, returns it in the start SSE event (or conversation_id field of the JSON response)
POST /api/chat (with conversation_id)	Server reuses the existing row; both user message and assistant reply are appended
User message	Persisted before the LLM runs, so it survives a model failure
Assistant message	Persisted after the stream completes, with the RAG sources attached
GET /api/conversations/{id}	Returns the full thread; 404 if unknown
DELETE /api/conversations/{id}	Cascade-deletes messages

Client responsibilities:

1. On the first message, omit conversation_id. Capture the one the server returns in the start SSE event.
2. Store it client-side (localStorage["rony-chat-conv"] in the widget).
3. On every subsequent message, send the ID back.
4. On page load, if you have a stored ID, call GET /api/conversations/{id} to restore the thread. If 404, clear the stored ID and start fresh.

The widget (web/chat-widget.js) implements all four steps. Any other client (a custom React component, an Astro endpoint, a CLI replay tool) follows the same protocol.

Auth model:

The conversation ID is the only access token for GET /api/conversations/{id}. It is 128 bits of random entropy, so guessing one is infeasible. For a public portfolio bot this is the right trade-off — anyone who knows the URL can read its history. For private contexts, add an auth layer in front of the bot (proxy) that gates the conversation endpoints.

---

🧠 4. RAG (Retrieval-Augmented Generation)

✅ The four decisions this section left open have been made and measured:

• FTS5 tokenizer — unicode61 remove_diacritics 2, as specced. Stemming was not the bottleneck; the language gap was, and embeddings close it.
• SQLite driver — modernc.org/sqlite (pure Go, no CGO). Benchmark below.
• Chunking — by markdown heading, not fixed size, with oversized sections split at ### before falling back to byte offsets. §4.1.
• Semantic similarity — added. BM25 alone returned nothing at all for a Spanish question about an English document. §4.2.

4.0 Driver decision: benchmark results

Reproducible con CGO_ENABLED=1 go test -tags sqlite_fts5 -bench=. ./bench/. Datos: 4 markdowns → 11 chunks.

Operación	mattn (CGO)	modernc (puro Go)	Diferencia
Insert (11 chunks)	2,802,843 ns/op	1,465,646 ns/op	modernc 1.9× más rápido
Insert alloc	2,124,299 B/op	9,770 B/op	modernc usa 217× menos memoria
Query (8 queries BM25)	244,047 ns/op	555,162 ns/op	mattn 2.3× más rápido
Round-trip (insert + 8 queries)	3,543,417 ns/op	2,267,669 ns/op	modernc 1.6× más rápido
Binary size	11 MB	11 MB	igual
Build deps	gcc, CGO=1	nada	modernc gana
CI/CD portable	requiere toolchain C	go build puro	modernc gana

Decisión: modernc.org/sqlite.

Justificación:

1. Ambas latencias de query (~250µs vs ~550µs) son 2 órdenes de magnitud por debajo del target de 50ms — imperceptible vs el LLM (varios segundos).
2. modernc gana en inserts (1.9×) y round-trip (1.6×), que es el path de reindex.
3. Sin CGO = CI/CD más simple (sin gcc, sin Alpine musl-dev, binarios reproducibles).
4. Si en el futuro el cuello de botella pasa a ser query latency (corpus >10k chunks), se puede reconsiderar. Hoy no.

4.1 Indexing pipeline

data/projects/*.{md,mdx}   kind=project → announced in the catalogue
data/docs/*.{md,mdx}       kind=doc     → retrievable, never announced
    ↓ (skip each directory's README — those are instructions, not content)
Raw markdown
    ↓ (split by heading; sections over the limit split at ###, then by size)
    ↓ (drop the frontmatter chunk)
Chunks
    ├─→ SQLite FTS5 virtual table "portfolio_chunks"
    └─→ embeddings endpoint → "portfolio_vectors" (id, content_hash, dim, vec)
Indexed corpus

Two kinds of source. Everything under data_path is one of Victor's projects and is listed in the catalogue injected into every prompt. Everything under docs_path is searchable evidence that is not a project — his CV, an about page. The CV is what someone deciding whether to hire actually reads, and it was unreachable while it lived only in the Astro site; but filing it under projects made the bot list "cv" as one of his works.

Three things are excluded or reshaped, each because of a measured failure:

Rule	Failure it fixes
Skip README* in both directories	data/projects/README.md was indexed, so the catalogue announced "README" and "README.es" as projects of Victor's
Drop the frontmatter chunk	Dense metadata in a very short chunk is a magnet for short queries — a CV's location: field answered "¿Dónde ha trabajado Victor?" with a city instead of a work history
Split oversized sections at ###	A CV's Experience section is a list of jobs; size-splitting cut one mid-word, stranding the employer's name in the previous chunk. Chunks now hold one job each, named Experience — Metrimex — Frontend Developer

When it runs:

• Manually: ./chat-bot reindex
• On startup with serve --reindex-on-start
• Via HTTP: POST /api/reindex

Vectors are built at index time, so enabling embeddings requires a reindex. The chunk table is derived data: OpenStore rebuilds it when its columns are missing, so upgrading an existing install needs no migration step. Conversation tables are never touched by that rebuild.

4.2 Retrieval pipeline

User query "¿Con qué se paga en la tienda de ropa?"
    ↓
    ├─→ FTS5 MATCH, BM25 ranking      → top 15 (topK × 3)
    └─→ embed(query) → cosine vs vecs → top 15 (topK × 3)
    ↓ (Reciprocal Rank Fusion, k=60)
Top 5 chunks
    ↓ (system prompt + project catalogue + excerpts + language directive)
LLM generates answer

Both halves are load-bearing. Keyword search matches words exactly — no stemming, no translation. The corpus is written in English and visitors ask in Spanish, so the words carrying the meaning score zero: measured on the real corpus, "paga" appears 0 times in a document that says "Payments: Stripe" and "trabajado" 0 times in one that says "worked". The question above retrieved nothing at all. Embeddings (nomic-embed-v2-moe, multilingual) put all three tienda-ropa chunks on top — but they blur exact rare tokens, where BM25 is sharp.

Why RRF rather than a weighted score. A BM25 score and a cosine share no scale, so blending them numerically means inventing a conversion factor and retuning it whenever the corpus changes. RRF ignores the magnitudes and ranks by agreement between the two orderings: each list contributes 1/(60 + rank). A chunk both halves like beats one that either loves alone.

Degradation. If the embeddings endpoint is down or disabled, the vector half returns nothing and retrieval continues keyword-only rather than failing the request.

Stale vectors. Chunk ids are derived from position, so they survive edits to the body. Without a guard, editing a document leaves vectors that still describe the text that was removed — reproduced live by changing a project's payment provider and watching the old one keep coming back. Each vector stores a hash of the exact text it was built from, and rows whose hash no longer matches the chunk are skipped with a warning until the next reindex.

The catalogue. Top-K returns the best-matching sections, so a broad question like "what has Victor built?" cannot be answered from retrieval alone, and a small model asked to enumerate from partial hits invents the rest. The full project list is injected every turn at a cost of ~10 tokens per project. This is what stopped the bot naming projects that do not exist.

4.3 Schema

// internal/portfolio/indexer.go
const schema = `
CREATE VIRTUAL TABLE IF NOT EXISTS portfolio_chunks USING fts5(
    id UNINDEXED,
    project_id UNINDEXED,
    kind UNINDEXED,          -- 'project' | 'doc'
    source_file UNINDEXED,
    section UNINDEXED,       -- the heading this chunk came from
    chunk_index UNINDEXED,
    content,
    tokenize = 'unicode61 remove_diacritics 2'
);
`

// Vectors live in an ordinary table keyed by chunk id. There is no ANN index:
// a portfolio is hundreds of chunks, not millions, so a full scan with a dot
// product is microseconds and needs no extension.
const vectorSchema = `
CREATE TABLE IF NOT EXISTS portfolio_vectors (
    chunk_id     TEXT PRIMARY KEY,
    content_hash TEXT NOT NULL,   -- sha256 of the exact text embedded
    dim          INTEGER NOT NULL,
    vec          BLOB NOT NULL    -- little-endian float32, unit-normalised
);
`

FTS5 has no ALTER TABLE ADD COLUMN, so ensureChunkSchema drops and recreates portfolio_chunks when an older database is missing a column. That is safe precisely because the table is a derived index — every row is regenerated from the markdown on the next reindex. It deliberately touches only the index tables; conversations live in the same file and are real user data.

Vectors are unit-normalised at write time, so a dot product is the cosine and retrieval needs no division per comparison. dim is stored so a change of embedding model is detected rather than silently producing garbage similarity: rows whose dimension does not match the query vector are skipped.

Key API:

Function	File	Purpose
SourcesFor(dataPath, docsPath)	indexer.go	Builds the []Source pair; an empty path is skipped, so docs_path is optional without branching
Store.Reindex(ctx, sources, cfg)	indexer.go	Rebuilds both tables from disk
Store.Search(ctx, query, topK)	indexer.go	BM25 only, excludes section = 'frontmatter'
Store.HybridSearch(ctx, emb, q, topK)	hybrid.go	BM25 + vectors fused with RRF
Store.Catalog(ctx)	indexer.go	Project list for the prompt; filters kind = 'project'
embed.Client.Embed(ctx, texts)	internal/embed/	OpenAI-compatible embeddings, reordered by the response index field

4.4 Assembling the prompt

agent.Runner.BuildMessages runs once per request and produces exactly one system message followed by the conversational turns. The order of operations is load-bearing:

// internal/agent/runner.go
func (r *Runner) BuildMessages(ctx context.Context, history []Message) ([]Message, string, error) {
    // 1. Pull system-role notes (the compactor's summary) out of the turn list.
    notes, history := foldSystemNotes(history)

    // 2. The visitor's language selects which rendition of the prompt we
    //    build on, so it is resolved before anything else.
    lang := detectLanguage(history)
    systemPrompt := r.promptFor(lang)

    // 3. Resolved before retrieval so the excerpt budget accounts for it.
    catalog := r.catalogBlock(ctx)

    // 4. Hybrid retrieval on the last user turn.
    hits, err := r.store.HybridSearch(ctx, r.embedder, last.Content, r.topK)
    ragContext = r.limitRAGContext(systemPrompt, catalog, formatHits(hits), history)

    // 5. prompt → catalogue → excerpts → summary → language directive.
    system := botpersona.BuildSystemPrompt(systemPrompt, catalog, ragContext)
    system += "\n\n" + strings.Join(notes, "\n\n")
    system += "\n\n" + r.languageDirective(lang)

    history, err = r.fitHistory(system, history)
    return append([]Message{{Role: RoleSystem, Content: system}}, history...), ragContext, nil
}

Why exactly one system message. Gemma 3's chat template raises "Conversation roles must alternate user/assistant/..." on any system message after the first, which llama-server surfaces as HTTP 400 — enabling compaction used to kill the conversation outright the first time it fired. Anthropic's API rejects mid-conversation system turns too, so folding is the portable behaviour rather than a Gemma workaround. foldSystemNotes never aliases the caller's slice; the handler reuses the history it passes in.

Why the language directive goes last. It is the instruction a small model is most likely to still be holding when it starts generating. Position was not enough on its own, though — see below.

Matching the visitor's language took three attempts, measured on gemma-3-1b over the same five Spanish questions:

Approach	Result
English prompt + "reply in the user's language"	1/5 answered in Spanish
English prompt + Spanish few-shot examples	5/5 Spanish, but ~2/5 were the example reply copied verbatim instead of an answer
A full Spanish rendition of the prompt (system_prompt_es)	4/5 Spanish, 4/5 real answers

So internal/i18n detects the language and promptFor selects the rendition; the directive reinforces a prompt already written in the right language rather than trying to override one written in the wrong one. The two renditions in the YAML have to be kept in sync by hand.

Why keyword search alone was not enough. The original plan cited three reasons to skip embeddings — no model to run, one file and one driver, and BM25 being strong on structured project docs. The first two still hold and cost what was predicted (~0.91 GB resident, a second process). The third was right about the corpus and wrong about the questions: BM25 is excellent at retrieving English documents given English keywords, and this bot is asked in Spanish. That is not a morphology problem a trigram tokenizer fixes — it is a translation problem. Hence §4.2, and hence both halves are kept.

---

🗜️ 4.5 Auto-compaction

Long conversations eventually run out of context. Auto-compaction folds the older portion of the conversation into a single summary system message when the previous turn's input tokens cross a configurable threshold.

How much headroom there actually is. Measured over 20 real requests at context_size: 4096, the largest prompt this bot ever built was 1255 tokens — system prompt, project catalogue, five retrieved chunks and the question — with a median of 1069. That leaves room for roughly a dozen short turns before the 75% threshold (~3070 tokens) is reached, not the 2–3 an earlier draft of this document estimated. Compaction is therefore a safety net for genuinely long threads rather than something that fires in a typical visit; across the whole benchmark it never triggered and nothing was truncated.

When it fires

agent.Runner.Compact runs once per /api/chat request, before the RAG search. It compares the runner's most recent Usage.InputTokens (reported by the provider in the previous streamed chunk) against client.Capabilities().MaxContextWindow × threshold_ratio.

Setting	Default	What it controls
compaction.enabled	false	Master switch.
compaction.threshold_ratio	0.75	Trigger when used tokens ≥ window × ratio.
compaction.keep_recent_turns	4	How many of the latest user turns are kept verbatim after compaction.
compaction.summary_system_prompt	(built-in bilingual)	Override the instruction sent to the LLM when summarizing.

Short-circuits silently when compaction is disabled, the provider doesn't report a window (Capabilities().MaxContextWindow == 0), the history is shorter than keep_recent_turns, or usage is still unknown (first turn).

How the summary is made

1. splitByTurns(history, keep_recent_turns) divides messages into (older, recent) on user-role boundaries so a kept turn's user/assistant pair always stays together.
2. renderTranscript(older) flattens older messages into a User: / Assistant: transcript (skipping tool messages and empty assistant placeholders).
3. The runner calls client.Generate(...) with the summary prompt + transcript and a 512-token cap so the compaction step itself stays cheap.
4. The returned text is prepended as a system message ("Earlier conversation summary:\n…"), followed by the recent tail.
5. LastCompaction() returns CompactionStats so the SSE handler can emit a compaction event right before the streamed chunks.

Failure mode

If Generate errors or returns an empty summary, compaction falls back to truncateToBudget: drop oldest user-turns one at a time until the remaining slice fits threshold tokens (heuristic: len(s) / 4 + 1). The current user turn is always preserved. The fallback is logged at WARN and the request still proceeds — a flaky summarize call never fails the user's request.

Wire protocol

Streaming responses gain an optional compaction event:

data: {"type":"compaction","older_turns":6,"kept_turns":2,"summary_tokens":120,"window_tokens":4096,"used_tokens":3500}

Emitted after start (when applicable) and before sources / chunk. The widget can render this as a subtle "Context compacted" hint or ignore it — both are valid.

Persistence

Compaction is per-request. The full transcript is still saved to messages in data/portfolio.db verbatim, so GET /api/conversations/{id} always returns the original history. Only what we send to the LLM is reduced — the next session can re-read the full thread from the DB.

---

🌐 5. Embedding the widget

The bot ships with a drop-in vanilla-JS widget. Add two files to your site and it works.

5.1 The widget (any site)

<link rel="stylesheet" href="/path/to/chat-widget.css">
<script src="/path/to/chat-widget.js"
        data-api-url="https://chat.example.com"
        data-title="Ask me anything"
        data-greeting="Hi! Ask me about the projects."
        data-position="bottom-right"
        data-theme="auto"
        defer></script>

A bubble appears bottom-right, opens a panel, talks SSE to /api/chat, streams the response, and cites sources. No build step, no React/Vue, no framework lock-in.

Browser→bot options:

Topology	Trade-offs
Direct (browser → bot, same domain or CORS)	Simplest. Add the bot's origin to cors_origins in YAML.
Reverse proxy (nginx/Caddy in front)	Bot stays on private network, single public domain, no CORS to manage.
Site proxies the bot (Astro/Next API route)	Adds a hop and a bit of code, but gives you auth/session hooks in your site.

The widget works the same in all three. Pick the topology that matches your infra.

Default dev setup is direct + CORS. cors_origins in configs/portfolio-bot.yaml controls which sites can call the bot. Add your site's origin there.

5.2 Astro: drop-in via Layout

The widget works in Astro without writing a React component. Add this to your shared layout:

---
// src/layouts/BaseLayout.astro
import "../path/to/chat-widget.css";
const apiUrl = import.meta.env.PUBLIC_CHAT_API_URL || "http://localhost:7331";
---
<html>
  <body>
    <slot />
    <script src="/path/to/chat-widget.js"
            data-api-url={apiUrl}
            data-title="Ask me anything"
            data-position="bottom-right"
            data-theme="auto"
            defer is:inline></script>
  </body>
</html>

is:inline keeps Astro from hashing/transforming the script tag, so the data-* attributes survive.

5.3 React / Next.js: same script tag

// app/layout.tsx
import Script from "next/script";

export default function RootLayout({ children }) {
  return (
    <html>
      <head>
        <link rel="stylesheet" href="/chat-widget.css" />
        <Script src="/chat-widget.js"
                data-api-url={process.env.NEXT_PUBLIC_CHAT_API_URL}
                data-title="Ask me anything"
                data-position="bottom-right"
                data-theme="auto"
                strategy="afterInteractive" />
      </head>
      <body>{children}</body>
    </html>
  );
}

5.4 If you want a server proxy (Astro/Next API route)

The widget can also call a same-origin endpoint that forwards to the bot. This is the right call when you need:

• Auth on /api/chat (logged-in users only)
• Centralized rate limiting at the site level
• Hiding the bot's origin from the browser

// src/pages/api/chat.ts (Astro) or app/api/chat/route.ts (Next)
const CHAT_BOT_URL = process.env.CHAT_BOT_URL || "http://localhost:7331";

export const POST = async ({ request }) => {
    const body = await request.json();
    // (optional) auth check, rate limit, session lookup here

    const resp = await fetch(`${CHAT_BOT_URL}/api/chat`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(body),
    });

    return new Response(resp.body, {
        status: resp.status,
        headers: {
            "Content-Type": "text/event-stream",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        },
    });
};

Then point the widget at /api/chat (same origin) instead of the bot's URL.

5.5 Widget configuration reference

All options are data-* attributes on the <script> tag:

Attribute	Default	Notes
data-api-url	(required)	Base URL of the bot. No trailing slash.
data-title	"Chat"	Header text.
data-greeting	""	First assistant message when the panel opens.
data-position	"bottom-right"	"bottom-right" or "bottom-left".
data-theme	"auto"	"auto" (follows OS), "light", "dark".

Theming is via CSS custom properties on .rony-chat-widget-root (see web/chat-widget.css):

.rony-chat-widget-root {
  --rony-accent: #ff6b35;
  --rony-radius: 4px;
  --rony-font: "Inter", sans-serif;
}

5.6 What the widget doesn't do (yet)

• Richer markdown (tables, images) — the built-in renderer handles the common cases; for full CommonMark, swap renderMarkdown in chat-widget.js for marked or markdown-it.
• Mobile swipe-to-dismiss — panel goes full-screen on phones.
• Conversation history sidebar — only the active conversation is shown (the backend exposes GET /api/conversations for a future sidebar).

---

🤖 6. Self-hosting with llama.cpp (default)

6.1 Setup

llama-server is a separate process that the bot connects to over HTTP. Both ports (the bot's and llama-server's) are configurable — pick what fits your environment.

# 1. Make sure you have a GGUF model available
# Download from Hugging Face, e.g.:
#   https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF
export RONY_MODELS_PATH=/path/to/models
ls $RONY_MODELS_PATH/qwen2.5-3b-instruct-q4_k_m.gguf

# 2. Start llama-server (port is configurable; default llama.cpp is 8080)
llama-server \
  -m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --host 127.0.0.1 \
  --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05

# 3. Start the embeddings server (second process, second terminal)
llama-server \
  -m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2

# 4. Make sure configs/portfolio-bot.yaml points to the same ports
#    providers[0].endpoint: http://localhost:9100/v1
#    embeddings.endpoint:   http://localhost:9200/v1

# 5. Build the index (needs the embedder up — vectors are built here)
./bin/chat-bot reindex

# 6. Start the bot (default port 7331, also configurable)
./bin/chat-bot serve
# → Serves on http://localhost:7331
# → Override with: ./bin/chat-bot serve --port 9101 --host 127.0.0.1

Flags that are not optional, each for a measured reason:

Flag	Why
--device none	llama.cpp brings up a compiled-in GPU backend even with -ngl 0, and on a GPU-less host those buffers come out of system RAM. Measured on qwen2.5-3b: 2.54 GB with a GPU absorbing them, 3.66 GB without. Budget from the second number
--parallel 1	--ctx-size is divided across slots and the default is 4, so --ctx-size 4096 without it gives each request 1024 tokens
--mlock	Prevents swap — critical on a shared VPS
--temp / --top-k / --top-p	Use the model authors' published values, not llama.cpp's defaults. gemma-3-1b at temperature 0.7 with the rest unset produced 16-token stub answers
--pooling mean (embedder)	Without it the endpoint does not return one vector per input and the client rejects the response

Port reference:

What	Default	How to change
llama-server HTTP port	8080 (llama.cpp convention)	--port N flag when starting llama-server
embeddings llama-server port	8080 (same convention)	--port N; this repo uses 9200
chat-bot HTTP port	7331	--port N flag on serve, or server.port in YAML
chat-bot → llama-server URL	http://localhost:8080/v1	endpoint field on the provider in YAML

Keep context_size in the YAML equal to --ctx-size: the bot sizes its RAG and compaction budgets from that number and never asks the server what it actually has, so a mismatch means prompts the server rejects.

The llamacpp provider is imported from rony-llm-agent/pkg/llm/providers/llamacpp and speaks HTTP to the server above — no CGO, no linking against llama.cpp.

6.2 Alternative: Ollama (easier for development)

If you don't want to manage GGUF files manually, Ollama provides the same models with a simpler workflow:

# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 2. Download chat model
ollama pull qwen2.5:1.5b

# 3. Verify
ollama list

# 4. Edit configs/portfolio-bot.yaml to mark ollama-local as default:
#    providers[0].default: true (and remove default from llamacpp-local)
#    Ollama exposes an OpenAI-compatible API on :11434/v1

# 5. Start the bot
ollama serve &
./bin/chat-bot serve

6.3 Alternative: llama.cpp direct (advanced)

For more control or if Ollama doesn't work in your setup:

providers:
  - name: llamacpp-local
    type: llamacpp
    model: qwen2.5-3b-instruct
    endpoint: http://localhost:9100/v1   # configurable, see §6.1
    context_size: 4096                   # must match --ctx-size
    max_tokens: 640
    temperature: 0.7                     # Qwen's published instruct defaults
    top_k: 20
    top_p: 0.8
    repeat_penalty: 1.05
    default: true

The llamacpp adapter is imported from rony-llm-agent/pkg/llm/providers/llamacpp and speaks HTTP to llama-server — no CGO, no linking against llama.cpp.

---

📦 7. Bot CLI

7.1 Commands

# Start HTTP server
chat-bot serve [--port 7331] [--host 0.0.0.0] [--reindex-on-start]

# Re-index (reads data/projects/ + data/docs/ *.md and *.mdx → FTS5 + vectors)
# Needs the embeddings server up if embeddings are enabled.
chat-bot reindex

# Single question (no server, useful for tests)
chat-bot ask "What projects does Victor have?" [--no-rag]

# Validate config
chat-bot config validate

# Health check (useful for monitoring)
chat-bot health

# Version
chat-bot version

7.2 Implementation with Cobra

// cmd/chat-bot/main.go
package main

import (
    "github.com/spf13/cobra"
)

func main() {
    root := &cobra.Command{
        Use:   "chat-bot",
        Short: "Portfolio chatbot HTTP server",
    }
    
    root.AddCommand(serveCmd())
    root.AddCommand(reindexCmd())
    root.AddCommand(askCmd())
    root.AddCommand(configCmd())
    root.AddCommand(healthCmd())
    root.AddCommand(versionCmd())
    
    if err := root.Execute(); err != nil {
        os.Exit(1)
    }
}

func serveCmd() *cobra.Command {
    var port int
    var host string
    var reindexOnStart bool
    
    cmd := &cobra.Command{
        Use:   "serve",
        Short: "Start HTTP server",
        RunE: func(cmd *cobra.Command, args []string) error {
            return server.Serve(server.Config{
                Port:           port,
                Host:           host,
                ReindexOnStart: reindexOnStart,
            })
        },
    }
    
    cmd.Flags().IntVar(&port, "port", 7331, "HTTP port")
    cmd.Flags().StringVar(&host, "host", "0.0.0.0", "HTTP host")
    cmd.Flags().BoolVar(&reindexOnStart, "reindex-on-start", false, "Re-index RAG before serving")
    
    return cmd
}

---

🚀 8. Deployment

8.1 Recommendation: Self-hosted on VPS

Target: 2 CPU cores, 8 GB RAM, no GPU. Three processes — the bot and two llama-server instances — so three units. The bot depends on both.

# 1. Build
go build -o /usr/local/bin/chat-bot ./cmd/chat-bot

# 2. The LLM
cat > /etc/systemd/system/llama-chat.service <<EOF
[Unit]
Description=llama-server (chat model)
After=network.target

[Service]
Type=simple
User=chatbot
ExecStart=/usr/local/bin/llama-server \\
  -m /opt/models/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \\
  --port 9100 --host 127.0.0.1 \\
  --ctx-size 4096 --parallel 1 \\
  --device none --threads 2 --mlock \\
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
Restart=on-failure
# --mlock needs the memory to be lockable
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

# 3. The embedder
cat > /etc/systemd/system/llama-embed.service <<EOF
[Unit]
Description=llama-server (embeddings)
After=network.target

[Service]
Type=simple
User=chatbot
ExecStart=/usr/local/bin/llama-server \\
  -m /opt/models/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \\
  --port 9200 --host 127.0.0.1 \\
  --embedding --pooling mean \\
  --ctx-size 2048 --parallel 1 --device none --threads 2
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 4. The bot
cat > /etc/systemd/system/chat-bot.service <<EOF
[Unit]
Description=Portfolio Chat Bot
After=network.target llama-chat.service llama-embed.service
Wants=llama-chat.service llama-embed.service

[Service]
Type=simple
User=chatbot
WorkingDirectory=/opt/chat-bot
ExecStart=/usr/local/bin/chat-bot serve
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now llama-chat llama-embed chat-bot

# 5. Build the index once both model servers are up
sudo -u chatbot /usr/local/bin/chat-bot reindex

Memory budget. 3.64 GB (LLM) + 0.91 GB (embedder) + 0.02 GB (bot) ≈ 4.6 GB resident, leaving ~3.4 GB for whatever else shares the VPS. Budget with --device none in place: without it the numbers look ~1.1 GB smaller on a machine with a GPU and then do not reproduce in production. See vps-context-sizing.md.

reindex has to be re-run after editing the markdown and after enabling or changing the embeddings model — vectors are built at index time, and a model change alters the dimension.

8.2 Reverse proxy (Caddy)

# /etc/caddy/Caddyfile
chat.victorvargas.dev {
    reverse_proxy localhost:7331
}

8.3 Monitoring

# Health check periodic
curl -s http://localhost:7331/api/health | jq

# Logs
journalctl -u chat-bot -f

---

🧪 9. Testing

9.1 Unit tests

// internal/server/chat_test.go
package server

func TestHandleChat_ValidRequest(t *testing.T) {
    s := newTestServer(t)
    
    req := httptest.NewRequest("POST", "/api/chat", strings.NewReader(`{
        "messages": [{"role": "user", "content": "hello"}]
    }`))
    req.Header.Set("Content-Type", "application/json")
    
    w := httptest.NewRecorder()
    s.handleChat(w, req)
    
    assert.Equal(t, 200, w.Code)
    assert.Equal(t, "text/event-stream", w.Header().Get("Content-Type"))
}

func TestHandleChat_RateLimit(t *testing.T) {
    s := newTestServerWithConfig(t, server.Config{
        RateLimit: 1, // 1 request per minute
    })
    
    // First request OK
    req1 := newChatRequest("hello")
    w1 := httptest.NewRecorder()
    s.handleChat(w1, req1)
    assert.Equal(t, 200, w1.Code)
    
    // Second request denied
    req2 := newChatRequest("hello again")
    w2 := httptest.NewRecorder()
    s.handleChat(w2, req2)
    assert.Equal(t, 429, w2.Code)
}

9.2 Integration tests with mock LLM

// internal/agent/runner_test.go
func TestRunner_RAGContextIsInjected(t *testing.T) {
    mockLLM := mock.New(mock.Responses{
        {Match: "projects", Response: "Victor has several projects..."},
    })
    
    memory := newMockMemoryWithDocs(t, []rag.Fragment{
        {Content: "Rony Harness: AI agent harness...", ProjectID: "rony-harness"},
        {Content: "rony-llm-agent: Go library...", ProjectID: "rony-llm-agent"},
    })
    
    runner := agent.NewRunner(agent.Config{
        LLM:    mockLLM,
        Memory: memory,
        Persona: testPersona,
    })
    
    resp, _ := runner.Run(context.Background(), []llm.Message{
        {Role: llm.RoleUser, Content: "what projects does Victor have?"},
    })
    
    // Verify LLM received context chunks in system prompt
    lastReq := mockLLM.LastRequest()
    assert.Contains(t, lastReq.Messages[0].Content, "Rony Harness")
    assert.Contains(t, lastReq.Messages[0].Content, "rony-llm-agent")
}

9.3 E2E test with Astro

# 1. Start chat-bot on :7331
./bin/chat-bot serve &

# 2. Start Astro on :4321
cd ../portfolio && npm run dev &

# 3. Make request to Astro's proxy
curl -X POST http://localhost:4321/api/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hello"}]}'

# 4. Verify SSE stream

9.4 Retrieval tests

Retrieval is the part of this bot where a regression is silent — nothing errors, answers just get subtly worse — so each failure found in testing has a test pinning it. The code samples above are illustrative; these are real.

Test	What it pins
TestHybridSearchFindsChunkKeywordSearchCannot	The Spanish-question / English-document gap, the reason embeddings exist
TestHybridSearchFallsBackWhenEmbedderFails	A dead embeddings endpoint degrades to keyword-only, never fails the request
TestVectorSearchIgnoresVectorsWhoseChunkChanged	Stale vectors after a body edit are skipped
TestVectorSearchIgnoresMismatchedDimensions	Changing the embedding model does not produce garbage similarity
TestReindexSkipsTheDirectoryReadmes	The catalogue never announces README as a project
TestReindexIndexesMdxAndSeparatesDocsFromProjects	.mdx is indexed; the CV is retrievable but never listed
TestOpenStoreMigratesIndexWithoutKindColumn	Upgrading an existing install needs no migration step
TestSubSplitPrefersH3BoundariesOverByteOffsets	CV jobs stay whole instead of being cut mid-word
TestBuildMessagesFoldsSystemNotesIntoOneSystemMessage	Compaction cannot re-introduce the HTTP 400
TestLanguageDirectiveIsLastInSystemPrompt	The directive keeps the position it needs to work

They run against real SQLite and a stub embedder, so no model server is needed: go test ./... is enough.

---

📂 10. Project Structure

rony-chat-bot/
├── cmd/
│   └── chat-bot/
│       └── main.go                 # CLI entrypoint
│
├── internal/
│   ├── server/                     # HTTP handlers
│   │   ├── server.go               # chi router + middleware
│   │   ├── handlers.go             # /api/chat, /api/health, /api/info, /api/reindex, /api/conversations
│   │   ├── conversations_test.go   # round-trip, continue, list, 404, delete, streaming
│   │   └── middleware.go           # RequestID, Logging, CORS, RateLimit
│   │
│   ├── agent/                      # LLM client + RAG runner
│   │   ├── runner.go               # Stream wrapper, RAG injection into system prompt
│   │   └── client.go               # NewClient factory: llamacpp / ollama / openai / anthropic
│   │
│   ├── portfolio/                  # RAG: markdown → SQLite FTS5 + vectors, conversation persistence
│   │   ├── chunker.go              # Heading-based splitter, ### sub-split
│   │   ├── indexer.go              # Store: schema, Reindex, Search (BM25), Catalog
│   │   ├── hybrid.go               # HybridSearch: BM25 ⊕ vectors via RRF, EmbedChunks
│   │   ├── conversations.go        # Conversation + Message CRUD, persisted alongside RAG
│   │   └── chunker_test.go / store_test.go / hybrid_test.go
│   │
│   ├── embed/                      # Embeddings client
│   │   ├── embed.go                # Embed, Normalize, Similarity, Encode/Decode
│   │   └── embed_test.go
│   │
│   ├── persona/                    # Persona bridge to rony-llm-agent
│   │   └── persona.go              # FromConfig, BuildSystemPrompt (with RAG context)
│   │
│   ├── streaming/                  # SSE protocol helpers
│   │   └── sse.go                  # WriteStart/Chunk/Sources/Done/Error
│   │
│   ├── i18n/                       # Language detection (ES/EN) for the response
│   │
│   └── config/                     # YAML loader + validation
│
├── web/                            # ← DROP-IN CHAT WIDGET
│   ├── chat-widget.js              # Vanilla JS, ~12 KB
│   ├── chat-widget.css             # Scoped styles, CSS-custom-prop themable
│   ├── example.html                # Local demo (python -m http.server)
│   └── README.md                   # Integration guide (HTML, Astro, Next.js)
│
├── data/
│   ├── projects/                   # ← One .md/.mdx per project — listed in the catalogue
│   │   ├── rony-harness.md
│   │   ├── rony-llm-agent.md
│   │   ├── example-project.md
│   │   └── README.md               # Instructions; skipped by the indexer
│   │
│   └── docs/                       # ← Reference material that is NOT a project
│       ├── cv.mdx                  # Usually a symlink; gitignored
│       └── README.md               # Instructions; skipped by the indexer
│
├── configs/
│   └── portfolio-bot.yaml          # Provider + RAG + persona config
│
├── docs/
│   ├── architecture.md             # ← THIS FILE
│   └── architecture.es.md
│
├── bench/                          # Reproducible SQLite driver benchmark
│
├── go.mod                          # require rony-llm-agent, modernc.org/sqlite
└── README.md

---

📅 11. Roadmap

Phase 1: MVP — done

• Project setup (go mod init, structure)
• HTTP server with /api/chat endpoint
• Functional SSE streaming
• RAG indexer (data/projects/ + data/docs/, .md + .mdx → FTS5)
• RAG retriever (query → top-k chunks)
• Persona loader from YAML
• llama.cpp integration — qwen2.5-3b, not the 1.5b originally planned
• CLI: serve, reindex, ask
• Basic tests

Phase 2: Integration with Astro — done, differently

• Drop-in vanilla-JS widget — replaced the planned React component and Astro proxy route. No build step, no framework lock-in, and it works in all three topologies in §5.1 rather than only behind a proxy
• Widget styling — scoped CSS with custom properties, not TailwindCSS; a drop-in widget cannot assume the host site's toolchain
• Automated E2E test: Astro → chat-bot → response (still manual, §9.3)

Phase 3: Polish — done

• Rate limiting per IP
• Structured logging (JSON)
• Health checks for monitoring
• systemd service files (§8.1)
• README + deployment docs

Phase 4: Optionals — mostly done

• Multiple conversations (session ID)
• Persisted chat history
• Multi-language (EN/ES) — detection plus a full Spanish prompt, §4.4
• Auto-compaction for long threads, §4.5
• Analysis of frequent questions
• More polished standalone CLI version (chat-bot ask)

Phase 5: Answer quality — done

Everything here came out of measuring real answers rather than from the original plan; each item exists because something was observably wrong.

• Hybrid retrieval (BM25 ⊕ embeddings, RRF) — §4.2
• Content-hash guard against stale vectors — §4.2
• Project catalogue injected every turn, to stop invented project names
• Reference documents separate from projects, so the CV is retrievable without being listed as a project — §4.1
• Vendor sampling parameters wired through config to llama.cpp
• context_size cut from 8192 to 4096 on measured usage — §4.5

Known and unfixed

Recorded so they are not re-filed as new bugs:

• The model reads dates out of the CV correctly but does the arithmetic on them wrong — "Jul 2024 – Jun 2026" reported as three years.
• It occasionally attributes a fact to the wrong source file.
• "¿Dónde ha trabajado Victor?" answers with projects rather than employers. Phrasing-specific: "¿En qué empresas ha trabajado?" and "¿Cuánto tiempo estuvo en Metrimex?" both answer correctly.

---

📐 12. Quality Specifications

12.1 Performance metrics

Metric	Target	Measured on the 2-core CPU-only target
Retrieval latency (top-5, hybrid)	<50ms	~40ms — 37ms of it is the query embedding round-trip; BM25 and the vector scan are sub-millisecond
Bot process memory	<150MB	~20MB
Total footprint (bot + LLM + embedder)	fits in 8GB with room to spare	~4.6GB (3.64 + 0.91 + 0.02), leaving ~3.4GB for the rest of the host
Generation throughput	—	21.0 tok/s steady state, ~11 tok/s on the first cold request
TTFT (Time-to-first-token)	<500ms	Not met, and not reachable here. Two threads have to prefill a ~1100-token prompt before the first token. The original target assumed a machine with a GPU
End-to-end (question → complete response)	<3s	Not met: ~21s for a typical answer. The bot is not the bottleneck; a 3B model on 2 cores is

The last two rows are the honest cost of the hardware constraint, and the widget is built around it: responses stream token by token, so the visitor sees text moving within a couple of seconds rather than waiting 21s for a block. Buying either target back means a smaller model, and the 20-question benchmark in configs/portfolio-bot.yaml measures what that costs in accuracy — gemma-3-1b averages 13.8s against qwen's 21.0s, and answers 6 fewer of every 10 questions correctly.

12.2 Required tests

• Unit tests: coverage ≥70%
• Integration tests: with mock LLM + in-memory SQLite FTS5
• E2E: at least one complete Astro → chat-bot flow

---

🔒 13. Security

13.1 Implemented

• Rate limiting per IP (default 30 req/min)
• Restrictive CORS — only configured origins
• Input validation — JSON schema validation on requests
• No PII storage — we don't save conversations by default
• Local-only by default — no calls to cloud APIs

13.2 Deferred / Optional

• Auth with API key (for private use)
• Query logging for analytics
• IP anonymization in logs
• HTTPS via reverse proxy (Caddy/nginx)

---

📚 14. References

• SSE Spec: https://html.spec.whatwg.org/multipage/server-sent-events.html
• Ollama API: https://github.com/ollama/ollama/blob/main/docs/api.md
• SQLite FTS5: https://www.sqlite.org/fts5.html
• Go SQLite driver: https://github.com/mattn/go-sqlite3 (CGO) or https://modernc.org/sqlite (pure Go)
• qwen2.5-3b-instruct: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
• nomic-embed-text-v2-moe: https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF
• Reciprocal Rank Fusion: Cormack, Clarke & Büttcher (2009), Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
• Astro API routes: https://docs.astro.build/en/guides/endpoints/
• rony-llm-agent: https://github.com/VictorVargas/rony-llm-agent

---

Document ready for implementation. 🚀