rony-chat-bot/docs/architecture.md

# 📋 Rony Chat Bot — Technical Design Document

**Version:** 1.0
**Author:** Victor Hugo Vargas
**Date:** 2026-06-28
**Status:** Complete specification for implementation
**Path:** `rony-chat-bot/docs/architecture.md`

> 🌐 **Language:** [English](./architecture.md) | [Español](./architecture.es.md)
>
> 📚 **Workspace:** This project is part of the `Rony/` workspace. See [`../README.md`](../../README.md).
>
> 🔑 **Depends on:** [`rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) — core library that provides agent loop, LLM clients, RAG, persona system.
>
> 📐 **Methodology:** This project follows the **SDD + DDD + Hexagonal Architecture** approach. Functional Requirements are numbered as `CRF-XXX`. See [`../../METHODOLOGY.md`](../../METHODOLOGY.md).

---

## 🎯 1. Project Vision

### 1.1 What is Chat-Bot?

An **HTTP chatbot** that answers questions about Victor Hugo Vargas and his projects. Uses **RAG (Retrieval-Augmented Generation)** over markdown files describing each project, and a local LLM (or cloud) to generate responses.

### 1.2 Primary use case

Victor has a portfolio website (Astro + React). On the site there's a chat widget where visitors can ask:

- "What projects has Victor done?"
- "What's his experience with Go?"
- "How does Rony Harness work?"
- "Has Victor worked with PostgreSQL?"

The bot responds with accurate information extracted from the projects' markdown files + bio + skills.

### 1.3 Secondary use cases (future)

- **Client adaptation:** The same bot, with other data and another persona, serves car dealerships, restaurants, etc.
- **Standalone CLI:** `./chat-bot ask "what do you know about X?"` for terminal use.
- **Slack/Discord bot:** Wrapper that consumes the HTTP API.

### 1.4 Philosophy

- **Self-hosted by default** — works 100% local with Ollama + 1-3B models
- **Cloud optional** — if you need more quality, swap to Anthropic API
- **Portable** — easy to fork/customize for other contexts
- **Streaming** — token-by-token responses with SSE (no waiting for complete response)
- **Reuses `rony-llm-agent`** — doesn't reinvent the agent loop

---

## 🏗️ 2. Architecture

### 2.1 Overview

```
┌─────────────────────────────────────────────────────────────────┐
│  Browser (Astro site)                                            │
│      ↓ HTTP POST /api/chat                                       │
│  Astro SSR (proxy)  ←────────── Serves portfolio + proxy chat    │
│      ↓ HTTP POST /api/chat                                       │
│  Chat-Bot HTTP server (:7331)                                    │
│      ↓                                                           │
│  Agent loop (rony-llm-agent)                                       │
│      ↓                                                           │
│  Hybrid RAG → SQLite FTS5 (BM25) ⊕ vectors, fused with RRF        │
│      ↓                    over data/projects/ + data/docs/        │
│      ├─→ embeddings server (:9200, nomic-embed-v2-moe)            │
│      ↓                                                           │
│  LLM (llama.cpp local default / Ollama or Anthropic optional)   │
└─────────────────────────────────────────────────────────────────┘
```

Two local model servers, not one. Both are plain `llama-server` processes the
bot talks to over HTTP; neither is linked into the binary.

### 2.2 Main components

| Component | Path | Responsibility |
|---|---|---|
| **HTTP server** | `internal/server/` | chi handlers, SSE streaming |
| **Agent runner** | `internal/agent/` | Wrapper over `rony-llm-agent`: prompt assembly, retrieval, language selection, compaction |
| **Portfolio loader** | `internal/portfolio/` | Reads `data/projects/` and `data/docs/` (`.md` + `.mdx`), indexes into SQLite FTS5 + vectors, hybrid search |
| **Embeddings client** | `internal/embed/` | OpenAI-compatible embeddings, normalisation, float32 codec |
| **Persona** | `internal/persona/` | Loads persona from `configs/portfolio-bot.yaml` |
| **CLI** | `cmd/chat-bot/` | Commands: `serve`, `reindex`, `ask`, `version` |

### 2.3 Tech stack

| Layer | Technology | Reason |
|---|---|---|
| **Language** | Go 1.26+ | Same as rony-harness, leverage `os.Root`, `iter.Seq` |
| **HTTP router** | `net/http` + `chi` | Stdlib + chi for middleware (CORS, logging) |
| **SSE** | `net/http` Flusher | Stdlib is enough, no external library needed |
| **Config** | `gopkg.in/yaml.v3` | Same as harness |
| **RAG backend** | SQLite + FTS5 (BM25) ⊕ dense vectors | Zero external deps, single file. No ANN index: a portfolio is hundreds of chunks, so a full scan is microseconds |
| **LLM** | llama.cpp (qwen2.5-3b-instruct Q4_K_M) — default; Ollama as alt | Self-hosted by default. 3B, not 1.5B: see the benchmark in `configs/portfolio-bot.yaml` |
| **Embeddings** | `nomic-embed-v2-moe` Q5_K_M, 768-dim | Multilingual — the whole point is matching Spanish questions to English documents |
| **Tests** | stdlib + testify | Consistency with the rest |

---

## 🔌 3. HTTP API

### 3.1 Endpoints

#### `POST /api/chat` — Chat with SSE streaming

**Request:**
```json
{
  "messages": [
    {"role": "user", "content": "What projects does Victor have?"}
  ],
  "stream": true,
  "conversation_id": "57f4aa3c7fab466bc4de9c43b296903e"
}
```

| Field | Required | Notes |
|---|---|---|
| `messages` | yes | At least one user message; alternation is not enforced. |
| `stream` | no, default `true` | `false` returns a single JSON body instead of SSE. |
| `conversation_id` | no | Hex string. If omitted, the server mints a new one and returns it (see below). Pass an existing ID to keep the thread. |

**Response (SSE):**
```
data: {"type":"start","conversation_id":"57f4aa3c7fab466bc4de9c43b296903e"}

data: {"type":"chunk","content":"Victor"}
data: {"type":"chunk","content":" has"}
data: {"type":"chunk","content":" several"}
data: {"type":"chunk","content":" projects"}

data: {"type":"sources","documents":["rony-harness.md","rony-llm-agent.md"]}

data: {"type":"done","usage":{"input_tokens":245,"output_tokens":38}}
```

The `conversation_id` in the `start` event is what the client should store
(see §3.4 — *Conversation persistence*). When the client passed an
existing ID the server echoes it back; otherwise it's freshly minted.

**Without streaming** (`"stream": false`):
```json
{
  "conversation_id": "57f4aa3c7fab466bc4de9c43b296903e",
  "content": "Victor has several projects...",
  "sources": ["rony-harness.md", "rony-llm-agent.md"],
  "usage": {"input_tokens": 245, "output_tokens": 38}
}
```

#### `GET /api/conversations` — List recent conversations

Returns the most recent conversation summaries, newest first. Useful for a
"show my chats" sidebar in a custom UI.

**Query params:**
- `limit` (1–200, default 50)

**Response:**
```json
{
  "count": 2,
  "conversations": [
    {
      "id": "57f4aa3c7fab466bc4de9c43b296903e",
      "created_at": "2026-07-17T05:02:07Z",
      "updated_at": "2026-07-17T05:04:31Z",
      "preview": "What projects does Victor have?"
    }
  ]
}
```

#### `GET /api/conversations/{id}` — Fetch one conversation

Returns the full history of a conversation with all messages in
chronological order.

**Response (200):**
```json
{
  "id": "57f4aa3c7fab466bc4de9c43b296903e",
  "created_at": "2026-07-17T05:02:07Z",
  "updated_at": "2026-07-17T05:04:31Z",
  "messages": [
    {"id": 1, "role": "user",      "content": "What projects does Victor have?", "created_at": "..."},
    {"id": 2, "role": "assistant", "content": "Victor has several projects...", "sources": ["..."], "created_at": "..."}
  ]
}
```

**Response (404):** when the ID is unknown (e.g. server DB was wiped or the
client lost sync). The widget treats this as "start fresh".

> ⚠️ **Auth note:** the conversation ID is the only access token. For a
> public bot this is fine; for private contexts add auth at the proxy layer
> (e.g. require a session cookie before forwarding to this endpoint).

#### `DELETE /api/conversations/{id}` — Delete a conversation

Removes the conversation and all its messages (cascade). Returns 204 on
success, 404 if the ID doesn't exist.

#### `POST /api/reindex` — Re-index portfolio

Useful when files in `data/projects/` are modified.

**Request:** empty
**Response:**
```json
{
  "indexed_files": 12,
  "total_chunks": 87,
  "duration_ms": 4321
}
```

#### `GET /api/health` — Health check (real)

Probes the LLM provider and the SQLite store in parallel and returns their
states. Designed for monitoring/load balancers. **Returns 200 when healthy
or degraded, 503 when unhealthy.**

- `?deep=true` adds a chunk count to the store probe (same latency budget).

**Status taxonomy:**

| `status` | HTTP | Meaning |
|---|---|---|
| `healthy` | 200 | LLM up, store up |
| `degraded` | 200 | LLM up, store down — bot still answers, just without RAG |
| `unhealthy` | 503 | LLM down — bot cannot answer, no point routing traffic here |

**Probe details:**

| Component | Probe | Latency |
|---|---|---|
| `llm` | `GET {provider}/health` (llamacpp, ollama) or `/models` (openai) | ~1ms for local llama-server |
| `store` | `SELECT 1` on the SQLite handle | ~100µs |

Each probe has a 2s timeout; the whole call returns within ~2.5s even if a
dependency hangs.

**Response shape (healthy):**

```json
{
  "status": "healthy",
  "version": "0.2.0-dev",
  "checked_at": "2026-07-17T05:02:07Z",
  "components": {
    "llm": {
      "status": "up",
      "latency": "1.028ms",
      "details": {"provider": "llamacpp", "model": "qwen2.5-3b-instruct", "url": "http://localhost:9100/health"}
    },
    "store": {
      "status": "up",
      "latency": "107µs"
    }
  }
}
```

**Response shape (degraded, with `?deep=true`):**

```json
{
  "status": "degraded",
  "version": "0.2.0-dev",
  "checked_at": "2026-07-17T05:02:07Z",
  "components": {
    "llm": {"status": "up", "latency": "0.8ms", "details": {...}},
    "store": {"status": "up", "latency": "70µs", "details": {"chunks": 28}}
  }
}
```

**Response shape (unhealthy):** HTTP 503, same JSON with `"status": "unhealthy"` and the failed component reporting `"status": "down"` plus an `error` field.

#### `GET /api/info` — Bot metadata

```json
{
  "name": "Rony Chat Bot",
  "model": "qwen2.5-3b-instruct",
  "persona": "...",
  "topics": ["projects", "experience", "technical skills"]
}
```

### 3.2 SSE Implementation

```go
// internal/server/chat.go
package server

import (
    "encoding/json"
    "fmt"
    "net/http"
    "github.com/VictorVargas/rony-llm-agent/pkg/agent"
)

func (s *Server) handleChat(w http.ResponseWriter, r *http.Request) {
    // SSE headers
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("X-Accel-Buffering", "no")
    
    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "SSE not supported", http.StatusInternalServerError)
        return
    }
    
    // Parse request
    var req ChatRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        writeError(w, flusher, "invalid request", err)
        return
    }
    
    // Start event
    writeSSE(w, flusher, "start", map[string]string{
        "conversation_id": generateConvID(),
    })
    
    // Run agent with streaming
    sources := []string{}
    for chunk, err := range s.agent.RunStream(r.Context(), req.Messages) {
        if err != nil {
            writeSSE(w, flusher, "error", map[string]string{"message": err.Error()})
            return
        }
        if chunk.Type == "source" {
            sources = append(sources, chunk.Source)
        }
        writeSSE(w, flusher, chunk.Type, chunk.Data)
    }
    
    // Done event
    writeSSE(w, flusher, "done", map[string]any{
        "usage": map[string]int{
            "input_tokens":  245,
            "output_tokens": 38,
        },
    })
}

func writeSSE(w http.ResponseWriter, flusher http.Flusher, eventType string, data any) {
    payload, _ := json.Marshal(data)
    fmt.Fprintf(w, "data: {\"type\":%q,\"data\":%s}\n\n", eventType, payload)
    flusher.Flush()
}
```

### 3.3 Middleware

```go
// internal/server/middleware.go
package server

func (s *Server) loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        // Wrap response writer to capture status
        rw := &statusRecorder{ResponseWriter: w, status: 200}
        next.ServeHTTP(rw, r)
        
        slog.Info("http.request",
            "method", r.Method,
            "path", r.URL.Path,
            "status", rw.status,
            "duration_ms", time.Since(start).Milliseconds(),
            "ip", r.RemoteAddr,
        )
    })
}

func (s *Server) corsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        origin := r.Header.Get("Origin")
        for _, allowed := range s.config.Server.CORSOrigins {
            if origin == allowed {
                w.Header().Set("Access-Control-Allow-Origin", origin)
                w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS")
                w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
                break
            }
        }
        if r.Method == "OPTIONS" {
            w.WriteHeader(204)
            return
        }
        next.ServeHTTP(w, r)
    })
}

func (s *Server) rateLimitMiddleware(next http.Handler) http.Handler {
    limiter := rate.NewLimiter(rate.Every(time.Minute/time.Duration(s.config.Server.RateLimit.RequestsPerMinute)), s.config.Server.RateLimit.Burst)
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !limiter.Allow() {
            http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
            return
        }
        next.ServeHTTP(w, r)
    })
}
```

### 3.4 Conversation persistence

The bot persists conversation threads in the same SQLite database as the
RAG index (`./data/portfolio.db`). Schema lives in `internal/portfolio/conversations.go`.

```sql
CREATE TABLE conversations (
    id         TEXT PRIMARY KEY,         -- 16-byte random hex (32 chars)
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);

CREATE TABLE messages (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    conversation_id TEXT    NOT NULL,
    role            TEXT    NOT NULL,    -- user | assistant | system
    content         TEXT    NOT NULL,
    sources         TEXT,                 -- JSON array, nullable
    created_at      INTEGER NOT NULL,
    FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE CASCADE
);
CREATE INDEX idx_messages_conv ON messages(conversation_id, id);
```

**Lifecycle:**

| When | What |
|---|---|
| `POST /api/chat` (no `conversation_id`) | Server mints a new hex ID, returns it in the `start` SSE event (or `conversation_id` field of the JSON response) |
| `POST /api/chat` (with `conversation_id`) | Server reuses the existing row; both user message and assistant reply are appended |
| User message | Persisted **before** the LLM runs, so it survives a model failure |
| Assistant message | Persisted **after** the stream completes, with the RAG sources attached |
| `GET /api/conversations/{id}` | Returns the full thread; 404 if unknown |
| `DELETE /api/conversations/{id}` | Cascade-deletes messages |

**Client responsibilities:**

1. On the first message, omit `conversation_id`. Capture the one the server
   returns in the `start` SSE event.
2. Store it client-side (`localStorage["rony-chat-conv"]` in the widget).
3. On every subsequent message, send the ID back.
4. On page load, if you have a stored ID, call `GET /api/conversations/{id}`
   to restore the thread. If 404, clear the stored ID and start fresh.

The widget (`web/chat-widget.js`) implements all four steps. Any other
client (a custom React component, an Astro endpoint, a CLI replay tool)
follows the same protocol.

**Auth model:**

The conversation ID is the only access token for `GET /api/conversations/{id}`.
It is 128 bits of random entropy, so guessing one is infeasible. For a
public portfolio bot this is the right trade-off — anyone who knows the
URL can read its history. For private contexts, add an auth layer in front
of the bot (proxy) that gates the conversation endpoints.

---

## 🧠 4. RAG (Retrieval-Augmented Generation)

> ✅ **The four decisions this section left open have been made and measured:**
>
> - **FTS5 tokenizer** — `unicode61 remove_diacritics 2`, as specced. Stemming
>   was not the bottleneck; the language gap was, and embeddings close it.
> - **SQLite driver** — `modernc.org/sqlite` (pure Go, no CGO). Benchmark below.
> - **Chunking** — by markdown heading, not fixed size, with oversized sections
>   split at `###` before falling back to byte offsets. §4.1.
> - **Semantic similarity** — added. BM25 alone returned **nothing at all** for
>   a Spanish question about an English document. §4.2.

### 4.0 Driver decision: benchmark results

Reproducible con `CGO_ENABLED=1 go test -tags sqlite_fts5 -bench=. ./bench/`. Datos: 4 markdowns → 11 chunks.

| Operación | mattn (CGO) | modernc (puro Go) | Diferencia |
|---|---|---|---|
| **Insert** (11 chunks) | 2,802,843 ns/op | **1,465,646 ns/op** | modernc 1.9× más rápido |
| Insert alloc | 2,124,299 B/op | **9,770 B/op** | modernc usa 217× menos memoria |
| **Query** (8 queries BM25) | **244,047 ns/op** | 555,162 ns/op | mattn 2.3× más rápido |
| **Round-trip** (insert + 8 queries) | 3,543,417 ns/op | **2,267,669 ns/op** | modernc 1.6× más rápido |
| Binary size | 11 MB | 11 MB | igual |
| Build deps | gcc, CGO=1 | nada | modernc gana |
| CI/CD portable | requiere toolchain C | `go build` puro | modernc gana |

**Decisión: `modernc.org/sqlite`**.

Justificación:
1. Ambas latencias de query (~250µs vs ~550µs) son **2 órdenes de magnitud por debajo** del target de 50ms — imperceptible vs el LLM (varios segundos).
2. modernc gana en inserts (1.9×) y round-trip (1.6×), que es el path de reindex.
3. Sin CGO = CI/CD más simple (sin gcc, sin Alpine musl-dev, binarios reproducibles).
4. Si en el futuro el cuello de botella pasa a ser query latency (corpus >10k chunks), se puede reconsiderar. Hoy no.

### 4.1 Indexing pipeline

```
data/projects/*.{md,mdx}   kind=project → announced in the catalogue
data/docs/*.{md,mdx}       kind=doc     → retrievable, never announced
    ↓ (skip each directory's README — those are instructions, not content)
Raw markdown
    ↓ (split by heading; sections over the limit split at ###, then by size)
    ↓ (drop the frontmatter chunk)
Chunks
    ├─→ SQLite FTS5 virtual table "portfolio_chunks"
    └─→ embeddings endpoint → "portfolio_vectors" (id, content_hash, dim, vec)
Indexed corpus
```

**Two kinds of source.** Everything under `data_path` is one of Victor's
projects and is listed in the catalogue injected into every prompt. Everything
under `docs_path` is searchable evidence that is *not* a project — his CV, an
about page. The CV is what someone deciding whether to hire actually reads, and
it was unreachable while it lived only in the Astro site; but filing it under
projects made the bot list "cv" as one of his works.

**Three things are excluded or reshaped, each because of a measured failure:**

| Rule | Failure it fixes |
|---|---|
| Skip `README*` in both directories | `data/projects/README.md` was indexed, so the catalogue announced "README" and "README.es" as projects of Victor's |
| Drop the frontmatter chunk | Dense metadata in a very short chunk is a magnet for short queries — a CV's `location:` field answered *"¿Dónde ha trabajado Victor?"* with a city instead of a work history |
| Split oversized sections at `###` | A CV's Experience section is a list of jobs; size-splitting cut one mid-word, stranding the employer's name in the previous chunk. Chunks now hold one job each, named `Experience — Metrimex — Frontend Developer` |

**When it runs:**
- Manually: `./chat-bot reindex`
- On startup with `serve --reindex-on-start`
- Via HTTP: `POST /api/reindex`

Vectors are built at index time, so enabling embeddings requires a reindex.
The chunk table is derived data: `OpenStore` rebuilds it when its columns are
missing, so upgrading an existing install needs no migration step. Conversation
tables are never touched by that rebuild.

### 4.2 Retrieval pipeline

```
User query "¿Con qué se paga en la tienda de ropa?"
    ↓
    ├─→ FTS5 MATCH, BM25 ranking      → top 15 (topK × 3)
    └─→ embed(query) → cosine vs vecs → top 15 (topK × 3)
    ↓ (Reciprocal Rank Fusion, k=60)
Top 5 chunks
    ↓ (system prompt + project catalogue + excerpts + language directive)
LLM generates answer
```

**Both halves are load-bearing.** Keyword search matches words exactly — no
stemming, no translation. The corpus is written in English and visitors ask in
Spanish, so the words carrying the meaning score zero: measured on the real
corpus, `"paga"` appears 0 times in a document that says *"Payments: Stripe"*
and `"trabajado"` 0 times in one that says *"worked"*. The question above
retrieved **nothing at all**. Embeddings (`nomic-embed-v2-moe`, multilingual)
put all three `tienda-ropa` chunks on top — but they blur exact rare tokens,
where BM25 is sharp.

**Why RRF rather than a weighted score.** A BM25 score and a cosine share no
scale, so blending them numerically means inventing a conversion factor and
retuning it whenever the corpus changes. RRF ignores the magnitudes and ranks
by agreement between the two orderings: each list contributes `1/(60 + rank)`.
A chunk both halves like beats one that either loves alone.

**Degradation.** If the embeddings endpoint is down or disabled, the vector
half returns nothing and retrieval continues keyword-only rather than failing
the request.

**Stale vectors.** Chunk ids are derived from position, so they survive edits
to the body. Without a guard, editing a document leaves vectors that still
describe the text that was removed — reproduced live by changing a project's
payment provider and watching the old one keep coming back. Each vector stores
a hash of the exact text it was built from, and rows whose hash no longer
matches the chunk are skipped with a warning until the next reindex.

**The catalogue.** Top-K returns the best-matching *sections*, so a broad
question like "what has Victor built?" cannot be answered from retrieval alone,
and a small model asked to enumerate from partial hits invents the rest. The
full project list is injected every turn at a cost of ~10 tokens per project.
This is what stopped the bot naming projects that do not exist.

### 4.3 Schema

```go
// internal/portfolio/indexer.go
const schema = `
CREATE VIRTUAL TABLE IF NOT EXISTS portfolio_chunks USING fts5(
    id UNINDEXED,
    project_id UNINDEXED,
    kind UNINDEXED,          -- 'project' | 'doc'
    source_file UNINDEXED,
    section UNINDEXED,       -- the heading this chunk came from
    chunk_index UNINDEXED,
    content,
    tokenize = 'unicode61 remove_diacritics 2'
);
`

// Vectors live in an ordinary table keyed by chunk id. There is no ANN index:
// a portfolio is hundreds of chunks, not millions, so a full scan with a dot
// product is microseconds and needs no extension.
const vectorSchema = `
CREATE TABLE IF NOT EXISTS portfolio_vectors (
    chunk_id     TEXT PRIMARY KEY,
    content_hash TEXT NOT NULL,   -- sha256 of the exact text embedded
    dim          INTEGER NOT NULL,
    vec          BLOB NOT NULL    -- little-endian float32, unit-normalised
);
`
```

FTS5 has no `ALTER TABLE ADD COLUMN`, so `ensureChunkSchema` drops and
recreates `portfolio_chunks` when an older database is missing a column. That
is safe precisely because the table is a derived index — every row is
regenerated from the markdown on the next reindex. It deliberately touches
only the index tables; conversations live in the same file and are real user
data.

Vectors are unit-normalised at write time, so a dot product **is** the cosine
and retrieval needs no division per comparison. `dim` is stored so a change of
embedding model is detected rather than silently producing garbage similarity:
rows whose dimension does not match the query vector are skipped.

**Key API:**

| Function | File | Purpose |
|---|---|---|
| `SourcesFor(dataPath, docsPath)` | `indexer.go` | Builds the `[]Source` pair; an empty path is skipped, so `docs_path` is optional without branching |
| `Store.Reindex(ctx, sources, cfg)` | `indexer.go` | Rebuilds both tables from disk |
| `Store.Search(ctx, query, topK)` | `indexer.go` | BM25 only, excludes `section = 'frontmatter'` |
| `Store.HybridSearch(ctx, emb, q, topK)` | `hybrid.go` | BM25 + vectors fused with RRF |
| `Store.Catalog(ctx)` | `indexer.go` | Project list for the prompt; filters `kind = 'project'` |
| `embed.Client.Embed(ctx, texts)` | `internal/embed/` | OpenAI-compatible embeddings, reordered by the response `index` field |

### 4.4 Assembling the prompt

`agent.Runner.BuildMessages` runs once per request and produces exactly one
system message followed by the conversational turns. The order of operations
is load-bearing:

```go
// internal/agent/runner.go
func (r *Runner) BuildMessages(ctx context.Context, history []Message) ([]Message, string, error) {
    // 1. Pull system-role notes (the compactor's summary) out of the turn list.
    notes, history := foldSystemNotes(history)

    // 2. The visitor's language selects which rendition of the prompt we
    //    build on, so it is resolved before anything else.
    lang := detectLanguage(history)
    systemPrompt := r.promptFor(lang)

    // 3. Resolved before retrieval so the excerpt budget accounts for it.
    catalog := r.catalogBlock(ctx)

    // 4. Hybrid retrieval on the last user turn.
    hits, err := r.store.HybridSearch(ctx, r.embedder, last.Content, r.topK)
    ragContext = r.limitRAGContext(systemPrompt, catalog, formatHits(hits), history)

    // 5. prompt → catalogue → excerpts → summary → language directive.
    system := botpersona.BuildSystemPrompt(systemPrompt, catalog, ragContext)
    system += "\n\n" + strings.Join(notes, "\n\n")
    system += "\n\n" + r.languageDirective(lang)

    history, err = r.fitHistory(system, history)
    return append([]Message{{Role: RoleSystem, Content: system}}, history...), ragContext, nil
}
```

**Why exactly one system message.** Gemma 3's chat template raises
*"Conversation roles must alternate user/assistant/..."* on any system message
after the first, which llama-server surfaces as HTTP 400 — enabling compaction
used to kill the conversation outright the first time it fired. Anthropic's API
rejects mid-conversation system turns too, so folding is the portable
behaviour rather than a Gemma workaround. `foldSystemNotes` never aliases the
caller's slice; the handler reuses the history it passes in.

**Why the language directive goes last.** It is the instruction a small model
is most likely to still be holding when it starts generating. Position was not
enough on its own, though — see below.

**Matching the visitor's language** took three attempts, measured on gemma-3-1b
over the same five Spanish questions:

| Approach | Result |
|---|---|
| English prompt + "reply in the user's language" | 1/5 answered in Spanish |
| English prompt + Spanish few-shot examples | 5/5 Spanish, but ~2/5 were the example reply copied verbatim instead of an answer |
| A full Spanish rendition of the prompt (`system_prompt_es`) | 4/5 Spanish, 4/5 real answers |

So `internal/i18n` detects the language and `promptFor` selects the rendition;
the directive reinforces a prompt already written in the right language rather
than trying to override one written in the wrong one. The two renditions in
the YAML have to be kept in sync by hand.

**Why keyword search alone was not enough.** The original plan cited three
reasons to skip embeddings — no model to run, one file and one driver, and
BM25 being strong on structured project docs. The first two still hold and
cost what was predicted (~0.91 GB resident, a second process). The third was
right about the corpus and wrong about the questions: BM25 is excellent at
retrieving English documents given English keywords, and this bot is asked in
Spanish. That is not a morphology problem a `trigram` tokenizer fixes — it is a
translation problem. Hence §4.2, and hence both halves are kept.

---

## 🗜️ 4.5 Auto-compaction

Long conversations eventually run out of context. Auto-compaction folds the older portion of the conversation into a single summary system message when the previous turn's input tokens cross a configurable threshold.

**How much headroom there actually is.** Measured over 20 real requests at `context_size: 4096`, the largest prompt this bot ever built was **1255 tokens** — system prompt, project catalogue, five retrieved chunks and the question — with a median of 1069. That leaves room for roughly a dozen short turns before the 75% threshold (~3070 tokens) is reached, not the 2–3 an earlier draft of this document estimated. Compaction is therefore a safety net for genuinely long threads rather than something that fires in a typical visit; across the whole benchmark it never triggered and nothing was truncated.

### When it fires

`agent.Runner.Compact` runs once per `/api/chat` request, before the RAG search. It compares the runner's most recent `Usage.InputTokens` (reported by the provider in the previous streamed chunk) against `client.Capabilities().MaxContextWindow × threshold_ratio`.

| Setting | Default | What it controls |
|---|---|---|
| `compaction.enabled` | `false` | Master switch. |
| `compaction.threshold_ratio` | `0.75` | Trigger when used tokens ≥ window × ratio. |
| `compaction.keep_recent_turns` | `4` | How many of the latest user turns are kept verbatim after compaction. |
| `compaction.summary_system_prompt` | *(built-in bilingual)* | Override the instruction sent to the LLM when summarizing. |

Short-circuits silently when compaction is disabled, the provider doesn't report a window (`Capabilities().MaxContextWindow == 0`), the history is shorter than `keep_recent_turns`, or usage is still unknown (first turn).

### How the summary is made

1. `splitByTurns(history, keep_recent_turns)` divides messages into `(older, recent)` on user-role boundaries so a kept turn's user/assistant pair always stays together.
2. `renderTranscript(older)` flattens older messages into a `User:` / `Assistant:` transcript (skipping tool messages and empty assistant placeholders).
3. The runner calls `client.Generate(...)` with the summary prompt + transcript and a 512-token cap so the compaction step itself stays cheap.
4. The returned text is prepended as a system message (`"Earlier conversation summary:\n…"`), followed by the recent tail.
5. `LastCompaction()` returns `CompactionStats` so the SSE handler can emit a `compaction` event right before the streamed chunks.

### Failure mode

If `Generate` errors or returns an empty summary, compaction falls back to `truncateToBudget`: drop oldest user-turns one at a time until the remaining slice fits `threshold` tokens (heuristic: `len(s) / 4 + 1`). The current user turn is always preserved. The fallback is logged at WARN and the request still proceeds — a flaky summarize call never fails the user's request.

### Wire protocol

Streaming responses gain an optional `compaction` event:

```
data: {"type":"compaction","older_turns":6,"kept_turns":2,"summary_tokens":120,"window_tokens":4096,"used_tokens":3500}
```

Emitted after `start` (when applicable) and before `sources` / `chunk`. The widget can render this as a subtle "Context compacted" hint or ignore it — both are valid.

### Persistence

Compaction is **per-request**. The full transcript is still saved to `messages` in `data/portfolio.db` verbatim, so `GET /api/conversations/{id}` always returns the original history. Only what we send to the LLM is reduced — the next session can re-read the full thread from the DB.

---

## 🌐 5. Embedding the widget

The bot ships with a drop-in vanilla-JS widget. Add two files to your site and it works.

### 5.1 The widget (any site)

```html
<link rel="stylesheet" href="/path/to/chat-widget.css">
<script src="/path/to/chat-widget.js"
        data-api-url="https://chat.example.com"
        data-title="Ask me anything"
        data-greeting="Hi! Ask me about the projects."
        data-position="bottom-right"
        data-theme="auto"
        defer></script>
```

A bubble appears bottom-right, opens a panel, talks SSE to `/api/chat`, streams the response, and cites sources. No build step, no React/Vue, no framework lock-in.

**Browser→bot options:**

| Topology | Trade-offs |
|---|---|
| **Direct** (browser → bot, same domain or CORS) | Simplest. Add the bot's origin to `cors_origins` in YAML. |
| **Reverse proxy** (nginx/Caddy in front) | Bot stays on private network, single public domain, no CORS to manage. |
| **Site proxies the bot** (Astro/Next API route) | Adds a hop and a bit of code, but gives you auth/session hooks in your site. |

The widget works the same in all three. Pick the topology that matches your infra.

> **Default dev setup is direct + CORS.** `cors_origins` in `configs/portfolio-bot.yaml` controls which sites can call the bot. Add your site's origin there.

### 5.2 Astro: drop-in via Layout

The widget works in Astro without writing a React component. Add this to your shared layout:

```astro
---
// src/layouts/BaseLayout.astro
import "../path/to/chat-widget.css";
const apiUrl = import.meta.env.PUBLIC_CHAT_API_URL || "http://localhost:7331";
---
<html>
  <body>
    <slot />
    <script src="/path/to/chat-widget.js"
            data-api-url={apiUrl}
            data-title="Ask me anything"
            data-position="bottom-right"
            data-theme="auto"
            defer is:inline></script>
  </body>
</html>
```

`is:inline` keeps Astro from hashing/transforming the script tag, so the `data-*` attributes survive.

### 5.3 React / Next.js: same script tag

```tsx
// app/layout.tsx
import Script from "next/script";

export default function RootLayout({ children }) {
  return (
    <html>
      <head>
        <link rel="stylesheet" href="/chat-widget.css" />
        <Script src="/chat-widget.js"
                data-api-url={process.env.NEXT_PUBLIC_CHAT_API_URL}
                data-title="Ask me anything"
                data-position="bottom-right"
                data-theme="auto"
                strategy="afterInteractive" />
      </head>
      <body>{children}</body>
    </html>
  );
}
```

### 5.4 If you want a server proxy (Astro/Next API route)

The widget can also call a same-origin endpoint that forwards to the bot. This is the right call when you need:
- Auth on `/api/chat` (logged-in users only)
- Centralized rate limiting at the site level
- Hiding the bot's origin from the browser

```typescript
// src/pages/api/chat.ts (Astro) or app/api/chat/route.ts (Next)
const CHAT_BOT_URL = process.env.CHAT_BOT_URL || "http://localhost:7331";

export const POST = async ({ request }) => {
    const body = await request.json();
    // (optional) auth check, rate limit, session lookup here

    const resp = await fetch(`${CHAT_BOT_URL}/api/chat`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(body),
    });

    return new Response(resp.body, {
        status: resp.status,
        headers: {
            "Content-Type": "text/event-stream",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        },
    });
};
```

Then point the widget at `/api/chat` (same origin) instead of the bot's URL.

### 5.5 Widget configuration reference

All options are `data-*` attributes on the `<script>` tag:

| Attribute | Default | Notes |
|---|---|---|
| `data-api-url` | *(required)* | Base URL of the bot. No trailing slash. |
| `data-title` | `"Chat"` | Header text. |
| `data-greeting` | `""` | First assistant message when the panel opens. |
| `data-position` | `"bottom-right"` | `"bottom-right"` or `"bottom-left"`. |
| `data-theme` | `"auto"` | `"auto"` (follows OS), `"light"`, `"dark"`. |

Theming is via CSS custom properties on `.rony-chat-widget-root` (see `web/chat-widget.css`):

```css
.rony-chat-widget-root {
  --rony-accent: #ff6b35;
  --rony-radius: 4px;
  --rony-font: "Inter", sans-serif;
}
```

### 5.6 What the widget doesn't do (yet)

- **Richer markdown** (tables, images) — the built-in renderer handles the common cases; for full CommonMark, swap `renderMarkdown` in `chat-widget.js` for `marked` or `markdown-it`.
- **Mobile swipe-to-dismiss** — panel goes full-screen on phones.
- **Conversation history sidebar** — only the active conversation is shown (the backend exposes `GET /api/conversations` for a future sidebar).

---

## 🤖 6. Self-hosting with llama.cpp (default)

### 6.1 Setup

llama-server is a separate process that the bot connects to over HTTP. **Both ports (the bot's and llama-server's) are configurable** — pick what fits your environment.

```bash
# 1. Make sure you have a GGUF model available
# Download from Hugging Face, e.g.:
#   https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF
export RONY_MODELS_PATH=/path/to/models
ls $RONY_MODELS_PATH/qwen2.5-3b-instruct-q4_k_m.gguf

# 2. Start llama-server (port is configurable; default llama.cpp is 8080)
llama-server \
  -m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --host 127.0.0.1 \
  --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05

# 3. Start the embeddings server (second process, second terminal)
llama-server \
  -m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2

# 4. Make sure configs/portfolio-bot.yaml points to the same ports
#    providers[0].endpoint: http://localhost:9100/v1
#    embeddings.endpoint:   http://localhost:9200/v1

# 5. Build the index (needs the embedder up — vectors are built here)
./bin/chat-bot reindex

# 6. Start the bot (default port 7331, also configurable)
./bin/chat-bot serve
# → Serves on http://localhost:7331
# → Override with: ./bin/chat-bot serve --port 9101 --host 127.0.0.1
```

**Flags that are not optional, each for a measured reason:**

| Flag | Why |
|---|---|
| `--device none` | llama.cpp brings up a compiled-in GPU backend even with `-ngl 0`, and on a GPU-less host those buffers come out of system RAM. Measured on qwen2.5-3b: 2.54 GB with a GPU absorbing them, **3.66 GB without**. Budget from the second number |
| `--parallel 1` | `--ctx-size` is divided across slots and the default is 4, so `--ctx-size 4096` without it gives each request 1024 tokens |
| `--mlock` | Prevents swap — critical on a shared VPS |
| `--temp` / `--top-k` / `--top-p` | Use the model authors' published values, not llama.cpp's defaults. gemma-3-1b at `temperature 0.7` with the rest unset produced 16-token stub answers |
| `--pooling mean` (embedder) | Without it the endpoint does not return one vector per input and the client rejects the response |

**Port reference:**

| What | Default | How to change |
|---|---|---|
| `llama-server` HTTP port | 8080 (llama.cpp convention) | `--port N` flag when starting `llama-server` |
| embeddings `llama-server` port | 8080 (same convention) | `--port N`; this repo uses 9200 |
| chat-bot HTTP port | 7331 | `--port N` flag on `serve`, or `server.port` in YAML |
| chat-bot → llama-server URL | `http://localhost:8080/v1` | `endpoint` field on the provider in YAML |

Keep `context_size` in the YAML equal to `--ctx-size`: the bot sizes its RAG
and compaction budgets from that number and never asks the server what it
actually has, so a mismatch means prompts the server rejects.

The `llamacpp` provider is imported from `rony-llm-agent/pkg/llm/providers/llamacpp` and speaks HTTP to the server above — no CGO, no linking against llama.cpp.

### 6.2 Alternative: Ollama (easier for development)

If you don't want to manage GGUF files manually, Ollama provides the same models with a simpler workflow:

```bash
# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 2. Download chat model
ollama pull qwen2.5:1.5b

# 3. Verify
ollama list

# 4. Edit configs/portfolio-bot.yaml to mark ollama-local as default:
#    providers[0].default: true (and remove default from llamacpp-local)
#    Ollama exposes an OpenAI-compatible API on :11434/v1

# 5. Start the bot
ollama serve &
./bin/chat-bot serve
```

### 6.3 Alternative: llama.cpp direct (advanced)

For more control or if Ollama doesn't work in your setup:

```yaml
providers:
  - name: llamacpp-local
    type: llamacpp
    model: qwen2.5-3b-instruct
    endpoint: http://localhost:9100/v1   # configurable, see §6.1
    context_size: 4096                   # must match --ctx-size
    max_tokens: 640
    temperature: 0.7                     # Qwen's published instruct defaults
    top_k: 20
    top_p: 0.8
    repeat_penalty: 1.05
    default: true
```

The `llamacpp` adapter is imported from `rony-llm-agent/pkg/llm/providers/llamacpp` and speaks HTTP to `llama-server` — no CGO, no linking against llama.cpp.

---

## 📦 7. Bot CLI

### 7.1 Commands

```bash
# Start HTTP server
chat-bot serve [--port 7331] [--host 0.0.0.0] [--reindex-on-start]

# Re-index (reads data/projects/ + data/docs/ *.md and *.mdx → FTS5 + vectors)
# Needs the embeddings server up if embeddings are enabled.
chat-bot reindex

# Single question (no server, useful for tests)
chat-bot ask "What projects does Victor have?" [--no-rag]

# Validate config
chat-bot config validate

# Health check (useful for monitoring)
chat-bot health

# Version
chat-bot version
```

### 7.2 Implementation with Cobra

```go
// cmd/chat-bot/main.go
package main

import (
    "github.com/spf13/cobra"
)

func main() {
    root := &cobra.Command{
        Use:   "chat-bot",
        Short: "Portfolio chatbot HTTP server",
    }
    
    root.AddCommand(serveCmd())
    root.AddCommand(reindexCmd())
    root.AddCommand(askCmd())
    root.AddCommand(configCmd())
    root.AddCommand(healthCmd())
    root.AddCommand(versionCmd())
    
    if err := root.Execute(); err != nil {
        os.Exit(1)
    }
}

func serveCmd() *cobra.Command {
    var port int
    var host string
    var reindexOnStart bool
    
    cmd := &cobra.Command{
        Use:   "serve",
        Short: "Start HTTP server",
        RunE: func(cmd *cobra.Command, args []string) error {
            return server.Serve(server.Config{
                Port:           port,
                Host:           host,
                ReindexOnStart: reindexOnStart,
            })
        },
    }
    
    cmd.Flags().IntVar(&port, "port", 7331, "HTTP port")
    cmd.Flags().StringVar(&host, "host", "0.0.0.0", "HTTP host")
    cmd.Flags().BoolVar(&reindexOnStart, "reindex-on-start", false, "Re-index RAG before serving")
    
    return cmd
}
```

---

## 🚀 8. Deployment

### 8.1 Recommendation: Self-hosted on VPS

Target: **2 CPU cores, 8 GB RAM, no GPU.** Three processes — the bot and two
`llama-server` instances — so three units. The bot depends on both.

```bash
# 1. Build
go build -o /usr/local/bin/chat-bot ./cmd/chat-bot

# 2. The LLM
cat > /etc/systemd/system/llama-chat.service <<EOF
[Unit]
Description=llama-server (chat model)
After=network.target

[Service]
Type=simple
User=chatbot
ExecStart=/usr/local/bin/llama-server \\
  -m /opt/models/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \\
  --port 9100 --host 127.0.0.1 \\
  --ctx-size 4096 --parallel 1 \\
  --device none --threads 2 --mlock \\
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
Restart=on-failure
# --mlock needs the memory to be lockable
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

# 3. The embedder
cat > /etc/systemd/system/llama-embed.service <<EOF
[Unit]
Description=llama-server (embeddings)
After=network.target

[Service]
Type=simple
User=chatbot
ExecStart=/usr/local/bin/llama-server \\
  -m /opt/models/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \\
  --port 9200 --host 127.0.0.1 \\
  --embedding --pooling mean \\
  --ctx-size 2048 --parallel 1 --device none --threads 2
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 4. The bot
cat > /etc/systemd/system/chat-bot.service <<EOF
[Unit]
Description=Portfolio Chat Bot
After=network.target llama-chat.service llama-embed.service
Wants=llama-chat.service llama-embed.service

[Service]
Type=simple
User=chatbot
WorkingDirectory=/opt/chat-bot
ExecStart=/usr/local/bin/chat-bot serve
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now llama-chat llama-embed chat-bot

# 5. Build the index once both model servers are up
sudo -u chatbot /usr/local/bin/chat-bot reindex
```

**Memory budget.** 3.64 GB (LLM) + 0.91 GB (embedder) + 0.02 GB (bot) ≈
**4.6 GB resident**, leaving ~3.4 GB for whatever else shares the VPS. Budget
with `--device none` in place: without it the numbers look ~1.1 GB smaller on
a machine with a GPU and then do not reproduce in production. See
[`vps-context-sizing.md`](./vps-context-sizing.md).

`reindex` has to be re-run after editing the markdown **and** after enabling
or changing the embeddings model — vectors are built at index time, and a
model change alters the dimension.

### 8.2 Reverse proxy (Caddy)

```
# /etc/caddy/Caddyfile
chat.victorvargas.dev {
    reverse_proxy localhost:7331
}
```

### 8.3 Monitoring

```bash
# Health check periodic
curl -s http://localhost:7331/api/health | jq

# Logs
journalctl -u chat-bot -f
```

---

## 🧪 9. Testing

### 9.1 Unit tests

```go
// internal/server/chat_test.go
package server

func TestHandleChat_ValidRequest(t *testing.T) {
    s := newTestServer(t)
    
    req := httptest.NewRequest("POST", "/api/chat", strings.NewReader(`{
        "messages": [{"role": "user", "content": "hello"}]
    }`))
    req.Header.Set("Content-Type", "application/json")
    
    w := httptest.NewRecorder()
    s.handleChat(w, req)
    
    assert.Equal(t, 200, w.Code)
    assert.Equal(t, "text/event-stream", w.Header().Get("Content-Type"))
}

func TestHandleChat_RateLimit(t *testing.T) {
    s := newTestServerWithConfig(t, server.Config{
        RateLimit: 1, // 1 request per minute
    })
    
    // First request OK
    req1 := newChatRequest("hello")
    w1 := httptest.NewRecorder()
    s.handleChat(w1, req1)
    assert.Equal(t, 200, w1.Code)
    
    // Second request denied
    req2 := newChatRequest("hello again")
    w2 := httptest.NewRecorder()
    s.handleChat(w2, req2)
    assert.Equal(t, 429, w2.Code)
}
```

### 9.2 Integration tests with mock LLM

```go
// internal/agent/runner_test.go
func TestRunner_RAGContextIsInjected(t *testing.T) {
    mockLLM := mock.New(mock.Responses{
        {Match: "projects", Response: "Victor has several projects..."},
    })
    
    memory := newMockMemoryWithDocs(t, []rag.Fragment{
        {Content: "Rony Harness: AI agent harness...", ProjectID: "rony-harness"},
        {Content: "rony-llm-agent: Go library...", ProjectID: "rony-llm-agent"},
    })
    
    runner := agent.NewRunner(agent.Config{
        LLM:    mockLLM,
        Memory: memory,
        Persona: testPersona,
    })
    
    resp, _ := runner.Run(context.Background(), []llm.Message{
        {Role: llm.RoleUser, Content: "what projects does Victor have?"},
    })
    
    // Verify LLM received context chunks in system prompt
    lastReq := mockLLM.LastRequest()
    assert.Contains(t, lastReq.Messages[0].Content, "Rony Harness")
    assert.Contains(t, lastReq.Messages[0].Content, "rony-llm-agent")
}
```

### 9.3 E2E test with Astro

```bash
# 1. Start chat-bot on :7331
./bin/chat-bot serve &

# 2. Start Astro on :4321
cd ../portfolio && npm run dev &

# 3. Make request to Astro's proxy
curl -X POST http://localhost:4321/api/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hello"}]}'

# 4. Verify SSE stream
```

### 9.4 Retrieval tests

Retrieval is the part of this bot where a regression is silent — nothing
errors, answers just get subtly worse — so each failure found in testing has a
test pinning it. The code samples above are illustrative; these are real.

| Test | What it pins |
|---|---|
| `TestHybridSearchFindsChunkKeywordSearchCannot` | The Spanish-question / English-document gap, the reason embeddings exist |
| `TestHybridSearchFallsBackWhenEmbedderFails` | A dead embeddings endpoint degrades to keyword-only, never fails the request |
| `TestVectorSearchIgnoresVectorsWhoseChunkChanged` | Stale vectors after a body edit are skipped |
| `TestVectorSearchIgnoresMismatchedDimensions` | Changing the embedding model does not produce garbage similarity |
| `TestReindexSkipsTheDirectoryReadmes` | The catalogue never announces `README` as a project |
| `TestReindexIndexesMdxAndSeparatesDocsFromProjects` | `.mdx` is indexed; the CV is retrievable but never listed |
| `TestOpenStoreMigratesIndexWithoutKindColumn` | Upgrading an existing install needs no migration step |
| `TestSubSplitPrefersH3BoundariesOverByteOffsets` | CV jobs stay whole instead of being cut mid-word |
| `TestBuildMessagesFoldsSystemNotesIntoOneSystemMessage` | Compaction cannot re-introduce the HTTP 400 |
| `TestLanguageDirectiveIsLastInSystemPrompt` | The directive keeps the position it needs to work |

They run against real SQLite and a stub embedder, so no model server is
needed: `go test ./...` is enough.

---

## 📂 10. Project Structure

```
rony-chat-bot/
├── cmd/
│   └── chat-bot/
│       └── main.go                 # CLI entrypoint
│
├── internal/
│   ├── server/                     # HTTP handlers
│   │   ├── server.go               # chi router + middleware
│   │   ├── handlers.go             # /api/chat, /api/health, /api/info, /api/reindex, /api/conversations
│   │   ├── conversations_test.go   # round-trip, continue, list, 404, delete, streaming
│   │   └── middleware.go           # RequestID, Logging, CORS, RateLimit
│   │
│   ├── agent/                      # LLM client + RAG runner
│   │   ├── runner.go               # Stream wrapper, RAG injection into system prompt
│   │   └── client.go               # NewClient factory: llamacpp / ollama / openai / anthropic
│   │
│   ├── portfolio/                  # RAG: markdown → SQLite FTS5 + vectors, conversation persistence
│   │   ├── chunker.go              # Heading-based splitter, ### sub-split
│   │   ├── indexer.go              # Store: schema, Reindex, Search (BM25), Catalog
│   │   ├── hybrid.go               # HybridSearch: BM25 ⊕ vectors via RRF, EmbedChunks
│   │   ├── conversations.go        # Conversation + Message CRUD, persisted alongside RAG
│   │   └── chunker_test.go / store_test.go / hybrid_test.go
│   │
│   ├── embed/                      # Embeddings client
│   │   ├── embed.go                # Embed, Normalize, Similarity, Encode/Decode
│   │   └── embed_test.go
│   │
│   ├── persona/                    # Persona bridge to rony-llm-agent
│   │   └── persona.go              # FromConfig, BuildSystemPrompt (with RAG context)
│   │
│   ├── streaming/                  # SSE protocol helpers
│   │   └── sse.go                  # WriteStart/Chunk/Sources/Done/Error
│   │
│   ├── i18n/                       # Language detection (ES/EN) for the response
│   │
│   └── config/                     # YAML loader + validation
│
├── web/                            # ← DROP-IN CHAT WIDGET
│   ├── chat-widget.js              # Vanilla JS, ~12 KB
│   ├── chat-widget.css             # Scoped styles, CSS-custom-prop themable
│   ├── example.html                # Local demo (python -m http.server)
│   └── README.md                   # Integration guide (HTML, Astro, Next.js)
│
├── data/
│   ├── projects/                   # ← One .md/.mdx per project — listed in the catalogue
│   │   ├── rony-harness.md
│   │   ├── rony-llm-agent.md
│   │   ├── example-project.md
│   │   └── README.md               # Instructions; skipped by the indexer
│   │
│   └── docs/                       # ← Reference material that is NOT a project
│       ├── cv.mdx                  # Usually a symlink; gitignored
│       └── README.md               # Instructions; skipped by the indexer
│
├── configs/
│   └── portfolio-bot.yaml          # Provider + RAG + persona config
│
├── docs/
│   ├── architecture.md             # ← THIS FILE
│   └── architecture.es.md
│
├── bench/                          # Reproducible SQLite driver benchmark
│
├── go.mod                          # require rony-llm-agent, modernc.org/sqlite
└── README.md
```

---

## 📅 11. Roadmap

### Phase 1: MVP — done

- [x] Project setup (`go mod init`, structure)
- [x] HTTP server with `/api/chat` endpoint
- [x] Functional SSE streaming
- [x] RAG indexer (`data/projects/` + `data/docs/`, `.md` + `.mdx` → FTS5)
- [x] RAG retriever (query → top-k chunks)
- [x] Persona loader from YAML
- [x] llama.cpp integration — **qwen2.5-3b**, not the 1.5b originally planned
- [x] CLI: `serve`, `reindex`, `ask`
- [x] Basic tests

### Phase 2: Integration with Astro — done, differently

- [x] Drop-in vanilla-JS widget — **replaced** the planned React component and
      Astro proxy route. No build step, no framework lock-in, and it works in
      all three topologies in §5.1 rather than only behind a proxy
- [x] Widget styling — scoped CSS with custom properties, **not** TailwindCSS;
      a drop-in widget cannot assume the host site's toolchain
- [ ] Automated E2E test: Astro → chat-bot → response (still manual, §9.3)

### Phase 3: Polish — done

- [x] Rate limiting per IP
- [x] Structured logging (JSON)
- [x] Health checks for monitoring
- [x] systemd service files (§8.1)
- [x] README + deployment docs

### Phase 4: Optionals — mostly done

- [x] Multiple conversations (session ID)
- [x] Persisted chat history
- [x] Multi-language (EN/ES) — detection plus a full Spanish prompt, §4.4
- [x] Auto-compaction for long threads, §4.5
- [ ] Analysis of frequent questions
- [ ] More polished standalone CLI version (`chat-bot ask`)

### Phase 5: Answer quality — done

Everything here came out of measuring real answers rather than from the
original plan; each item exists because something was observably wrong.

- [x] Hybrid retrieval (BM25 ⊕ embeddings, RRF) — §4.2
- [x] Content-hash guard against stale vectors — §4.2
- [x] Project catalogue injected every turn, to stop invented project names
- [x] Reference documents separate from projects, so the CV is retrievable
      without being listed as a project — §4.1
- [x] Vendor sampling parameters wired through config to llama.cpp
- [x] `context_size` cut from 8192 to 4096 on measured usage — §4.5

### Known and unfixed

Recorded so they are not re-filed as new bugs:

- The model reads dates out of the CV correctly but does the arithmetic on
  them wrong — "Jul 2024 – Jun 2026" reported as three years.
- It occasionally attributes a fact to the wrong source file.
- *"¿Dónde ha trabajado Victor?"* answers with projects rather than employers.
  Phrasing-specific: *"¿En qué empresas ha trabajado?"* and *"¿Cuánto tiempo
  estuvo en Metrimex?"* both answer correctly.

---

## 📐 12. Quality Specifications

### 12.1 Performance metrics

| Metric | Target | Measured on the 2-core CPU-only target |
|---|---|---|
| Retrieval latency (top-5, hybrid) | <50ms | **~40ms** — 37ms of it is the query embedding round-trip; BM25 and the vector scan are sub-millisecond |
| Bot process memory | <150MB | **~20MB** |
| Total footprint (bot + LLM + embedder) | fits in 8GB with room to spare | **~4.6GB** (3.64 + 0.91 + 0.02), leaving ~3.4GB for the rest of the host |
| Generation throughput | — | **21.0 tok/s** steady state, ~11 tok/s on the first cold request |
| TTFT (Time-to-first-token) | <500ms | **Not met, and not reachable here.** Two threads have to prefill a ~1100-token prompt before the first token. The original target assumed a machine with a GPU |
| End-to-end (question → complete response) | <3s | **Not met: ~21s** for a typical answer. The bot is not the bottleneck; a 3B model on 2 cores is |

The last two rows are the honest cost of the hardware constraint, and the
widget is built around it: responses stream token by token, so the visitor
sees text moving within a couple of seconds rather than waiting 21s for a
block. Buying either target back means a smaller model, and the 20-question
benchmark in `configs/portfolio-bot.yaml` measures what that costs in
accuracy — gemma-3-1b averages 13.8s against qwen's 21.0s, and answers 6 fewer
of every 10 questions correctly.

### 12.2 Required tests

- Unit tests: coverage ≥70%
- Integration tests: with mock LLM + in-memory SQLite FTS5
- E2E: at least one complete Astro → chat-bot flow

---

## 🔒 13. Security

### 13.1 Implemented

- **Rate limiting** per IP (default 30 req/min)
- **Restrictive CORS** — only configured origins
- **Input validation** — JSON schema validation on requests
- **No PII storage** — we don't save conversations by default
- **Local-only by default** — no calls to cloud APIs

### 13.2 Deferred / Optional

- Auth with API key (for private use)
- Query logging for analytics
- IP anonymization in logs
- HTTPS via reverse proxy (Caddy/nginx)

---

## 📚 14. References

- **SSE Spec:** https://html.spec.whatwg.org/multipage/server-sent-events.html
- **Ollama API:** https://github.com/ollama/ollama/blob/main/docs/api.md
- **SQLite FTS5:** https://www.sqlite.org/fts5.html
- **Go SQLite driver:** https://github.com/mattn/go-sqlite3 (CGO) or https://modernc.org/sqlite (pure Go)
- **qwen2.5-3b-instruct:** https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
- **nomic-embed-text-v2-moe:** https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF
- **Reciprocal Rank Fusion:** Cormack, Clarke & Büttcher (2009), *Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods*
- **Astro API routes:** https://docs.astro.build/en/guides/endpoints/
- **rony-llm-agent:** https://github.com/VictorVargas/rony-llm-agent

---

**Document ready for implementation. 🚀**