rony-chat-bot/README.md

# Rony Chat Bot — HTTP Portfolio Bot

> 🌐 **Language:** [English](./README.md) | [Español](./README.es.md)
>
> 🤖 **HTTP chatbot that presents your portfolio and answers questions about your projects.**

**Rony Chat Bot** is a chatbot based on [`rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) that integrates with an Astro/React site to answer questions about Victor Hugo Vargas and his projects, using **RAG over markdown files**.

## ✨ Features

- 🌐 **HTTP server** with SSE (Server-Sent Events) streaming
- 🧠 **Hybrid RAG over markdown/MDX** — SQLite FTS5 keyword search fused with multilingual embeddings (Reciprocal Rank Fusion)
- 🎭 **Customizable persona** — responds as "Victor's assistant"
- ⚡ **Self-hosted** with llama.cpp (default) or Ollama (no cloud API key required)
- 💬 **Drop-in chat widget** — vanilla JS, no build step, works in any site
- 🛡️ **Rate limiting** and structured logging
- 📦 **Portable** — adaptable to other contexts (clients, products, etc.)

## 🚀 Quick start

```bash
# 1. Install
git clone https://github.com/VictorVargas/rony-chat-bot.git
cd rony-chat-bot

# 2. Resolve dependencies (creates go.sum with hashes)
go mod tidy

# 3. Download the models: an instruct LLM and a multilingual embedder
#    https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF        (~2 GB)
#    https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF (~370 MB)
export RONY_MODELS_PATH=/path/to/models

# 4. Load your projects in data/projects/
echo "# My Cool Project\nDescription..." > data/projects/my-project.md

# 5. Build
go build -o bin/chat-bot ./cmd/chat-bot

# 6. Start the LLM (CPU, 2 cores target — adjust --threads to your host)
llama-server \
  -m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05

# 7. Start the embedder (second terminal)
llama-server \
  -m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2

# 8. Build the index (needs the embedder running), then serve
./bin/chat-bot reindex
./bin/chat-bot serve
# → Serves on http://localhost:7331
```

**Minimum hardware:** 2 CPU cores, 8 GB RAM, no GPU. Measured resident memory
on CPU with the setup above: **3.64 GB** for the LLM, **0.91 GB** for the
embedder, **0.02 GB** for the bot — about **4.6 GB**, leaving ~3.4 GB for the
rest of the host. Generation runs at 21 tok/s on 2 threads once the system
prompt is warm in llama-server's prompt cache.

Three flags are easy to get wrong and each one costs you real quality:

- **`--device none`** — llama.cpp brings up a compiled-in GPU backend even with
  `-ngl 0`, and on a host with no GPU those buffers come out of system RAM.
  Measured on qwen2.5-3b: 2.54 GB with a GPU absorbing them, 3.66 GB without.
  Budget from the second number, and pass this flag so the measurement matches
  what production actually does.
- **`--parallel 1`** — `--ctx-size` is divided across slots and llama-server opens
  4 by default, so `--ctx-size 4096` without this gives each request only 1024
  tokens. The bot's rate limiter already caps concurrency.
- **`--temp` / `--top-k` / `--top-p`** — use the values the model's authors
  publish, not llama.cpp's defaults. The ones above are Qwen's for instruct
  chat. Getting this wrong is not subtle: gemma-3-1b at `temperature 0.7` with
  the rest unset produced 16-token stub answers.

**`--pooling mean` is mandatory on the embedder.** Without it the endpoint does
not return one vector per input and the client rejects the response.

Keep `context_size` in `configs/portfolio-bot.yaml` equal to `--ctx-size`; the
bot sizes its RAG and compaction budgets from that number and does not ask the
server what it actually has. Set them apart and the bot will build prompts the
server rejects.

**Why 4096 is enough.** Measured over 20 real requests, the largest prompt this
bot ever built was **1255 tokens** — system prompt, project catalogue, five
retrieved chunks and the question. Compaction only begins at 75% of the window
(~3070 tokens), so there is 2.4x headroom before it even starts. Halving the
window from 8192 saved **212 MB** of resident memory with zero truncations and
identical throughput (21.0 tok/s either way): the extra context was reserved
and never used.

## 📚 Projects vs. reference documents

The index has two kinds of source, both accepting `.md` and `.mdx`:

```yaml
rag:
  data_path: ./data/projects   # projects → listed in the catalogue
  docs_path: ./data/docs       # reference material → retrievable, never listed
```

Everything in `data_path` is one of your projects and is announced in the
project catalogue the bot injects into every prompt. Everything in `docs_path`
is searchable evidence that is *not* a project — your CV, an about page, a FAQ.

Your CV belongs in `docs_path`. It is the document that answers what someone
considering hiring you actually asks ("does he know Kubernetes?", "where has he
worked?"), and none of it is retrievable while it lives only in your site.
Symlink it so there's a single copy to maintain:

```bash
mkdir -p data/docs
ln -s ../../../portfolio/src/content/cv/cv.mdx data/docs/cv.mdx
./bin/chat-bot reindex
```

Without the distinction the CV has to go in `data_path` to be searchable, and
the bot then cheerfully lists "cv" as one of your projects.

Upgrading an existing install needs no migration step: the chunk table is
derived data, so the store rebuilds it on open and the next `reindex`
repopulates it.

## 🔍 Retrieval: keyword + embeddings

Retrieval is hybrid, and both halves are load-bearing.

**Keyword search (SQLite FTS5)** matches words exactly — no stemming, no
translation. That is precise for rare proper nouns and useless across
languages. The corpus is written in English and visitors ask in Spanish, so
the words carrying the meaning score zero: measured on the real corpus,
`"paga"` appears 0 times in a document that says *"Payments: Stripe"* and
`"trabajado"` 0 times in one that says *"worked"*. The question *"¿Con qué se
paga en la tienda de ropa?"* retrieved **nothing at all**.

**Embeddings** (`nomic-embed-v2-moe`, multilingual) close that gap: the same
question puts all three `tienda-ropa` chunks on top. They are fuzzier than
BM25 on an exact rare token, which is why both are kept and fused with
Reciprocal Rank Fusion — RRF ranks by agreement between the two orderings,
which avoids comparing a BM25 score against a cosine, quantities that share
no scale.

Enable it in `configs/portfolio-bot.yaml` under `embeddings:` and re-run
`reindex` — vectors are built at index time. If the endpoint is down or
disabled, retrieval degrades to keyword-only instead of failing.

Two things that cost real accuracy and are easy to miss:

- **Frontmatter is excluded from retrieval.** It is dense metadata (title,
  tags, repo, location) in a very short chunk, which makes it a magnet for
  short queries. A CV's `location:` field made *"¿Dónde ha trabajado
  Victor?"* retrieve the frontmatter instead of the work history, because
  "where" matches a location.
- **Long sections split at `###` headings, not byte offsets.** A CV's
  Experience section is a list of jobs; size-splitting cut one entry
  mid-word into a chunk beginning *"... id app for an on-demand ride-sharing
  service"*, with the employer name stranded in the previous piece. Chunks
  now keep one job each, named `Experience — Metrimex — Frontend Developer`.

## 📁 Structure

```
rony-chat-bot/
├── cmd/chat-bot/             # Entry point (CLI)
├── internal/
│   ├── server/               # HTTP handlers + SSE
│   ├── agent/                # LLM client + RAG + persona runner
│   ├── portfolio/            # Data loader (markdown → RAG)
│   ├── persona/              # Persona override
│   ├── streaming/            # SSE helpers
│   └── i18n/                 # Language detection (EN/ES)
├── web/                      # ← DROP-IN CHAT WIDGET
│   ├── chat-widget.js
│   ├── chat-widget.css
│   └── example.html
├── data/projects/            # ← YOUR PROJECTS IN MARKDOWN
│   ├── rony-harness.md
│   ├── rony-llm-agent.md
│   └── ...
├── configs/
│   └── portfolio-bot.yaml    # Provider + RAG + persona config
├── docs/
│   └── architecture.md        # ← Complete technical specification
└── go.mod                     # require rony-llm-agent
```

## 🎯 Embed in any site

The bot ships with a drop-in chat widget. Add two files and a `<script>` tag:

```html
<link rel="stylesheet" href="/chat-widget.css">
<script src="/chat-widget.js"
        data-api-url="https://chat.example.com"
        data-title="Ask me anything"
        data-position="bottom-right"
        data-theme="auto"
        defer></script>
```

See [`web/README.md`](./web/README.md) for the full configuration reference and Astro/Next.js integration snippets. Full architecture in [`docs/architecture.md`](./docs/architecture.md) §5.

## 🔄 Adapt to another client

This bot is designed to be **atomic** and reusable. To adapt it (e.g., chatbot for a car dealership):

1. Fork/clone this repo
2. Replace `data/projects/` with `data/inventory/` (or another domain)
3. Update `configs/portfolio-bot.yaml` with the new persona
4. Deploy

The `rony-llm-agent` library doesn't change.

## 📚 Documentation

- [**Architecture doc**](./docs/architecture.md) — Complete technical specification
- [Library: `rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) — Reusable core
- [Rony Harness](https://github.com/VictorVargas/rony-harness) — The other project using the same library

## 📄 License

MIT — see [`LICENSE`](./LICENSE).

## 🔗 Workspace projects

- [`rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) — Core library
- [`rony-harness`](https://github.com/VictorVargas/rony-harness) — AI agent harness (TUI)
- [`portfolio`](https://github.com/VictorVargas/portfolio) — Astro + React site (integrates this bot)