victorvargas/rony-chat-bot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
rony-chat-bot/README.md
Victor Hugo Vargas 129809067b feat(rag): hybrid retrieval, reference documents, and vendor sampling 
Answers were short, sometimes in the wrong language, and occasionally about
projects that do not exist. Measured on a 20-question battery against the real
corpus in both Spanish and English, this takes grounded content from 3/10 to
9/10 and language matching from 7/10 to 10/10.

Retrieval
- Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion.
  Both halves are load-bearing: the corpus is English and visitors ask in
  Spanish, so the meaningful words score zero. "paga" appears 0 times in a
  document that says "Payments: Stripe" — the question "¿Con qué se paga en la
  tienda de ropa?" retrieved nothing at all. Embeddings put all three of that
  project's chunks on top. RRF ranks by agreement rather than comparing a BM25
  score against a cosine, quantities with no shared scale.
- internal/embed: OpenAI-compatible embeddings client, unit-normalised so a
  dot product is the cosine. Reorders by the response `index` field.
- Store a content hash beside each vector and skip rows where it no longer
  matches the chunk. Chunk ids survive body edits, so without this an edited
  document keeps serving embeddings that describe text that is gone —
  reproduced live by changing a payment provider and watching the old one keep
  coming back.
- Degrade to keyword-only when the embedder is down instead of failing.

Reference documents that are not projects
- Index `.mdx` alongside `.md`, and split sources into projects (announced in
  the catalogue) and reference material (retrievable, never listed). A CV is
  what someone deciding whether to hire actually reads, and it was unreachable
  while it lived only in the Astro site — but filing it under projects made
  the bot list "cv" as one of Victor's works.
- Skip each directory's README. `data/projects/README.md` was being indexed,
  so the catalogue injected into every prompt announced "README" and
  "README.es" as projects of Victor's.
- Exclude frontmatter from retrieval. It is dense metadata in a very short
  chunk, which makes it a magnet for short queries: a CV's `location:` field
  answered "¿Dónde ha trabajado Victor?" with a city instead of a work history.
- Split oversized sections at `###` before falling back to byte offsets. A CV's
  Experience section is a list of jobs, and size-splitting cut one mid-word,
  stranding the employer's name in the previous chunk.

Prompt and sampling
- Inject the full project catalogue every turn. Top-K search returns the best
  matching sections, so "list every project" cannot be answered from retrieval
  alone, and a small model asked to enumerate from partial hits invents the
  rest. ~10 tokens per project; this is what stopped the invented names.
- Wire the sampling parameters the model authors publish (top_k, top_p, min_p,
  repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them
  at llama.cpp's defaults produced 16-token stub answers.
- Localised system prompt selected by detected language. The English prompt
  plus "reply in the user's language" answered 1/5 Spanish questions in
  Spanish; few-shot examples fixed the language but got copied verbatim into
  real answers.
- Fold compaction's system notes into the leading system message. Gemma's chat
  template rejects a system message that is not first, and the whole request
  failed with HTTP 400 the moment compaction fired.

Configuration and docs
- context_size 4096, down from 8192. The largest prompt this bot ever built
  over 20 real requests was 1255 tokens, compaction starts at ~3070, and the
  cut saved 212 MB resident with zero truncations and identical throughput.
- Correct the RAM figures throughout. They were measured with a GPU absorbing
  llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which
  is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs
  still started gemma-3-1b while the config defaulted to qwen, and neither
  started the embedder at all.

Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder
+ 0.02 GB bot, 21.0 tok/s steady state.

Known and unfixed, so they are not re-filed as new bugs: the model reads dates
out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha
trabajado Victor?" still answers with projects rather than employers, though
"¿En qué empresas ha trabajado?" works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 15:45:36 -07:00

232 lines
No EOL
10 KiB
Markdown
Raw Blame History

# Rony Chat Bot — HTTP Portfolio Bot
> 🌐 **Language:** [English](./README.md) | [Español](./README.es.md)
>
> 🤖 **HTTP chatbot that presents your portfolio and answers questions about your projects.**
**Rony Chat Bot** is a chatbot based on [`rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) that integrates with an Astro/React site to answer questions about Victor Hugo Vargas and his projects, using **RAG over markdown files**.
## ✨ Features
- 🌐 **HTTP server** with SSE (Server-Sent Events) streaming
- 🧠 **Hybrid RAG over markdown/MDX** — SQLite FTS5 keyword search fused with multilingual embeddings (Reciprocal Rank Fusion)
- 🎭 **Customizable persona** — responds as "Victor's assistant"
- ⚡ **Self-hosted** with llama.cpp (default) or Ollama (no cloud API key required)
- 💬 **Drop-in chat widget** — vanilla JS, no build step, works in any site
- 🛡️ **Rate limiting** and structured logging
- 📦 **Portable** — adaptable to other contexts (clients, products, etc.)
## 🚀 Quick start
```bash
# 1. Install
git clone https://github.com/VictorVargas/rony-chat-bot.git
cd rony-chat-bot
# 2. Resolve dependencies (creates go.sum with hashes)
go mod tidy
# 3. Download the models: an instruct LLM and a multilingual embedder
# https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF (~2 GB)
# https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF (~370 MB)
export RONY_MODELS_PATH=/path/to/models
# 4. Load your projects in data/projects/
echo "# My Cool Project\nDescription..." > data/projects/my-project.md
# 5. Build
go build -o bin/chat-bot ./cmd/chat-bot
# 6. Start the LLM (CPU, 2 cores target — adjust --threads to your host)
llama-server \
-m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
--port 9100 --ctx-size 4096 --parallel 1 \
--device none --threads 2 --mlock \
--temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
# 7. Start the embedder (second terminal)
llama-server \
-m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
--port 9200 --embedding --pooling mean \
--ctx-size 2048 --parallel 1 --device none --threads 2
# 8. Build the index (needs the embedder running), then serve
./bin/chat-bot reindex
./bin/chat-bot serve
# → Serves on http://localhost:7331
```
**Minimum hardware:** 2 CPU cores, 8 GB RAM, no GPU. Measured resident memory
on CPU with the setup above: **3.64 GB** for the LLM, **0.91 GB** for the
embedder, **0.02 GB** for the bot — about **4.6 GB**, leaving ~3.4 GB for the
rest of the host. Generation runs at 21 tok/s on 2 threads once the system
prompt is warm in llama-server's prompt cache.
Three flags are easy to get wrong and each one costs you real quality:
- **`--device none`** — llama.cpp brings up a compiled-in GPU backend even with
`-ngl 0`, and on a host with no GPU those buffers come out of system RAM.
Measured on qwen2.5-3b: 2.54 GB with a GPU absorbing them, 3.66 GB without.
Budget from the second number, and pass this flag so the measurement matches
what production actually does.
- **`--parallel 1`** — `--ctx-size` is divided across slots and llama-server opens
4 by default, so `--ctx-size 4096` without this gives each request only 1024
tokens. The bot's rate limiter already caps concurrency.
- **`--temp` / `--top-k` / `--top-p`** — use the values the model's authors
publish, not llama.cpp's defaults. The ones above are Qwen's for instruct
chat. Getting this wrong is not subtle: gemma-3-1b at `temperature 0.7` with
the rest unset produced 16-token stub answers.
**`--pooling mean` is mandatory on the embedder.** Without it the endpoint does
not return one vector per input and the client rejects the response.
Keep `context_size` in `configs/portfolio-bot.yaml` equal to `--ctx-size`; the
bot sizes its RAG and compaction budgets from that number and does not ask the
server what it actually has. Set them apart and the bot will build prompts the
server rejects.
**Why 4096 is enough.** Measured over 20 real requests, the largest prompt this
bot ever built was **1255 tokens** — system prompt, project catalogue, five
retrieved chunks and the question. Compaction only begins at 75% of the window
(~3070 tokens), so there is 2.4x headroom before it even starts. Halving the
window from 8192 saved **212 MB** of resident memory with zero truncations and
identical throughput (21.0 tok/s either way): the extra context was reserved
and never used.
## 📚 Projects vs. reference documents
The index has two kinds of source, both accepting `.md` and `.mdx`:
```yaml
rag:
data_path: ./data/projects # projects → listed in the catalogue
docs_path: ./data/docs # reference material → retrievable, never listed
```
Everything in `data_path` is one of your projects and is announced in the
project catalogue the bot injects into every prompt. Everything in `docs_path`
is searchable evidence that is *not* a project — your CV, an about page, a FAQ.
Your CV belongs in `docs_path`. It is the document that answers what someone
considering hiring you actually asks ("does he know Kubernetes?", "where has he
worked?"), and none of it is retrievable while it lives only in your site.
Symlink it so there's a single copy to maintain:
```bash
mkdir -p data/docs
ln -s ../../../portfolio/src/content/cv/cv.mdx data/docs/cv.mdx
./bin/chat-bot reindex
```
Without the distinction the CV has to go in `data_path` to be searchable, and
the bot then cheerfully lists "cv" as one of your projects.
Upgrading an existing install needs no migration step: the chunk table is
derived data, so the store rebuilds it on open and the next `reindex`
repopulates it.
## 🔍 Retrieval: keyword + embeddings
Retrieval is hybrid, and both halves are load-bearing.
**Keyword search (SQLite FTS5)** matches words exactly — no stemming, no
translation. That is precise for rare proper nouns and useless across
languages. The corpus is written in English and visitors ask in Spanish, so
the words carrying the meaning score zero: measured on the real corpus,
`"paga"` appears 0 times in a document that says *"Payments: Stripe"* and
`"trabajado"` 0 times in one that says *"worked"*. The question *"¿Con qué se
paga en la tienda de ropa?"* retrieved **nothing at all**.
**Embeddings** (`nomic-embed-v2-moe`, multilingual) close that gap: the same
question puts all three `tienda-ropa` chunks on top. They are fuzzier than
BM25 on an exact rare token, which is why both are kept and fused with
Reciprocal Rank Fusion — RRF ranks by agreement between the two orderings,
which avoids comparing a BM25 score against a cosine, quantities that share
no scale.
Enable it in `configs/portfolio-bot.yaml` under `embeddings:` and re-run
`reindex` — vectors are built at index time. If the endpoint is down or
disabled, retrieval degrades to keyword-only instead of failing.
Two things that cost real accuracy and are easy to miss:
- **Frontmatter is excluded from retrieval.** It is dense metadata (title,
tags, repo, location) in a very short chunk, which makes it a magnet for
short queries. A CV's `location:` field made *"¿Dónde ha trabajado
Victor?"* retrieve the frontmatter instead of the work history, because
"where" matches a location.
- **Long sections split at `###` headings, not byte offsets.** A CV's
Experience section is a list of jobs; size-splitting cut one entry
mid-word into a chunk beginning *"... id app for an on-demand ride-sharing
service"*, with the employer name stranded in the previous piece. Chunks
now keep one job each, named `Experience — Metrimex — Frontend Developer`.
## 📁 Structure
```
rony-chat-bot/
├── cmd/chat-bot/ # Entry point (CLI)
├── internal/
│ ├── server/ # HTTP handlers + SSE
│ ├── agent/ # LLM client + RAG + persona runner
│ ├── portfolio/ # Data loader (markdown → RAG)
│ ├── persona/ # Persona override
│ ├── streaming/ # SSE helpers
│ └── i18n/ # Language detection (EN/ES)
├── web/ # ← DROP-IN CHAT WIDGET
│ ├── chat-widget.js
│ ├── chat-widget.css
│ └── example.html
├── data/projects/ # ← YOUR PROJECTS IN MARKDOWN
│ ├── rony-harness.md
│ ├── rony-llm-agent.md
│ └── ...
├── configs/
│ └── portfolio-bot.yaml # Provider + RAG + persona config
├── docs/
│ └── architecture.md # ← Complete technical specification
└── go.mod # require rony-llm-agent
```
## 🎯 Embed in any site
The bot ships with a drop-in chat widget. Add two files and a `<script>` tag:
```html
<link rel="stylesheet" href="/chat-widget.css">
<script src="/chat-widget.js"
data-api-url="https://chat.example.com"
data-title="Ask me anything"
data-position="bottom-right"
data-theme="auto"
defer></script>
```
See [`web/README.md`](./web/README.md) for the full configuration reference and Astro/Next.js integration snippets. Full architecture in [`docs/architecture.md`](./docs/architecture.md) §5.
## 🔄 Adapt to another client
This bot is designed to be **atomic** and reusable. To adapt it (e.g., chatbot for a car dealership):
1. Fork/clone this repo
2. Replace `data/projects/` with `data/inventory/` (or another domain)
3. Update `configs/portfolio-bot.yaml` with the new persona
4. Deploy
The `rony-llm-agent` library doesn't change.
## 📚 Documentation
- [**Architecture doc**](./docs/architecture.md) — Complete technical specification
- [Library: `rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) — Reusable core
- [Rony Harness](https://github.com/VictorVargas/rony-harness) — The other project using the same library
## 📄 License
MIT — see [`LICENSE`](./LICENSE).
## 🔗 Workspace projects
- [`rony-llm-agent`](https://github.com/VictorVargas/rony-llm-agent) — Core library
- [`rony-harness`](https://github.com/VictorVargas/rony-harness) — AI agent harness (TUI)
- [`portfolio`](https://github.com/VictorVargas/portfolio) — Astro + React site (integrates this bot)
Reference in a new issue View git blame Copy permalink