Rony Chat Bot — HTTP Portfolio Bot

🌐 Language: English | Español

🤖 HTTP chatbot that presents your portfolio and answers questions about your projects.

Rony Chat Bot is a chatbot based on rony-llm-agent that integrates with an Astro/React site to answer questions about Victor Hugo Vargas and his projects, using RAG over markdown files.

✨ Features

• 🌐 HTTP server with SSE (Server-Sent Events) streaming
• 🧠 Hybrid RAG over markdown/MDX — SQLite FTS5 keyword search fused with multilingual embeddings (Reciprocal Rank Fusion)
• 🎭 Customizable persona — responds as "Victor's assistant"
• ⚡ Self-hosted with llama.cpp (default) or Ollama (no cloud API key required)
• 💬 Drop-in chat widget — vanilla JS, no build step, works in any site
• 🛡️ Rate limiting and structured logging
• 📦 Portable — adaptable to other contexts (clients, products, etc.)

🚀 Quick start

# 1. Install
git clone https://github.com/VictorVargas/rony-chat-bot.git
cd rony-chat-bot

# 2. Resolve dependencies (creates go.sum with hashes)
go mod tidy

# 3. Download the models: an instruct LLM and a multilingual embedder
#    https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF        (~2 GB)
#    https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF (~370 MB)
export RONY_MODELS_PATH=/path/to/models

# 4. Load your projects in data/projects/
echo "# My Cool Project\nDescription..." > data/projects/my-project.md

# 5. Build
go build -o bin/chat-bot ./cmd/chat-bot

# 6. Start the LLM (CPU, 2 cores target — adjust --threads to your host)
llama-server \
  -m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05

# 7. Start the embedder (second terminal)
llama-server \
  -m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2

# 8. Build the index (needs the embedder running), then serve
./bin/chat-bot reindex
./bin/chat-bot serve
# → Serves on http://localhost:7331

Minimum hardware: 2 CPU cores, 8 GB RAM, no GPU. Measured resident memory on CPU with the setup above: 3.64 GB for the LLM, 0.91 GB for the embedder, 0.02 GB for the bot — about 4.6 GB, leaving ~3.4 GB for the rest of the host. Generation runs at 21 tok/s on 2 threads once the system prompt is warm in llama-server's prompt cache.

Three flags are easy to get wrong and each one costs you real quality:

• --device none — llama.cpp brings up a compiled-in GPU backend even with -ngl 0, and on a host with no GPU those buffers come out of system RAM. Measured on qwen2.5-3b: 2.54 GB with a GPU absorbing them, 3.66 GB without. Budget from the second number, and pass this flag so the measurement matches what production actually does.
• --parallel 1 — --ctx-size is divided across slots and llama-server opens 4 by default, so --ctx-size 4096 without this gives each request only 1024 tokens. The bot's rate limiter already caps concurrency.
• --temp / --top-k / --top-p — use the values the model's authors publish, not llama.cpp's defaults. The ones above are Qwen's for instruct chat. Getting this wrong is not subtle: gemma-3-1b at temperature 0.7 with the rest unset produced 16-token stub answers.

--pooling mean is mandatory on the embedder. Without it the endpoint does not return one vector per input and the client rejects the response.

Keep context_size in configs/portfolio-bot.yaml equal to --ctx-size; the bot sizes its RAG and compaction budgets from that number and does not ask the server what it actually has. Set them apart and the bot will build prompts the server rejects.

Why 4096 is enough. Measured over 20 real requests, the largest prompt this bot ever built was 1255 tokens — system prompt, project catalogue, five retrieved chunks and the question. Compaction only begins at 75% of the window (~3070 tokens), so there is 2.4x headroom before it even starts. Halving the window from 8192 saved 212 MB of resident memory with zero truncations and identical throughput (21.0 tok/s either way): the extra context was reserved and never used.

📚 Projects vs. reference documents

The index has two kinds of source, both accepting .md and .mdx:

rag:
  data_path: ./data/projects   # projects → listed in the catalogue
  docs_path: ./data/docs       # reference material → retrievable, never listed

Everything in data_path is one of your projects and is announced in the project catalogue the bot injects into every prompt. Everything in docs_path is searchable evidence that is not a project — your CV, an about page, a FAQ.

Your CV belongs in docs_path. It is the document that answers what someone considering hiring you actually asks ("does he know Kubernetes?", "where has he worked?"), and none of it is retrievable while it lives only in your site. Symlink it so there's a single copy to maintain:

mkdir -p data/docs
ln -s ../../../portfolio/src/content/cv/cv.mdx data/docs/cv.mdx
./bin/chat-bot reindex

Without the distinction the CV has to go in data_path to be searchable, and the bot then cheerfully lists "cv" as one of your projects.

Upgrading an existing install needs no migration step: the chunk table is derived data, so the store rebuilds it on open and the next reindex repopulates it.

🔍 Retrieval: keyword + embeddings

Retrieval is hybrid, and both halves are load-bearing.

Keyword search (SQLite FTS5) matches words exactly — no stemming, no translation. That is precise for rare proper nouns and useless across languages. The corpus is written in English and visitors ask in Spanish, so the words carrying the meaning score zero: measured on the real corpus, "paga" appears 0 times in a document that says "Payments: Stripe" and "trabajado" 0 times in one that says "worked". The question "¿Con qué se paga en la tienda de ropa?" retrieved nothing at all.

Embeddings (nomic-embed-v2-moe, multilingual) close that gap: the same question puts all three tienda-ropa chunks on top. They are fuzzier than BM25 on an exact rare token, which is why both are kept and fused with Reciprocal Rank Fusion — RRF ranks by agreement between the two orderings, which avoids comparing a BM25 score against a cosine, quantities that share no scale.

Enable it in configs/portfolio-bot.yaml under embeddings: and re-run reindex — vectors are built at index time. If the endpoint is down or disabled, retrieval degrades to keyword-only instead of failing.

Two things that cost real accuracy and are easy to miss:

• Frontmatter is excluded from retrieval. It is dense metadata (title, tags, repo, location) in a very short chunk, which makes it a magnet for short queries. A CV's location: field made "¿Dónde ha trabajado Victor?" retrieve the frontmatter instead of the work history, because "where" matches a location.
• Long sections split at ### headings, not byte offsets. A CV's Experience section is a list of jobs; size-splitting cut one entry mid-word into a chunk beginning "... id app for an on-demand ride-sharing service", with the employer name stranded in the previous piece. Chunks now keep one job each, named Experience — Metrimex — Frontend Developer.

📁 Structure

rony-chat-bot/
├── cmd/chat-bot/             # Entry point (CLI)
├── internal/
│   ├── server/               # HTTP handlers + SSE
│   ├── agent/                # LLM client + RAG + persona runner
│   ├── portfolio/            # Data loader (markdown → RAG)
│   ├── persona/              # Persona override
│   ├── streaming/            # SSE helpers
│   └── i18n/                 # Language detection (EN/ES)
├── web/                      # ← DROP-IN CHAT WIDGET
│   ├── chat-widget.js
│   ├── chat-widget.css
│   └── example.html
├── data/projects/            # ← YOUR PROJECTS IN MARKDOWN
│   ├── rony-harness.md
│   ├── rony-llm-agent.md
│   └── ...
├── configs/
│   └── portfolio-bot.yaml    # Provider + RAG + persona config
├── docs/
│   └── architecture.md        # ← Complete technical specification
└── go.mod                     # require rony-llm-agent

🎯 Embed in any site

The bot ships with a drop-in chat widget. Add two files and a <script> tag:

<link rel="stylesheet" href="/chat-widget.css">
<script src="/chat-widget.js"
        data-api-url="https://chat.example.com"
        data-title="Ask me anything"
        data-position="bottom-right"
        data-theme="auto"
        defer></script>

See web/README.md for the full configuration reference and Astro/Next.js integration snippets. Full architecture in docs/architecture.md §5.

🔄 Adapt to another client

This bot is designed to be atomic and reusable. To adapt it (e.g., chatbot for a car dealership):

1. Fork/clone this repo
2. Replace data/projects/ with data/inventory/ (or another domain)
3. Update configs/portfolio-bot.yaml with the new persona
4. Deploy

The rony-llm-agent library doesn't change.

📚 Documentation

• Architecture doc — Complete technical specification
• Library: rony-llm-agent — Reusable core
• Rony Harness — The other project using the same library

📄 License

MIT — see LICENSE.

🔗 Workspace projects

• rony-llm-agent — Core library
• rony-harness — AI agent harness (TUI)
• portfolio — Astro + React site (integrates this bot)