Rony Chat Bot — HTTP Portfolio Bot

🌐 Language: English | Español

🤖 HTTP chatbot that presents your portfolio and answers questions about your projects.

Rony Chat Bot is a chatbot based on rony-llm-agent that integrates with an Astro/React site to answer questions about Victor Hugo Vargas and his projects, using RAG over markdown files.

✨ Features

• 🌐 HTTP server with SSE (Server-Sent Events) streaming
• 🧠 Hybrid RAG over markdown/MDX — SQLite FTS5 keyword search fused with multilingual embeddings (Reciprocal Rank Fusion)
• 🎭 Customizable persona — responds as "Victor's assistant"
• ⚡ Self-hosted with llama.cpp (default) or Ollama (no cloud API key required)
• 💬 Drop-in chat widget — vanilla JS, no build step, works in any site
• 🛡️ Rate limiting and structured logging
• 📦 Portable — adaptable to other contexts (clients, products, etc.)

🚀 Quick start

# 1. Install
git clone https://github.com/VictorVargas/rony-chat-bot.git
cd rony-chat-bot

# 2. Resolve dependencies (creates go.sum with hashes)
go mod tidy

# 3. Download the models: an instruct LLM and a multilingual embedder
#    https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF        (~2 GB)
#    https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF (~370 MB)
export RONY_MODELS_PATH=/path/to/models

# 4. Configure content sources in configs/portfolio-bot.yaml
#    Each entry is an HTTP endpoint returning a JSON array of projects or docs.
#    See "Content sources" below for the contract.

# 5. Build
go build -o bin/chat-bot ./cmd/chat-bot

# 6. Start the LLM (CPU, 2 cores target — adjust --threads to your host)
llama-server \
  -m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
  --port 9100 --ctx-size 4096 --parallel 1 \
  --device none --threads 2 --mlock \
  --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05

# 7. Start the embedder (second terminal)
llama-server \
  -m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
  --port 9200 --embedding --pooling mean \
  --ctx-size 2048 --parallel 1 --device none --threads 2

# 8. Build the index (needs the embedder running), then serve
./bin/chat-bot reindex
./bin/chat-bot serve
# → Serves on http://localhost:7331

🌐 CORS origins

The allowed browser origins are supplied through CORS_ORIGINS; the config expands that variable at startup and defaults to an empty list. Use a JSON array (JSON is valid YAML), with origins containing only the scheme, host and optional port — no path or trailing slash.

The local Compose file supplies the development origins automatically:

CORS_ORIGINS: '["http://localhost:4321","http://localhost:8000"]'

For Coolify, set CORS_ORIGINS in the service's runtime environment instead:

CORS_ORIGINS=["https://your-portfolio.example","https://ronybot.example"]

Use only the origin that actually hosts the widget. Do not use *; CORS is a browser access policy, not authentication, so private deployments still need authentication or a proxy boundary if the API must not be public.

Minimum hardware: 2 CPU cores, 8 GB RAM, no GPU. Measured resident memory on CPU with the setup above: 3.64 GB for the LLM, 0.91 GB for the embedder, 0.02 GB for the bot — about 4.6 GB, leaving ~3.4 GB for the rest of the host. Generation runs at 21 tok/s on 2 threads once the system prompt is warm in llama-server's prompt cache.

Three flags are easy to get wrong and each one costs you real quality:

• --device none — llama.cpp brings up a compiled-in GPU backend even with -ngl 0, and on a host with no GPU those buffers come out of system RAM. Measured on qwen2.5-3b: 2.54 GB with a GPU absorbing them, 3.66 GB without. Budget from the second number, and pass this flag so the measurement matches what production actually does.
• --parallel 1 — --ctx-size is divided across slots and llama-server opens 4 by default, so --ctx-size 4096 without this gives each request only 1024 tokens. The bot's rate limiter already caps concurrency.
• --temp / --top-k / --top-p — use the values the model's authors publish, not llama.cpp's defaults. The ones above are Qwen's for instruct chat. Getting this wrong is not subtle: gemma-3-1b at temperature 0.7 with the rest unset produced 16-token stub answers.

--pooling mean is mandatory on the embedder. Without it the endpoint does not return one vector per input and the client rejects the response.

Keep context_size in configs/portfolio-bot.yaml equal to --ctx-size; the bot sizes its RAG and compaction budgets from that number and does not ask the server what it actually has. Set them apart and the bot will build prompts the server rejects.

Why 4096 is enough. Measured over 20 real requests, the largest prompt this bot ever built was 1255 tokens — system prompt, project catalogue, five retrieved chunks and the question. Compaction only begins at 75% of the window (~3070 tokens), so there is 2.4x headroom before it even starts. Halving the window from 8192 saved 212 MB of resident memory with zero truncations and identical throughput (21.0 tok/s either way): the extra context was reserved and never used.

📚 Content sources

The bot does not own its content. Each source is an HTTP endpoint that returns a JSON array; the bot fetches it on every reindex. Chunks are namespaced by name: so two sources cannot collide on project_id, and a failed reindex in one source does not touch the others' corpus.

sources:
  - name: portfolio
    kind: project         # → listed in the catalogue
    endpoint: https://sersofts.org/projects.json
  - name: portfolio-docs
    kind: doc             # → retrievable, never listed
    endpoint: https://sersofts.org/docs.json

kind: project items show up in the project catalogue the bot injects into every prompt. kind: doc items are searchable evidence that is not a project — the CV, an about page, a FAQ. Without this distinction the CV would land in the catalogue and the bot would announce "cv" as one of your works.

The contract each endpoint must honour:

// GET {endpoint}
[
  {
    "slug": "tienda-ropa",          // kebab-case, unique within the source
    "title": "Tienda de Ropa",      // displayed in the catalogue
    "stack": ["Next.js", "Stripe"], // optional
    "status": "live",               // optional: live | archived | wip
    "demoUrl": "https://...",       // optional
    "repoUrl": "https://...",       // optional
    "description": "# Tienda de Ropa\n\n## Purpose\n\n..."
  }
]

description is the markdown body the chunker will split — the H1 (or title: line) is what the catalogue entry shows, and the H2s become the retrieval chunks. Endpoints can come from anywhere: this repo's portfolio emits /projects.json and /docs.json; a future refaccionaria could publish the same shape at /api/projects.json and be added to the list without code changes.

For local development, point PORTFOLIO_URL at a running Astro dev server:

PORTFOLIO_URL=http://localhost:4321/projects.json \
PORTFOLIO_DOCS_URL=http://localhost:4321/docs.json \
  ./bin/chat-bot reindex

Upgrading an existing install needs no migration step: the chunk table is derived data, so the store rebuilds it on open and the next reindex repopulates it.

🔍 Retrieval: keyword + embeddings

Retrieval is hybrid, and both halves are load-bearing.

Keyword search (SQLite FTS5) matches words exactly — no stemming, no translation. That is precise for rare proper nouns and useless across languages. The corpus is written in English and visitors ask in Spanish, so the words carrying the meaning score zero: measured on the real corpus, "paga" appears 0 times in a document that says "Payments: Stripe" and "trabajado" 0 times in one that says "worked". The question "¿Con qué se paga en la tienda de ropa?" retrieved nothing at all.

Embeddings (nomic-embed-v2-moe, multilingual) close that gap: the same question puts all three tienda-ropa chunks on top. They are fuzzier than BM25 on an exact rare token, which is why both are kept and fused with Reciprocal Rank Fusion — RRF ranks by agreement between the two orderings, which avoids comparing a BM25 score against a cosine, quantities that share no scale.

Enable it in configs/portfolio-bot.yaml under embeddings: and re-run reindex — vectors are built at index time. If the endpoint is down or disabled, retrieval degrades to keyword-only instead of failing.

Two things that cost real accuracy and are easy to miss:

• Frontmatter is excluded from retrieval. It is dense metadata (title, tags, repo, location) in a very short chunk, which makes it a magnet for short queries. A CV's location: field made "¿Dónde ha trabajado Victor?" retrieve the frontmatter instead of the work history, because "where" matches a location.
• Long sections split at ### headings, not byte offsets. A CV's Experience section is a list of jobs; size-splitting cut one entry mid-word into a chunk beginning "... id app for an on-demand ride-sharing service", with the employer name stranded in the previous piece. Chunks now keep one job each, named Experience — Metrimex — Frontend Developer.

📁 Structure

rony-chat-bot/
├── cmd/chat-bot/             # Entry point (CLI)
├── internal/
│   ├── server/               # HTTP handlers + SSE
│   ├── agent/                # LLM client + RAG + persona runner
│   ├── portfolio/            # Data loader (markdown → RAG)
│   ├── persona/              # Persona override
│   ├── streaming/            # SSE helpers
│   └── i18n/                 # Language detection (EN/ES)
├── web/                      # ← DROP-IN CHAT WIDGET
│   ├── chat-widget.js
│   ├── chat-widget.css
│   └── example.html
├── configs/
│   └── portfolio-bot.yaml    # Provider + sources + persona + RAG config
├── configs/
│   └── portfolio-bot.yaml    # Provider + RAG + persona config
├── docs/
│   └── architecture.md        # ← Complete technical specification
└── go.mod                     # require rony-llm-agent

🎯 Embed in any site

The bot ships with a drop-in chat widget. Add two files and a <script> tag:

<link rel="stylesheet" href="/chat-widget.css">
<script src="/chat-widget.js"
        data-api-url="https://chat.example.com"
        data-title="Ask me anything"
        data-position="bottom-right"
        data-theme="auto"
        defer></script>

See web/README.md for the full configuration reference and Astro/Next.js integration snippets. Full architecture in docs/architecture.md §5.

🔄 Adapt to another client

This bot is designed to be atomic and reusable. To adapt it (e.g., chatbot for a car dealership):

1. Fork/clone this repo
2. Edit configs/portfolio-bot.yaml — set sources: to point at the new site's JSON endpoint(s), and adjust the persona
3. Deploy

The rony-llm-agent library doesn't change.

🚢 Deploy (v0.1.5-alpha)

Three containers (LLM, embedder, bot) wired together. The bot image is fully self-contained: the YAML config and the pre-built RAG SQLite database are baked into the image at build time, so deployment does not require seeding configs/ or data/ volumes or running an in-place reindex. The only persistent named volume is models (the two GGUF files).

Designed to run under Coolify, but works in plain Docker too.

Option A — Coolify (recommended)

Full step-by-step: deploy/coolify/README.md.

The short version:

1. Coolify → + New → Service Stack → point at this repo and use docker-compose.coolify.yml (not the plain one — the Coolify variant pins the bot image tag to the registry release instead of building from source).
2. Expose the bot service on https://ronybot.sersofts.org (Coolify handles TLS via Traefik).
3. Seed the models volume with the two GGUF files. The fastest path is to download them directly on the Coolify host into the volume's mountpoint (full instructions in the linked README). The ./scripts/seed-models.sh rsync ... script also still works.
4. Set CORS_ORIGINS in Coolify to the JSON array of frontend origins that are allowed to call the bot.

That's it — no reindex step. The RAG DB is part of the image.

Option B — plain Docker

git clone https://src.sersofts.org/victorvargas/rony-chat-bot.git
cd rony-chat-bot
cp .env.example .env

# Put the two GGUF files into the `models` volume (one-time, ~2.4 GB):
./scripts/seed-models.sh docker

docker compose up -d --build
docker compose run --rm bot reindex   # first time only
curl -s http://localhost:7331/api/health

Expose port 7331 to your reverse proxy of choice (Caddy, nginx, etc.). The local Compose file configures local CORS origins. In production, configure CORS_ORIGINS in Coolify and redeploy the stack when the value changes.

What is in the image

Image / service	Image / source	Role
src.sersofts.org/victorvargas/llama.cpp:b10333	built from a fork of the upstream ggerganov/llama.cpp	LLM server (Qwen 2.5 3B q4_k_m)
src.sersofts.org/victorvargas/llama.cpp:b10333	built from a fork of the upstream ggerganov/llama.cpp	Embedder (nomic-embed-v2-moe Q5_K_M)
src.sersofts.org/victorvargas/rony-chat-bot:X.Y.Z-alpha	built from this repo's Dockerfile	HTTP bot on :7331 — config + RAG DB baked in

See deploy/coolify/README.md §9 for the measured RAM/CPU budget on a 2 vCPU / 8 GB host.

📚 Documentation

• Architecture doc — Complete technical specification
• Library: rony-llm-agent — Reusable core
• Rony Harness — The other project using the same library

📄 License

MIT — see LICENSE.

🔗 Workspace projects

• rony-llm-agent — Core library
• rony-harness — AI agent harness (TUI)
• portfolio — Astro + React site (integrates this bot)