2026-06-30 20:27:00 +00:00
# Rony Chat Bot — Portfolio Bot HTTP
> 🌐 **Idioma:** [English](README.md) | [Español](README.es.md)
> 🤖 **Chatbot HTTP que presenta tu portfolio y responde preguntas sobre tus proyectos.**
**Rony Chat Bot** es un chatbot basado en [`rony-llm-agent` ](https://github.com/VictorVargas/rony-llm-agent ) que se integra con un sitio Astro/React para responder preguntas sobre Victor Hugo Vargas y sus proyectos, usando **RAG sobre archivos markdown** .
## ✨ Features
- 🌐 **HTTP server** con streaming SSE (Server-Sent Events)
feat(rag): hybrid retrieval, reference documents, and vendor sampling
Answers were short, sometimes in the wrong language, and occasionally about
projects that do not exist. Measured on a 20-question battery against the real
corpus in both Spanish and English, this takes grounded content from 3/10 to
9/10 and language matching from 7/10 to 10/10.
Retrieval
- Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion.
Both halves are load-bearing: the corpus is English and visitors ask in
Spanish, so the meaningful words score zero. "paga" appears 0 times in a
document that says "Payments: Stripe" — the question "¿Con qué se paga en la
tienda de ropa?" retrieved nothing at all. Embeddings put all three of that
project's chunks on top. RRF ranks by agreement rather than comparing a BM25
score against a cosine, quantities with no shared scale.
- internal/embed: OpenAI-compatible embeddings client, unit-normalised so a
dot product is the cosine. Reorders by the response `index` field.
- Store a content hash beside each vector and skip rows where it no longer
matches the chunk. Chunk ids survive body edits, so without this an edited
document keeps serving embeddings that describe text that is gone —
reproduced live by changing a payment provider and watching the old one keep
coming back.
- Degrade to keyword-only when the embedder is down instead of failing.
Reference documents that are not projects
- Index `.mdx` alongside `.md`, and split sources into projects (announced in
the catalogue) and reference material (retrievable, never listed). A CV is
what someone deciding whether to hire actually reads, and it was unreachable
while it lived only in the Astro site — but filing it under projects made
the bot list "cv" as one of Victor's works.
- Skip each directory's README. `data/projects/README.md` was being indexed,
so the catalogue injected into every prompt announced "README" and
"README.es" as projects of Victor's.
- Exclude frontmatter from retrieval. It is dense metadata in a very short
chunk, which makes it a magnet for short queries: a CV's `location:` field
answered "¿Dónde ha trabajado Victor?" with a city instead of a work history.
- Split oversized sections at `###` before falling back to byte offsets. A CV's
Experience section is a list of jobs, and size-splitting cut one mid-word,
stranding the employer's name in the previous chunk.
Prompt and sampling
- Inject the full project catalogue every turn. Top-K search returns the best
matching sections, so "list every project" cannot be answered from retrieval
alone, and a small model asked to enumerate from partial hits invents the
rest. ~10 tokens per project; this is what stopped the invented names.
- Wire the sampling parameters the model authors publish (top_k, top_p, min_p,
repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them
at llama.cpp's defaults produced 16-token stub answers.
- Localised system prompt selected by detected language. The English prompt
plus "reply in the user's language" answered 1/5 Spanish questions in
Spanish; few-shot examples fixed the language but got copied verbatim into
real answers.
- Fold compaction's system notes into the leading system message. Gemma's chat
template rejects a system message that is not first, and the whole request
failed with HTTP 400 the moment compaction fired.
Configuration and docs
- context_size 4096, down from 8192. The largest prompt this bot ever built
over 20 real requests was 1255 tokens, compaction starts at ~3070, and the
cut saved 212 MB resident with zero truncations and identical throughput.
- Correct the RAM figures throughout. They were measured with a GPU absorbing
llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which
is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs
still started gemma-3-1b while the config defaulted to qwen, and neither
started the embedder at all.
Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder
+ 0.02 GB bot, 21.0 tok/s steady state.
Known and unfixed, so they are not re-filed as new bugs: the model reads dates
out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha
trabajado Victor?" still answers with projects rather than employers, though
"¿En qué empresas ha trabajado?" works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 21:45:37 +00:00
- 🧠 **RAG híbrido sobre markdown/MDX** — búsqueda por palabras con SQLite FTS5 fusionada con embeddings multilingües (Reciprocal Rank Fusion)
2026-06-30 20:27:00 +00:00
- 🎭 **Persona customizable** — responde como "asistente de Victor"
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
- ⚡ **Self-hosted** con llama.cpp (default) u Ollama (no requiere API key de cloud)
- 💬 **Widget de chat drop-in** — vanilla JS, sin build step, funciona en cualquier sitio
2026-06-30 20:27:00 +00:00
- 🛡️ **Rate limiting** y logging estructurado
- 📦 **Portable** — se puede adaptar a otros contextos (clientes, productos, etc.)
## 🚀 Quick start
```bash
# 1. Instalar
git clone https://github.com/VictorVargas/rony-chat-bot.git
feat(rag): hybrid retrieval, reference documents, and vendor sampling
Answers were short, sometimes in the wrong language, and occasionally about
projects that do not exist. Measured on a 20-question battery against the real
corpus in both Spanish and English, this takes grounded content from 3/10 to
9/10 and language matching from 7/10 to 10/10.
Retrieval
- Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion.
Both halves are load-bearing: the corpus is English and visitors ask in
Spanish, so the meaningful words score zero. "paga" appears 0 times in a
document that says "Payments: Stripe" — the question "¿Con qué se paga en la
tienda de ropa?" retrieved nothing at all. Embeddings put all three of that
project's chunks on top. RRF ranks by agreement rather than comparing a BM25
score against a cosine, quantities with no shared scale.
- internal/embed: OpenAI-compatible embeddings client, unit-normalised so a
dot product is the cosine. Reorders by the response `index` field.
- Store a content hash beside each vector and skip rows where it no longer
matches the chunk. Chunk ids survive body edits, so without this an edited
document keeps serving embeddings that describe text that is gone —
reproduced live by changing a payment provider and watching the old one keep
coming back.
- Degrade to keyword-only when the embedder is down instead of failing.
Reference documents that are not projects
- Index `.mdx` alongside `.md`, and split sources into projects (announced in
the catalogue) and reference material (retrievable, never listed). A CV is
what someone deciding whether to hire actually reads, and it was unreachable
while it lived only in the Astro site — but filing it under projects made
the bot list "cv" as one of Victor's works.
- Skip each directory's README. `data/projects/README.md` was being indexed,
so the catalogue injected into every prompt announced "README" and
"README.es" as projects of Victor's.
- Exclude frontmatter from retrieval. It is dense metadata in a very short
chunk, which makes it a magnet for short queries: a CV's `location:` field
answered "¿Dónde ha trabajado Victor?" with a city instead of a work history.
- Split oversized sections at `###` before falling back to byte offsets. A CV's
Experience section is a list of jobs, and size-splitting cut one mid-word,
stranding the employer's name in the previous chunk.
Prompt and sampling
- Inject the full project catalogue every turn. Top-K search returns the best
matching sections, so "list every project" cannot be answered from retrieval
alone, and a small model asked to enumerate from partial hits invents the
rest. ~10 tokens per project; this is what stopped the invented names.
- Wire the sampling parameters the model authors publish (top_k, top_p, min_p,
repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them
at llama.cpp's defaults produced 16-token stub answers.
- Localised system prompt selected by detected language. The English prompt
plus "reply in the user's language" answered 1/5 Spanish questions in
Spanish; few-shot examples fixed the language but got copied verbatim into
real answers.
- Fold compaction's system notes into the leading system message. Gemma's chat
template rejects a system message that is not first, and the whole request
failed with HTTP 400 the moment compaction fired.
Configuration and docs
- context_size 4096, down from 8192. The largest prompt this bot ever built
over 20 real requests was 1255 tokens, compaction starts at ~3070, and the
cut saved 212 MB resident with zero truncations and identical throughput.
- Correct the RAM figures throughout. They were measured with a GPU absorbing
llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which
is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs
still started gemma-3-1b while the config defaulted to qwen, and neither
started the embedder at all.
Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder
+ 0.02 GB bot, 21.0 tok/s steady state.
Known and unfixed, so they are not re-filed as new bugs: the model reads dates
out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha
trabajado Victor?" still answers with projects rather than employers, though
"¿En qué empresas ha trabajado?" works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 21:45:37 +00:00
cd rony-chat-bot
2026-06-30 20:27:00 +00:00
# 2. Resolver dependencias (crea go.sum con hashes)
go mod tidy
feat(rag): hybrid retrieval, reference documents, and vendor sampling
Answers were short, sometimes in the wrong language, and occasionally about
projects that do not exist. Measured on a 20-question battery against the real
corpus in both Spanish and English, this takes grounded content from 3/10 to
9/10 and language matching from 7/10 to 10/10.
Retrieval
- Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion.
Both halves are load-bearing: the corpus is English and visitors ask in
Spanish, so the meaningful words score zero. "paga" appears 0 times in a
document that says "Payments: Stripe" — the question "¿Con qué se paga en la
tienda de ropa?" retrieved nothing at all. Embeddings put all three of that
project's chunks on top. RRF ranks by agreement rather than comparing a BM25
score against a cosine, quantities with no shared scale.
- internal/embed: OpenAI-compatible embeddings client, unit-normalised so a
dot product is the cosine. Reorders by the response `index` field.
- Store a content hash beside each vector and skip rows where it no longer
matches the chunk. Chunk ids survive body edits, so without this an edited
document keeps serving embeddings that describe text that is gone —
reproduced live by changing a payment provider and watching the old one keep
coming back.
- Degrade to keyword-only when the embedder is down instead of failing.
Reference documents that are not projects
- Index `.mdx` alongside `.md`, and split sources into projects (announced in
the catalogue) and reference material (retrievable, never listed). A CV is
what someone deciding whether to hire actually reads, and it was unreachable
while it lived only in the Astro site — but filing it under projects made
the bot list "cv" as one of Victor's works.
- Skip each directory's README. `data/projects/README.md` was being indexed,
so the catalogue injected into every prompt announced "README" and
"README.es" as projects of Victor's.
- Exclude frontmatter from retrieval. It is dense metadata in a very short
chunk, which makes it a magnet for short queries: a CV's `location:` field
answered "¿Dónde ha trabajado Victor?" with a city instead of a work history.
- Split oversized sections at `###` before falling back to byte offsets. A CV's
Experience section is a list of jobs, and size-splitting cut one mid-word,
stranding the employer's name in the previous chunk.
Prompt and sampling
- Inject the full project catalogue every turn. Top-K search returns the best
matching sections, so "list every project" cannot be answered from retrieval
alone, and a small model asked to enumerate from partial hits invents the
rest. ~10 tokens per project; this is what stopped the invented names.
- Wire the sampling parameters the model authors publish (top_k, top_p, min_p,
repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them
at llama.cpp's defaults produced 16-token stub answers.
- Localised system prompt selected by detected language. The English prompt
plus "reply in the user's language" answered 1/5 Spanish questions in
Spanish; few-shot examples fixed the language but got copied verbatim into
real answers.
- Fold compaction's system notes into the leading system message. Gemma's chat
template rejects a system message that is not first, and the whole request
failed with HTTP 400 the moment compaction fired.
Configuration and docs
- context_size 4096, down from 8192. The largest prompt this bot ever built
over 20 real requests was 1255 tokens, compaction starts at ~3070, and the
cut saved 212 MB resident with zero truncations and identical throughput.
- Correct the RAM figures throughout. They were measured with a GPU absorbing
llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which
is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs
still started gemma-3-1b while the config defaulted to qwen, and neither
started the embedder at all.
Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder
+ 0.02 GB bot, 21.0 tok/s steady state.
Known and unfixed, so they are not re-filed as new bugs: the model reads dates
out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha
trabajado Victor?" still answers with projects rather than employers, though
"¿En qué empresas ha trabajado?" works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 21:45:37 +00:00
# 3. Descargá los modelos: un LLM instruct y un embebedor multilingüe
# https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF (~2 GB)
# https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe-GGUF (~370 MB)
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
export RONY_MODELS_PATH=/path/to/models
2026-06-30 20:27:00 +00:00
# 4. Cargar tus proyectos en data/projects/
echo "# Mi Proyecto Cool\nDescripción..." > data/projects/mi-proyecto.md
# 5. Build
go build -o bin/chat-bot ./cmd/chat-bot
feat(rag): hybrid retrieval, reference documents, and vendor sampling
Answers were short, sometimes in the wrong language, and occasionally about
projects that do not exist. Measured on a 20-question battery against the real
corpus in both Spanish and English, this takes grounded content from 3/10 to
9/10 and language matching from 7/10 to 10/10.
Retrieval
- Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion.
Both halves are load-bearing: the corpus is English and visitors ask in
Spanish, so the meaningful words score zero. "paga" appears 0 times in a
document that says "Payments: Stripe" — the question "¿Con qué se paga en la
tienda de ropa?" retrieved nothing at all. Embeddings put all three of that
project's chunks on top. RRF ranks by agreement rather than comparing a BM25
score against a cosine, quantities with no shared scale.
- internal/embed: OpenAI-compatible embeddings client, unit-normalised so a
dot product is the cosine. Reorders by the response `index` field.
- Store a content hash beside each vector and skip rows where it no longer
matches the chunk. Chunk ids survive body edits, so without this an edited
document keeps serving embeddings that describe text that is gone —
reproduced live by changing a payment provider and watching the old one keep
coming back.
- Degrade to keyword-only when the embedder is down instead of failing.
Reference documents that are not projects
- Index `.mdx` alongside `.md`, and split sources into projects (announced in
the catalogue) and reference material (retrievable, never listed). A CV is
what someone deciding whether to hire actually reads, and it was unreachable
while it lived only in the Astro site — but filing it under projects made
the bot list "cv" as one of Victor's works.
- Skip each directory's README. `data/projects/README.md` was being indexed,
so the catalogue injected into every prompt announced "README" and
"README.es" as projects of Victor's.
- Exclude frontmatter from retrieval. It is dense metadata in a very short
chunk, which makes it a magnet for short queries: a CV's `location:` field
answered "¿Dónde ha trabajado Victor?" with a city instead of a work history.
- Split oversized sections at `###` before falling back to byte offsets. A CV's
Experience section is a list of jobs, and size-splitting cut one mid-word,
stranding the employer's name in the previous chunk.
Prompt and sampling
- Inject the full project catalogue every turn. Top-K search returns the best
matching sections, so "list every project" cannot be answered from retrieval
alone, and a small model asked to enumerate from partial hits invents the
rest. ~10 tokens per project; this is what stopped the invented names.
- Wire the sampling parameters the model authors publish (top_k, top_p, min_p,
repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them
at llama.cpp's defaults produced 16-token stub answers.
- Localised system prompt selected by detected language. The English prompt
plus "reply in the user's language" answered 1/5 Spanish questions in
Spanish; few-shot examples fixed the language but got copied verbatim into
real answers.
- Fold compaction's system notes into the leading system message. Gemma's chat
template rejects a system message that is not first, and the whole request
failed with HTTP 400 the moment compaction fired.
Configuration and docs
- context_size 4096, down from 8192. The largest prompt this bot ever built
over 20 real requests was 1255 tokens, compaction starts at ~3070, and the
cut saved 212 MB resident with zero truncations and identical throughput.
- Correct the RAM figures throughout. They were measured with a GPU absorbing
llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which
is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs
still started gemma-3-1b while the config defaulted to qwen, and neither
started the embedder at all.
Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder
+ 0.02 GB bot, 21.0 tok/s steady state.
Known and unfixed, so they are not re-filed as new bugs: the model reads dates
out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha
trabajado Victor?" still answers with projects rather than employers, though
"¿En qué empresas ha trabajado?" works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 21:45:37 +00:00
# 6. Arrancar el LLM (CPU, target 2 cores — ajustá --threads a tu host)
llama-server \
-m $RONY_MODELS_PATH/Qwen2.5/qwen2.5-3b-instruct-q4_k_m.gguf \
--port 9100 --ctx-size 4096 --parallel 1 \
--device none --threads 2 --mlock \
--temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
# 7. Arrancar el embebedor (segunda terminal)
llama-server \
-m $RONY_MODELS_PATH/embeddings/nomic-embed-v2-moe.Q5_K_M.gguf \
--port 9200 --embedding --pooling mean \
--ctx-size 2048 --parallel 1 --device none --threads 2
# 8. Construir el índice (necesita el embebedor arriba) y servir
./bin/chat-bot reindex
2026-06-30 20:27:00 +00:00
./bin/chat-bot serve
# → Sirve en http://localhost:7331
```
feat(rag): hybrid retrieval, reference documents, and vendor sampling
Answers were short, sometimes in the wrong language, and occasionally about
projects that do not exist. Measured on a 20-question battery against the real
corpus in both Spanish and English, this takes grounded content from 3/10 to
9/10 and language matching from 7/10 to 10/10.
Retrieval
- Fuse FTS5 keyword search with dense vectors via Reciprocal Rank Fusion.
Both halves are load-bearing: the corpus is English and visitors ask in
Spanish, so the meaningful words score zero. "paga" appears 0 times in a
document that says "Payments: Stripe" — the question "¿Con qué se paga en la
tienda de ropa?" retrieved nothing at all. Embeddings put all three of that
project's chunks on top. RRF ranks by agreement rather than comparing a BM25
score against a cosine, quantities with no shared scale.
- internal/embed: OpenAI-compatible embeddings client, unit-normalised so a
dot product is the cosine. Reorders by the response `index` field.
- Store a content hash beside each vector and skip rows where it no longer
matches the chunk. Chunk ids survive body edits, so without this an edited
document keeps serving embeddings that describe text that is gone —
reproduced live by changing a payment provider and watching the old one keep
coming back.
- Degrade to keyword-only when the embedder is down instead of failing.
Reference documents that are not projects
- Index `.mdx` alongside `.md`, and split sources into projects (announced in
the catalogue) and reference material (retrievable, never listed). A CV is
what someone deciding whether to hire actually reads, and it was unreachable
while it lived only in the Astro site — but filing it under projects made
the bot list "cv" as one of Victor's works.
- Skip each directory's README. `data/projects/README.md` was being indexed,
so the catalogue injected into every prompt announced "README" and
"README.es" as projects of Victor's.
- Exclude frontmatter from retrieval. It is dense metadata in a very short
chunk, which makes it a magnet for short queries: a CV's `location:` field
answered "¿Dónde ha trabajado Victor?" with a city instead of a work history.
- Split oversized sections at `###` before falling back to byte offsets. A CV's
Experience section is a list of jobs, and size-splitting cut one mid-word,
stranding the employer's name in the previous chunk.
Prompt and sampling
- Inject the full project catalogue every turn. Top-K search returns the best
matching sections, so "list every project" cannot be answered from retrieval
alone, and a small model asked to enumerate from partial hits invents the
rest. ~10 tokens per project; this is what stopped the invented names.
- Wire the sampling parameters the model authors publish (top_k, top_p, min_p,
repeat_penalty, presence_penalty) through config to llama.cpp. Leaving them
at llama.cpp's defaults produced 16-token stub answers.
- Localised system prompt selected by detected language. The English prompt
plus "reply in the user's language" answered 1/5 Spanish questions in
Spanish; few-shot examples fixed the language but got copied verbatim into
real answers.
- Fold compaction's system notes into the leading system message. Gemma's chat
template rejects a system message that is not first, and the whole request
failed with HTTP 400 the moment compaction fired.
Configuration and docs
- context_size 4096, down from 8192. The largest prompt this bot ever built
over 20 real requests was 1255 tokens, compaction starts at ~3070, and the
cut saved 212 MB resident with zero truncations and identical throughput.
- Correct the RAM figures throughout. They were measured with a GPU absorbing
llama.cpp's buffers; on a GPU-less VPS those come out of system RAM, which
is 1.1 GB more for qwen2.5-3b and 2.8 GB more for granite. Both READMEs
still started gemma-3-1b while the config defaulted to qwen, and neither
started the embedder at all.
Measured on the 2-core, 8 GB CPU-only target: 3.64 GB LLM + 0.91 GB embedder
+ 0.02 GB bot, 21.0 tok/s steady state.
Known and unfixed, so they are not re-filed as new bugs: the model reads dates
out of the CV correctly but does the arithmetic on them wrong, and "¿Dónde ha
trabajado Victor?" still answers with projects rather than employers, though
"¿En qué empresas ha trabajado?" works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 21:45:37 +00:00
**Hardware mínimo:** 2 CPU cores, 8 GB RAM, sin GPU. Memoria residente medida
en CPU con este setup: **3,64 GB** el LLM, **0,91 GB** el embebedor, **0,02 GB**
el bot — unos **4,6 GB** , dejando ~3,4 GB para el resto del host. La generación
va a 21 tok/s con 2 threads una vez que el system prompt está caliente en la
caché de prompts de llama-server.
Tres flags son fáciles de errar y cada una te cuesta calidad real:
- **`--device none`** — llama.cpp levanta el backend de GPU compilado aunque
pases `-ngl 0` , y en un host sin GPU esos buffers salen de la RAM del sistema.
Medido en qwen2.5-3b: 2,54 GB con una GPU absorbiéndolos, 3,66 GB sin ella.
Presupuestá con el segundo número, y pasá la flag para que la medición
coincida con lo que hace producción.
- **`--parallel 1`** — `--ctx-size` se reparte entre slots y llama-server abre 4
por defecto, así que `--ctx-size 4096` sin esto le deja a cada request solo
1024 tokens. El rate limiter del bot ya limita la concurrencia.
- **`--temp` / `--top-k` / `--top-p` ** — usá los valores que publican los autores
del modelo, no los defaults de llama.cpp. Los de arriba son los de Qwen para
chat instruct. Errar esto no es sutil: gemma-3-1b con `temperature 0.7` y el
resto sin setear devolvía respuestas de 16 tokens.
**`--pooling mean` es obligatoria en el embebedor.** Sin ella el endpoint no
devuelve un vector por entrada y el cliente rechaza la respuesta.
Mantené `context_size` en `configs/portfolio-bot.yaml` igual a `--ctx-size` ; el
bot calcula sus presupuestos de RAG y compactación a partir de ese número y no
le pregunta al servidor qué tiene en realidad. Si los desacoplás, el bot armará
prompts que el servidor rechaza.
**Por qué 4096 alcanza.** Medido sobre 20 peticiones reales, el prompt más
grande que este bot llegó a construir fue de **1255 tokens** — system prompt,
catálogo de proyectos, cinco chunks recuperados y la pregunta. La compactación
recién arranca al 75% de la ventana (~3070 tokens), así que hay 2,4x de margen
antes de que empiece. Bajar de 8192 a 4096 ahorró **212 MB** de memoria
residente sin un solo truncamiento y con el mismo rendimiento (21,0 tok/s en
ambos casos): el contexto extra estaba reservado y nunca se usaba.
## 📚 Proyectos vs. documentos de referencia
El índice tiene dos tipos de fuente, y ambas aceptan `.md` y `.mdx` :
```yaml
rag:
data_path: ./data/projects # proyectos → aparecen en el catálogo
docs_path: ./data/docs # referencia → se busca, nunca se lista
```
Todo lo que está en `data_path` es un proyecto tuyo y se anuncia en el catálogo
que el bot inyecta en cada prompt. Todo lo que está en `docs_path` es evidencia
buscable que *no* es un proyecto: tu CV, una página "sobre mí", un FAQ.
Tu CV va en `docs_path` . Es el documento que responde lo que de verdad pregunta
quien está evaluando contratarte ("¿sabe Kubernetes?", "¿dónde ha trabajado?"),
y nada de eso es recuperable mientras viva solo en tu sitio. Enlazalo para
mantener una sola copia:
```bash
mkdir -p data/docs
ln -s ../../../portfolio/src/content/cv/cv.mdx data/docs/cv.mdx
./bin/chat-bot reindex
```
Sin esta distinción el CV tendría que ir en `data_path` para ser buscable, y
entonces el bot lista alegremente "cv" como uno de tus proyectos.
Actualizar una instalación existente no requiere migración: la tabla de chunks
es dato derivado, así que el store la reconstruye al abrir y el siguiente
`reindex` la repuebla.
## 🔍 Recuperación: keyword + embeddings
La recuperación es híbrida, y las dos mitades hacen falta.
**Keyword (SQLite FTS5)** hace coincidencia exacta de palabras: sin stemming,
sin traducción. Preciso para nombres propios raros, inútil entre idiomas. El
corpus está en inglés y los visitantes preguntan en español, así que las
palabras con carga semántica puntúan cero: medido sobre el corpus real,
`"paga"` aparece 0 veces en un documento que dice *"Payments: Stripe"* , y
`"trabajado"` 0 veces en uno que dice *"worked"* . La pregunta *"¿Con qué se
paga en la tienda de ropa?"* no recuperaba **nada** .
**Embeddings** (`nomic-embed-v2-moe`, multilingüe) cierran ese hueco: esa
misma pregunta pone los tres chunks de `tienda-ropa` arriba. Son más difusos
que BM25 ante un término raro exacto, y por eso se conservan ambos y se
fusionan con Reciprocal Rank Fusion — RRF ordena por acuerdo entre los dos
rankings, evitando comparar una puntuación BM25 con un coseno, magnitudes sin
escala común.
Se activa en `configs/portfolio-bot.yaml` bajo `embeddings:` y hay que
re-ejecutar `reindex` — los vectores se construyen al indexar. Si el endpoint
se cae o se desactiva, la recuperación degrada a keyword-only en vez de fallar.
Dos detalles que cuestan precisión y son fáciles de pasar por alto:
- **El frontmatter se excluye de la recuperación.** Son metadatos densos
(title, tags, repo, location) en un chunk muy corto, lo que lo convierte en
imán de consultas breves. El campo `location:` de un CV hacía que *"¿Dónde
ha trabajado Victor?"* recuperara el frontmatter en vez del historial
laboral, porque "dónde" casa con una ubicación.
- **Las secciones largas se cortan en encabezados `###` , no por bytes.** La
sección Experience de un CV es una lista de empleos; el corte por tamaño
partía una entrada a mitad de palabra, dejando un chunk que empezaba *"...
id app for an on-demand ride-sharing service"* y el nombre del empleador
huérfano en el trozo anterior. Ahora cada chunk es un empleo, nombrado
`Experience — Metrimex — Frontend Developer` .
2026-06-30 20:27:00 +00:00
## 📁 Estructura
```
2026-06-30 21:39:54 +00:00
rony-chat-bot/
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
├── cmd/chat-bot/ # Entry point (CLI)
2026-06-30 20:27:00 +00:00
├── internal/
│ ├── server/ # HTTP handlers + SSE
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
│ ├── agent/ # LLM client + RAG + persona runner
2026-06-30 20:27:00 +00:00
│ ├── portfolio/ # Data loader (markdown → RAG)
│ ├── persona/ # Persona override
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
│ ├── streaming/ # SSE helpers
│ └── i18n/ # Detección de idioma (EN/ES)
├── web/ # ← WIDGET DE CHAT DROP-IN
│ ├── chat-widget.js
│ ├── chat-widget.css
│ └── example.html
2026-06-30 20:27:00 +00:00
├── data/projects/ # ← TUS PROYECTOS EN MARKDOWN
│ ├── rony-tui.md
│ ├── rony-llm-agent.md
│ └── ...
├── configs/
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
│ └── portfolio-bot.yaml # Provider + RAG + persona config
2026-06-30 20:27:00 +00:00
├── docs/
│ └── architecture.md # ← Especificación técnica completa
└── go.mod # require rony-llm-agent
```
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
## 🎯 Embebido en cualquier sitio
El bot viene con un widget de chat drop-in. Agrega dos archivos y un tag `<script>` :
```html
< link rel = "stylesheet" href = "/chat-widget.css" >
< script src = "/chat-widget.js"
data-api-url="https://chat.example.com"
data-title="Pregúntame lo que sea"
data-position="bottom-right"
data-theme="auto"
defer>< / script >
2026-06-30 20:27:00 +00:00
```
feat: bootstrap rony-chat-bot Go module
Initial implementation of the bot:
- cmd/chat-bot: CLI entrypoint (serve, reindex, ask, version)
- internal/agent: LLM provider client + agent runner with RAG injection
- internal/config: YAML config loader (providers, RAG, persona, server)
- internal/i18n: response-language detection (EN/ES)
- internal/persona: persona system prompt assembly from YAML
- internal/portfolio: heading-based chunker + SQLite FTS5 indexer
- internal/server: chi router with /api/chat (SSE), /api/health, /api/info,
/api/reindex, middleware (RequestID, Logging, CORS, RateLimit)
- internal/streaming: SSE protocol helpers (start, chunk, sources, done, error)
- web/: drop-in vanilla-JS chat widget (no build, no deps) + demo + README
- bench/: reproducible driver benchmark (modernc vs mattn SQLite)
- configs/portfolio-bot.yaml: llama.cpp default provider, SQLite RAG, canine persona
- docs/architecture.md / .es.md: aligned with SQLite FTS5 + llama.cpp decisions
- data/projects/README*.md: project data documentation
- README.md / .es.md: updated for current implementation
All tests pass (go test ./...). Bot is functional end-to-end with the
configured LLM provider.
2026-07-17 07:56:06 +00:00
Ver [`web/README.md` ](./web/README.md ) para la referencia completa de configuración y snippets de integración con Astro/Next.js. Arquitectura completa en [`docs/architecture.md` ](./docs/architecture.md ) §5.
2026-06-30 20:27:00 +00:00
## 🔄 Adaptar a otro cliente
Este bot está diseñado para ser **atómico** y reusable. Para adaptarlo (ej. chatbot para un concesionario):
1. Fork/clone este repo
2. Reemplaza `data/projects/` con `data/inventory/` (u otro dominio)
3. Actualiza `configs/portfolio-bot.yaml` con la nueva persona
4. Deploy
La librería `rony-llm-agent` no cambia.
## 📚 Documentación
- [**Architecture doc** ](./docs/architecture.md ) — Especificación técnica completa
- [Library: `rony-llm-agent` ](https://github.com/VictorVargas/rony-llm-agent ) — Core reutilizable
- [Harness ](https://github.com/VictorVargas/rony-harness ) — El otro proyecto que usa la misma librería
## 📄 Licencia
MIT — ver [`LICENSE` ](./LICENSE ).
## 🔗 Proyectos del workspace
- [`rony-llm-agent` ](https://github.com/VictorVargas/rony-llm-agent ) — Librería core
- [`harness` ](https://github.com/VictorVargas/rony-harness ) — AI agent harness (TUI)
- [`portfolio` ](https://github.com/VictorVargas/portfolio ) — Astro + React site (integra este bot)
