feat(portfolio): consume content from HTTP sources; per-source isolation #1
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/http-content-sources"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Contexto
El bot duplicaba los .md en data/projects/ y se desincronizaba del portfolio. Este PR elimina la duplicación: el bot consume el contenido por HTTP desde cada entrada de sources: en el YAML. Una sola fuente de verdad por proyecto; añadir una refaccionaria o portfolio2 después es un cambio solo del YAML.
Match con portfolio PR #7 (lado emisor: /projects.json y /docs.json).
Cambios principales
Verificación
PORTFOLIO_URL=http://127.0.0.1:4321/projects.json PORTFOLIO_DOCS_URL=http://127.0.0.1:4321/docs.json ./bin/chat-bot reindex
→ 7 files / 30 chunks
Tras borrar tienda-ropa.mdx del portfolio y reindexear:
→ 6 files / 27 chunks, catálogo sin portfolio:tienda-ropa, search tienda=0 hits, search stripe=0 hits.
→ Bot en widget: "Lo siento, no tengo información sobre un proyecto llamado Tienda de Ropa" (grounding honrado, no alucina).
Tests: go test ./... OK. go build ./... limpio.
Plan completo en portfolio/docs/single-source-of-truth.md.
The bot no longer owns its corpus. Each 'sources:' entry in the YAML points at an HTTP endpoint that returns a JSON array; the bot fetches it on every reindex. Chunks are namespaced by 'name:' so two sources cannot collide on project_id, and a failed reindex in one source does not touch the others' corpus. Adding a refaccionaria or portfolio2 later is a YAML-only change. - internal/portfolio/fetcher.go (new) Fetcher interface + HTTPFetcher (15s timeout, ETag-aware) + LocalFetcher (test/dev fallback) + ProjectJSON. Required fields (slug, title, description) are validated; empty ones return ErrSourceMalformed. - internal/portfolio/indexer.go Source{Name, Endpoint, Kind, Fetcher} replaces Path-based sources. reindexSource runs each source in its own transaction: DELETE WHERE source_name=X for both portfolio_chunks and the joined vectors, re-insert, commit. A failure rolls back that source only. New 'source_name' column on portfolio_chunks; existing ensureChunkSchema drops+recreates on schema mismatch (the standard pattern for FTS5 virtual tables). New portfolio_catalog table tracks (project_id, title, source_name) populated from the JSON title — independent from chunks so an empty H1 section (e.g. '# Title\n\n## Purpose') doesn't break the catalogue display. - internal/portfolio/hybrid.go VectorSearch scans the new source_name column. - internal/config/config.go cfg.Sources []Source; new envWithDefault expander supports ${PORTFOLIO_URL:-https://sersofts.org/projects.json} so a local reindex is one env var away. - configs/portfolio-bot.yaml rag.data_path / rag.docs_path removed. sources: replaces them with the two production endpoints (portfolio and portfolio-docs). server.corsors: 'https://victorvargas.dev' -> 'https://sersofts.org'. - cmd/chat-bot/main.go, internal/server/handlers.go SourcesFor(...) -> SourcesFromConfig(cfg). - Tests updated for namespaced project_ids (e.g. 'portfolio:tienda-ropa') and the new SourcesFromPaths helper for the local-fetcher path. - Legacy data/projects/*.md and data/docs/README.md deleted; the bot no longer reads local files in production. - .gitignore: ignore legacy data/projects and data/docs (kept on disk during migration). - README.md, docs/architecture.md, docs/architecture.es.md Document the new sources: model and the JSON contract each endpoint must honour. Endpoints for a refaccionaria or portfolio2 slot in without code changes. End-to-end verified: 'chat-bot reindex' against a local Astro dev indexed 6 projects + 1 CV into 30 chunks; deleting a project from the collection and reindexing correctly wiped its chunks, catalogue entry and FTS5 hits. Asking the bot about the deleted project produced a 'no tengo información' answer (grounding rule honoured), not a hallucination. Match with portfolio PR #7: portfolio serves /projects.json and /docs.json with the same shape this bot consumes.