RAG is no magic trick — what actually works
Retrieval-augmented generation has two phases of life: demo day (always works) and week three in production (rarely works). Here's what we learned in the first 15 projects.
Spend the first 15 minutes in a RAG demo and everything looks beautiful. The AI answers every question about your documentation accurately, cites sources, phrases things politely. Three weeks later you’re wondering why that same AI suddenly claims contract X runs until 2031 (in reality: expired in 2024).
The problem is almost never the model. It’s the retrieval layer.
Three anti-patterns we see again and again
1. Vector search as a cure-all
Vector embeddings are great for semantic similarity. They are catastrophic when you’re searching for precise values (prices, dates, contract numbers). The cosine similarity between „contract runs until 2024” and „contract runs until 2031” is frighteningly high.
Solution: hybrid retrieval. First BM25 (classic keyword search) for structured facts, then vector search for the fuzzy questions.
2. One index for everything
If your RAG system serves multiple tenants, those tenants need separate vector spaces. Otherwise client A’s knowledge leaks into client B’s answer. In nexxus this is tenant-separated out of the box (via pgvector with the tenant ID as a filter constraint in the index).
3. No eval pipeline
90% of teams have no automated test that checks, after every model or prompt update, whether the answers are still correct. You notice when the first real customer calls.
What we recommend
- Start small, evaluate rigorously. Better 50 perfectly working answers than 5,000 mediocre ones.
- Source citations are mandatory. If the AI says „X is valid until 2024”, it must show which document that came from.
- Fresh data = fresh answers. A RAG system that doesn’t re-index regularly lies politely.
If you’re working on RAG yourself right now and this sounds like a summary of your last three weeks — write to us. We see it more often than we’d like.