The short answer
RAG is the right answer when the knowledge changes frequently, lives across many documents, and queries need source attribution. It is the wrong answer when the data is small enough for long context, structured enough for SQL, narrow enough for fine-tuning, or relational enough for graph queries. Most production failures are vector-only RAG forced onto problems where hybrid retrieval (BM25 + vector + structured) or a different approach entirely would have worked.
When RAG is the right answer
Knowledge that changes more often than you would re-fine-tune. Knowledge that spans more documents than fits in a context window. Workflows that require source attribution (compliance, legal, customer-facing claims). For these, retrieval-augmented generation is the right tool — and hybrid retrieval (BM25 + dense vectors + metadata filtering) is what works in production, not vector-only.
When long context beats RAG
Frontier models now handle 200K–2M token contexts. If the relevant knowledge for a workflow fits in a single context window and stays mostly stable, long context is simpler, lower-latency, and often higher-quality than RAG. Examples: full contract review where the contract is the entire input, codebase-aware refactoring on small repos, document comparison.
When structured queries beat RAG
If your question is 'what is the revenue for customer X in Q3' the right answer is SQL against the warehouse, not vector search over financial documents. RAG over structured data is a category error that produces confidently wrong numbers. Use the LLM to translate questions into queries, then execute the query deterministically.
When fine-tuning beats RAG
When the workflow needs a specific style, format, or domain vocabulary that prompting cannot reliably elicit. Examples: medical-coding output in a strict format, legal-citation generation in jurisdiction-specific style, customer-support tone calibration. Fine-tuning encodes the style; retrieval still supplies the facts.
When graph retrieval beats vector RAG
When relationships are the primary information (org structures, dependency graphs, supply chains, knowledge graphs). Vector search loses the graph; GraphRAG or hybrid graph + vector preserves it. Microsoft's GraphRAG and Neo4j's vector-plus-graph patterns are mature enough for production.
The hybrid pattern that beats every pure approach
Production retrieval almost always combines BM25 (keyword), dense vectors (semantic), and structured filters (metadata, ACLs). Add a reranker on top. The system that combines four retrieval strategies and reranks the union beats every pure approach in evaluation, every time.
Key takeaways
- RAG is the default, not a universal answer. Match retrieval to the data shape.
- Hybrid retrieval (BM25 + vector + structured + reranker) outperforms vector-only in production.
- Long context, SQL, fine-tuning, and graph retrieval each have workflows where they beat RAG.
- RAG over structured numeric data is the most common architectural mistake we see.
Frequently asked questions
Should we still use vector databases?+
Yes — they are excellent for semantic retrieval. The mistake is using them as the only retrieval mechanism instead of one component of a hybrid system.
Is fine-tuning still worth it with long-context models?+
Yes, for workflows requiring strict format or domain style. For knowledge updates, prompt + retrieval is faster and cheaper than re-tuning.
How do we know which retrieval approach to use?+
Run an evaluation across approaches on 50–100 representative queries. The pattern is consistent across industries: hybrid wins on accuracy, latency, and cost — almost regardless of domain.