KAG vs RAG: why we switched to knowledge-augmented generation for legal AI
I led the AI work on the document AI we shipped into Smart Advocate, a legal case management platform used by 20,000+ legal professionals, and we tried both approaches. Standard RAG struggled with questions that span several documents. The knowledge graph, together with a fine-tuned model, gave a 40% accuracy gain over standard RAG.
How we measured the 40% and what it doesn't isolate
What RAG gets right, and where it breaks
Standard RAG works well for FAQ bots and document Q&A. Legal case files are a harder fit. They are long. A fact on page 3 can contradict one on page 112, and a witness in deposition 1 can conflict with testimony in deposition 4.
Standard RAG chunks documents and retrieves the top-K similar passages. It has no way to know that two passages are related, contradict each other or belong to the same chain of reasoning. Each chunk is retrieved on its own, so it misses those links.
What the graph changes
KAG replaces the flat vector index with a knowledge graph. Instead of storing document chunks independently, it extracts entities (people, dates, events, claims, contracts) and the relationships between them into a graph database. We used Neo4j.
At query time, instead of retrieving isolated text chunks, we traverse the graph. We can ask: what are all the claims made by Witness A, and which of them conflict with the insurance policy terms from Exhibit 3? Vector search misses those, because the passages it needs are often not similar to the query or to each other.
How it works
The system we built for Smart Advocate uses a four-stage pipeline managed by LangGraph:
- PDFs, scanned exhibits and court transcripts go through an OCR pipeline. We fine-tuned a model with LoRA for layout-aware extraction, because standard OCR loses table structure in legal filings.
- An LLM agent extracts named entities and structured relationships and writes them to Neo4j. A witness mention gets linked to their deposition, their employer and every claim they made.
- At query time, the LLM turns the user's question into a Cypher query: a graph traversal that retrieves connected facts rather than similar sentences.
- The retrieved subgraph goes to the LLM with a prompt that tells it to flag conflicts. The system surfaces contradictions that a human reviewer would have to read the entire file to find.
Results
After switching from standard RAG to the knowledge-graph approach with a fine-tuned model, we measured a 40% accuracy gain over standard RAG. The biggest gains were on multi-hop questions: queries that required connecting information from more than two documents.
For contradiction detection, the graph links conflicting passages directly.
How we measured the 40%
The 40% is the accuracy gain of the knowledge-graph system, which used a fine-tuned model, over standard RAG. Because both changed together, this doesn't isolate the graph's share. A GPT-4-class model scored each answer as an LLM judge, BLEU and ROUGE compared the generated answers with reference answers, and precision@k and recall@k checked whether the right passages came back in the top k results.
None of these is enough on its own. BLEU and ROUGE reward word overlap, so a correct answer worded differently can score low and a fluent wrong one can score well. An LLM judge is only as good as its rubric, so good practice is to spot-check its scores against human review. Precision@k and recall@k tell you whether retrieval found the right material, not whether the answer used it correctly. That is why it helps to read them together.
When to use RAG vs KAG
Use RAG when your documents are relatively independent and queries are single-hop. For FAQ bots, product documentation and customer support, RAG is faster to build and good enough.
Use KAG when the relationships between facts matter, as they do in legal and medical case files: anywhere a question spans several documents and the connections between them are what you are looking for.