Knowledge Graphs + RAG: Structured Context for Better Answers
By Diesel
ragknowledge-graphsstructured-data
## The Limitation Nobody Mentions
Vector search is great at finding text that's similar to your query. It's terrible at answering questions that require understanding relationships between entities.
"Who reports to the VP of Engineering?" Vector search will find documents that mention the VP of Engineering. It might find an org chart page. It might find a job posting. It might find a meeting note where someone mentioned them. What it won't do is reliably return the specific reporting relationship you asked about.
"What services depend on the authentication service?" Vector search finds docs that mention auth. What it can't do is trace the dependency graph.
"Which customers in EMEA have active contracts expiring in Q4 that use our Enterprise tier?" Vector search is going to have a really bad day with this one. This connects directly to [combining structured and vector search](/blog/hybrid-search-rag-production).
These are relationship queries. They require structured knowledge, not text similarity. And in enterprise settings, relationship queries are everywhere.
## What a Knowledge Graph Brings
A knowledge graph is a database of entities (things) and relationships (how things connect). Nodes and edges, if you prefer graph theory language.
```
[John Smith] --reports_to--> [Sarah Chen]
[Sarah Chen] --manages--> [Engineering Team]
[Engineering Team] --owns--> [Auth Service]
[Auth Service] --dependency_of--> [Payment Service]
[Payment Service] --used_by--> [Enterprise Tier]
```
This structure lets you answer relationship queries directly. Traverse the graph. Follow the edges. No semantic similarity needed. The answer is structural, not textual.
## Graph + RAG: The Architecture
The idea is simple. Use the knowledge graph for structured queries (entities, relationships, attributes). Use vector search for unstructured queries (concepts, explanations, discussions). Combine the results.
```python
class GraphRAG:
def __init__(self, graph_db, vector_store, llm):
self.graph = graph_db # Neo4j, Neptune, etc.
self.vectors = vector_store # Pinecone, Qdrant, etc.
self.llm = llm
async def query(self, question: str):
# Step 1: Extract entities from the question
entities = await self.extract_entities(question)
# Step 2: Query the graph for structured context
graph_context = await self.graph_search(entities, question)
# Step 3: Query vectors for unstructured context
vector_context = await self.vectors.search(question, top_k=5)
# Step 4: Combine and generate
return await self.llm.generate(
context=merge_contexts(graph_context, vector_context),
question=question,
)
async def extract_entities(self, question: str):
"""Use LLM to identify entities and relationships in the question."""
response = await self.llm.generate(
ENTITY_EXTRACTION_PROMPT.format(question=question)
)
return parse_entities(response)
async def graph_search(self, entities, question):
"""Query knowledge graph for relevant subgraph."""
results = []
for entity in entities:
# Find the entity in the graph
node = self.graph.find(entity.name, entity.type)
if node:
# Get its neighborhood (1-2 hops)
subgraph = self.graph.neighborhood(node, depth=2)
results.append(subgraph)
return results
``` The related post on [agentic retrieval strategies](/blog/agentic-rag-dynamic-retrieval) goes further on this point.
## Building the Knowledge Graph
This is where most people stall. "Building a knowledge graph sounds like a massive upfront investment." It can be. But it doesn't have to be.
### Approach 1: LLM-Extracted Graphs
Use an LLM to extract entities and relationships from your existing documents during ingestion. No manual graph construction needed.
```python
EXTRACTION_PROMPT = """
Extract entities and relationships from this text.
Format each as: (entity1) -[relationship]-> (entity2)
Types of entities: Person, Team, Service, Product, Document, Project, Company
Types of relationships: reports_to, manages, owns, depends_on, uses,
authored, approved, related_to
Text: {text}
Entities and relationships:
"""
async def extract_graph_from_document(document, llm):
chunks = chunk_document(document)
triples = []
for chunk in chunks:
response = await llm.generate(
EXTRACTION_PROMPT.format(text=chunk.text)
)
triples.extend(parse_triples(response))
return deduplicate_triples(triples)
```
This won't produce a perfect graph. Entity resolution is messy ("John Smith", "J. Smith", "John" might all be the same person). Relationships can be wrong or ambiguous. But a noisy graph is still more useful than no graph for relationship queries.
### Approach 2: Structured Data First
Most organizations already have structured data that forms a natural graph. Org charts. Service catalogs. CRM data. Project management tools. HR systems.
```python
# Import from existing structured sources
def build_graph_from_sources(graph_db):
# Org chart from HR system
for employee in hr_api.list_employees():
graph_db.create_node("Person", {
"name": employee.name,
"title": employee.title,
"department": employee.department,
})
if employee.manager:
graph_db.create_edge(
employee.id, "reports_to", employee.manager_id
)
# Service catalog
for service in service_catalog.list_services():
graph_db.create_node("Service", {
"name": service.name,
"team": service.owning_team,
})
for dep in service.dependencies:
graph_db.create_edge(service.id, "depends_on", dep.id)
```
Start with the structured data you already have. Supplement with LLM extraction from unstructured documents. You don't need a complete graph to get value. Even a partial graph with the most important entities and relationships dramatically improves retrieval for relationship queries.
### Approach 3: Microsoft's GraphRAG
Microsoft's open-source GraphRAG approach automatically builds a knowledge graph from documents using community detection. It clusters related entities into communities, generates summaries for each community, and uses these for high-level queries that span many documents.
```python
# GraphRAG creates hierarchical community summaries
# Level 0: Individual entities and relationships
# Level 1: Communities of related entities
# Level 2: Higher-level themes and topics
# Query routing based on scope
if query_requires_global_understanding:
# Use community summaries (map-reduce over communities)
results = graph_rag.global_search(question)
else:
# Use local entity search + vector retrieval
results = graph_rag.local_search(question)
```
This is particularly powerful for questions like "What are the main themes in our customer feedback this quarter?" that require synthesizing across many documents rather than finding specific chunks.
## When Graphs Beat Vectors (And Vice Versa)
**Graph wins:**
- "What teams are affected if Service X goes down?" (dependency traversal)
- "Who can approve this type of purchase?" (permission chains)
- "Show me all projects related to customer Y" (entity-centric queries)
- "What changed in the last sprint?" (temporal relationship queries)
**Vectors win:**
- "How do we handle customer complaints about shipping?" (conceptual)
- "Explain our security architecture" (descriptive)
- "What's the best practice for database migrations?" (knowledge retrieval)
**Both together:**
- "What's the security posture of services owned by Team Alpha?" Graph finds Team Alpha's services. Vectors find security-related content for those services.
- "Summarize the impact of Project Phoenix on our Q3 targets." Graph finds Project Phoenix's entities and relationships. Vectors find the relevant discussions, reports, and decisions.
## The Entity Resolution Problem
The hardest part of knowledge graphs isn't building them. It's keeping entities consistent.
"John Smith" in an email, "J. Smith" in a doc, "[email protected]" in a ticket, "Smith, John" in the HR system. Are these the same person? Probably. But your graph needs to know for certain.
```python
class EntityResolver:
def resolve(self, entity_mention: str, entity_type: str):
"""Find or create the canonical entity for a mention."""
# Exact match
exact = self.graph.find_exact(entity_mention, entity_type)
if exact:
return exact
# Fuzzy match
candidates = self.graph.find_similar(entity_mention, entity_type)
if candidates:
best = self.rank_candidates(entity_mention, candidates)
if best.confidence > 0.9:
return best.entity
# No match - create new entity
return self.graph.create_node(entity_type, {"name": entity_mention})
```
Entity resolution is an ongoing process, not a one-time setup. As new documents are ingested, new mentions appear, and the resolver needs to continuously merge and deduplicate. This connects directly to [how documents are chunked](/blog/chunking-strategies-at-scale).
## Start Small, Grow Deliberately
Don't try to build a complete enterprise knowledge graph on day one. That way lies madness and abandoned projects.
1. **Start with one structured data source** (org chart, service catalog). Import it as a graph.
2. **Add LLM extraction** for the most important document type in your RAG pipeline.
3. **Wire it into retrieval** alongside your existing vector search.
4. **Measure** whether relationship queries improve.
5. **Expand** the graph incrementally based on what queries are actually failing.
A knowledge graph isn't a project with a finish line. It's infrastructure that grows with your organization. Build it like that, and it'll serve you well. Try to build the whole thing at once, and you'll end up with a beautiful graph that nobody maintains.