LegalRAG
Built during an AI engineering internship · Jan 2025 – May 2025
A legal intelligence chat platform where an autonomous agent decides when to search a vector database of legal documents, grounds its answers in retrieved passages, and streams every tool call and citation to the UI in real time.
60%+
faster document review
30%
fewer low-confidence responses
75%
less manual lookup time
Problem
Legal research is retrieval-heavy and precision-critical: attorneys and paralegals spend hours locating the right clause, precedent, or definition across large document sets, and a confidently wrong answer is worse than no answer.
LegalRAG turns that workflow into a conversation — but one where every answer is auditable. The agent searches a legal document database on its own judgment, and the UI shows exactly which tools ran and which passages (document, page, paragraph) grounded each response.
Architecture
Chat UI
streaming render · tool usage panel · citations
LangChain agent
Gemini 2.5 Flash · AgentExecutor
Weaviate
hybrid vector + BM25 · Legal_Files collection
PostgreSQL
chat history · auth · Drizzle ORM
Grounded answer
streamed output + sources + tool steps
Technical Decisions
An agent, not a fixed RAG pipeline
A hardwired embed-retrieve-prompt chain searches on every message, even when the user is asking a follow-up that needs no retrieval. A LangChain tool-calling agent decides when to invoke legal_document_search based on conversation context — fewer wasted searches, better multi-turn behavior, and the architecture extends naturally to additional tools.
Hybrid retrieval in Weaviate
Legal language is exact-match-sensitive — section numbers, defined terms, party names — which pure vector similarity handles poorly, while pure keyword search misses paraphrased concepts. Hybrid vector + BM25 retrieval over the structured legal corpus covers both, with each result carrying document, page, and paragraph metadata so citations are precise enough to verify.
A custom streaming protocol
The /api/llm route streams a typed JSON event stream over a ReadableStream — output events for generated text, sources events for intermediate tool steps. The client parses it incrementally with TextDecoder. This is what makes the tool-observability UI possible: the frontend learns about a search the moment the agent starts it, not after the answer finishes.
Observability as a product feature
In a legal context, 'trust me' is not an answer. The ToolUsageDisplay shows every tool call with live status (pending → running → completed → error), inputs, outputs, and timing, and the Sources panel quotes the exact retrieved text. Explainability was designed in, not bolted on.
Production auth and persistence from the start
Better Auth with PostgreSQL and Drizzle: email/password with verification, GitHub OAuth, password reset, server-side session checks. Chat history is persisted to Postgres, so research sessions survive across devices and time — table stakes for a tool professionals would actually rely on.
Challenges
Hallucination and context poisoning
Early versions would blend retrieved passages with the model's prior beliefs, producing fluent answers that cited nothing. I analyzed failure modes — hallucination on sparse retrievals, context poisoning when an irrelevant chunk dominated the window — and implemented retrieval constraints and chunking strategies that measurably improved factual consistency on ambiguous queries.
Relevance vs. noise in context selection
Stuffing more chunks into context felt safer but degraded answers: the model anchored on marginal passages. Experimenting with embedding retrieval strategies and ranking methods to optimize the relevance/noise trade-off cut incorrect or low-confidence responses by about 30%.
Streaming two channels through one response
Interleaving token output and tool-step events in a single HTTP stream meant designing framing the client can parse incrementally without ever seeing a torn message. Getting buffering, flush timing, and error propagation right — an agent error mid-stream has to surface in the UI, not silently truncate the answer — took real protocol work.
Concurrency under load
The API layer had to hold up under concurrent LLM interactions and adversarial or malformed inputs. Stress-testing surfaced issues like unbounded streams on abandoned requests and prompt-shaped inputs in document text; hardening against them is what made the system stable under high load rather than just demo-stable.