Context-Aware Support Chatbot with RAG

Problem

A generic LLM chatbot gave confident but incorrect answers about product-specific behavior, and the support team didn't trust it enough to deploy. It needed to answer only from the company's own docs, tickets, and FAQs — and cite where each answer came from.

Challenges

• Keeping answers grounded in real documentation instead of model hallucination
• Serving streamed responses with low first-token latency over a persistent connection
• Controlling inference and embedding cost as document volume and traffic grew
• Routing qualified leads to a human in real time without adding conversational latency

Architecture

Incoming documents are chunked, embedded, and stored in a vector index. On each user message, the backend retrieves the top-k relevant chunks, assembles a grounded prompt, and streams the model's response token-by-token to the client over a WebSocket connection rather than a REST round-trip, cutting perceived latency significantly. A lightweight classifier runs in parallel to score buying intent and pushes qualified conversations to Slack instantly.

Database Design

Conversation history and document metadata live in Postgres, while vector embeddings are stored in a dedicated vector index optimized for approximate nearest-neighbor search. Embeddings are versioned so documentation updates can be re-indexed without downtime, with the previous version serving traffic until the new index is warm.

Scalability

Embedding generation is cached and deduplicated so re-ingesting an unchanged document costs nothing. Rate limiting and request batching keep inference spend predictable under bursty traffic, and the retrieval layer is decoupled from the generation layer so either can scale independently.