A demo is a prompt and an API call. A product is everything around it. Here's the production architecture I'd start with.
Request Path Architecture
- API Gateway: Authentication, rate limiting, request size limits.
- Chat Service: Manages sessions and orchestrates each request.
- Input Guardrails: PII detection, abuse and prompt injection screening.
- Retrieval Engine: Hybrid search (vector plus keyword) with permission filters, followed by a reranker.
- LLM Gateway: Central internal layer for model calls with timeouts, retries, model fallbacks, and cost tracking.
- Output Guardrails: Grounding check, sensitive data check, and format validation.
- Streaming Response: Stream token-by-token so users see results immediately.
Background Processing Path
An ingestion pipeline (queue-based) that parses, chunks, embeds, and indexes documents, with automated re-indexing when sources change.
Supporting Systems
- Cache: Caching for repeated questions and semantic similarity.
- Observability: Trace every request end-to-end, including retrieved chunks, prompt version, latency, token count, and cost.
- Evaluation: Fixed test sets run in CI, plus sampled review of live conversations.
- Feedback Loop: User thumbs up/down feeding back into your evaluation dataset.
- Human Handoff: Smooth transition to human support when model confidence is low.
Core Design Principles
- Every external call can fail, so define graceful degradation when it does.
- "I don't know" is a valid and preferred answer over hallucination.
- Measure quality, latency, and cost from day one.
What would you add to this list?
LinkedIn Post Snippet & Hashtags
Share this key insight directly with your LinkedIn network:
A demo is a prompt and an API call. A product is everything around it: guardrails, retrieval, caching, evaluation, and observability. Here's the architecture I'd start with. 👇
#SoftwareArchitecture #AIEngineering #LLM #RAG #SystemDesign
Tags:
#RAG
#LLM
#AIEngineering
#SoftwareArchitecture
#SystemDesign