LLM + Redis + Vector DB + Microservices: How the Pieces Fit Together

Modern AI apps look complicated because several specialized tools each do one job. Here's how they fit into a cohesive system.

Component Roles

• The LLM: Generates language. It has no memory between calls and no direct access to your database.
• The Vector Database: Long-term knowledge store. It retrieves relevant document chunks based on semantic meaning.
• Redis: Fast, short-term operational layer for:
  • Conversation memory (recent chat history per session)
  • Caching (including semantic caching for near-identical queries)
  • Rate limiting per user or API key
  • Queues and distributed locks for background tasks
• Microservices: Split responsibilities cleanly:
  • Auth Service: User identification and access control
  • Orchestrator Service: Manages execution flow
  • Ingestion Service: Processes documents asynchronously
  • LLM Gateway: Centralized model dispatch, fallback, and billing

One Request, End to End

1. The request hits the API gateway, and auth confirms user identity and permissions.
2. The orchestrator checks Redis for a cached answer and fetches recent chat history.
3. On a cache miss, it queries the vector database using the user's permission filters.
4. It builds the final prompt and sends it to the LLM via the LLM gateway.
5. The answer streams back to the client, and the orchestrator updates Redis.

Monolith or microservices for your AI app?