Generative AI
Streaming Support Copilot
A live assistant that drafts grounded responses for support agents while a conversation is in progress.
Streamed
Output
Retrieval-based
Grounding
Accept / edit rate
Signal
Architecture
Hover or tap any stage to see its role in the pipeline.
Data flow
Conversation turns are summarised into rolling context, retrieval pulls relevant articles and similar resolved cases, the LLM streams a suggested reply into the agent UI, and accept/edit/reject feedback is logged for evaluation.
Problem statement
Agents were switching between knowledge bases and case history mid-conversation, slowing responses and producing inconsistent wording.
Business objective
Surface a grounded, editable draft response and relevant sources inside the agent workflow in real time.
Solution overview
A streaming assistant that combines live conversation context with retrieval over knowledge and case history, producing suggested replies the agent can accept or edit.
AI / ML approach
Rolling conversation summarisation, retrieval over knowledge plus prior resolutions, and token streaming so drafts appear as the conversation develops.
Model / LLM used
Streaming instruction-tuned LLM with retrieval grounding
Technology stack
PythonFastAPILangChainFAISSRedisAWS
Key capabilities
- Token-streamed suggestions
- Grounded in knowledge and prior resolutions
- Agent accept / edit / reject feedback
- Tone and policy constraints
Evaluation approach
- Acceptance and edit-distance tracking on suggestions
- Groundedness review on sampled drafts
- Latency to first token
Challenges
- Latency inside a live conversation
- Context window pressure
- Over-confident drafts
Solution decisions
- Rolling summarisation instead of full transcript prompting
- First-token latency treated as the primary UX metric
- Drafts always require agent confirmation before sending
Business impact
Kept agents in one workflow with consistent, source-grounded wording, while every suggestion stays under human control.