The problem
The client’s support team was buried. Twelve thousand pages of internal documentation lived across Confluence, Notion, and an aging help center. Tier-1 agents were spending the first ten minutes of every ticket searching for the right runbook. Knowledge was there, but unreachable in the moment.
A previous “AI chatbot” pilot built on a generic vector search had failed: hallucinated answers, no citations, and no way to evaluate whether it was getting better or worse. Leadership wanted AI in production, but only if it was safer than the human baseline.
What we built
A production RAG pipeline embedded directly inside Zendesk’s agent workspace.
- Ingestion: A nightly job that crawls Confluence, Notion, and the help center, chunking pages by semantic boundaries and tracking source URLs and last-modified dates.
- Retrieval: Hybrid search over Pinecone with BM25 reranking. The retrieval step returns the top eight chunks with their source links.
- Generation: Claude 3.5 Sonnet, prompted to answer only from retrieved context and to refuse if confidence is low. Every answer cites its sources inline.
- Guardrails: A second model pass checks the answer against the retrieved context for hallucination signals. Borderline answers are flagged for human review.
- Evaluation: A Langfuse-backed eval harness runs nightly on a hand-curated set of 500 real support questions. Any regression blocks deploys.
How we approached it
Two weeks of discovery to map the data, the support workflow, and the constraints. A week to ship a working spike that handled one product area end to end. Then four weeks to harden it, add the eval harness, and integrate with Zendesk.
The eval harness was the unlock. Once support leadership could see accuracy on the same 500 questions trend up week over week, they trusted the system enough to roll it out.
The outcome
After eight weeks in production:
- Average tier-1 resolution time fell 60%, from 14 minutes to 5.5
- 40% of tier-1 tickets were resolved without an agent touching them
- Eval accuracy held above 92% on the gold set
- Cost per resolved ticket dropped from $4.20 to $0.31
The client’s support team owns the eval harness, the prompts, and the runbooks. We hand off all three.