A weekly review process for improving a voice assistant

Improving a voice assistant is an operating process. The goal is to understand a recurring failure, make a focused change and check the result. More conversations alone do not prove that the assistant is learning or that quality is increasing.

Review evidence by failure type

Separate wrong business facts, misunderstood details, awkward turn handling, unsupported workflow behavior, failed actions and telephone routing problems. They have different remedies. Rewriting a prompt will not fix an expired calendar credential or an unanswered human destination.

Keep the call's selected version, channel and action evidence. A transcript is useful for wording, but caller-heard audio and telephone behavior need the relevant session evidence.

Choose one change

Prioritize a failure that affects a meaningful task. Describe the expected behavior and the smallest change needed. For a repeated question, remove conflicting instructions. For an unknown policy, ask the business owner for an approved answer. For a tool failure, investigate the integration rather than inventing a reassuring response.

Burki learning suggestions can support this review where enabled. Treat generated suggestions as proposals; candidates require evaluation and approval before production use.

Maintain a fixed scenario set

Keep examples covering the main task, a correction, an unknown answer, an interruption and a failed action. Add a regression scenario when a real failure is understood. Preserve examples that already work so an improvement in one branch does not obscure a regression elsewhere.

Text evaluations are useful for some instruction and action checks. They do not replace browser or telephone acceptance when audio or routing changes.

Release and inspect

Confirm which draft or published version the actual runtime will use. Retain the prior accepted version and the process for recovery. Review the targeted metric after release, along with adjacent errors and total cost.

Do not count every completed call as success. A correct callback request, verified booking, accepted transfer, evaluator assessment and customer rating are different evidence types. Label them accordingly.

A regular review can be small and focused. Keep the decision record, including rejected proposals and unresolved dependencies, so the next review starts from what was learned rather than another broad rewrite. See the learning release checklist.