How often should you re-test a chatbot?

By Ganesh Aarya, founder of Cendo Labs

Short answer: Re-test your chatbot every time something changes, and at least once a month even if nothing seems to have. 'Something' includes edits to its instructions, new prices or policies, a new service, and updates to the chatbot platform or the AI model behind it, many of which happen without you doing anything. A chatbot that passed last month can fail this month.

Why a chatbot that worked can stop working

A traditional website page says the same thing until someone edits it. An AI chatbot doesn't. Its answers depend on its instructions, the information it has, and the AI model underneath, and any of those can change.

• You change something. New prices, a new policy, an edited instruction. Even small edits can change how the bot behaves elsewhere.
• Your platform changes something. Chatbot platforms update their features and switch or upgrade the AI models they use, often without telling you in detail.
• Customers change. New tricks and ways of phrasing things spread quickly, especially after a viral chatbot story.

DPD's chatbot started swearing at customers after what the company described as a system update. Read the write-up.

A simple schedule

1. Before launch: a full test.
2. After every change to the bot's instructions, your prices, policies or services: a re-test.
3. After a platform or AI model update you're told about: a re-test.
4. Every month, even if nothing seems to have changed: a full test, to catch the updates you weren't told about.

What a re-test should cover

Re-test everything, not just what you fixed. Fixes often have side effects: a bot told to be more careful about refunds may start refusing ordinary questions too. A good re-test checks that the problems are gone and that nothing that used to work has broken.

Comparing results over time

Use the same scenarios every time so results are comparable. That way you can see whether your bot is getting better or worse, and which change caused it. Cendo Labs reports show what improved and what slipped since your last test.

Making it effortless

Monthly re-testing is easy to plan and easy to forget. Cendo Labs' Monthly Assurance plan runs a full test every month plus extra re-tests after changes, and fixes what it finds, so you don't have to remember.