When it comes to Artificial Intelligence (AI), chatbots make up the largest category of global AI software products(1). The global conversational AI market size accounted for USD 15.5 billion in 2024 and is anticipated to reach around USD 132.86 billion by 2034, growing at a CAGR (compound annual growth rate) of 23.97% between now and then(2).    

In the not too distant past, it would be fair to say that the perception of chatbots wasn’t always as positive as we conversational designers and AI developers might have hoped. Eight or so years ago there were more FAQ (frequently asked questions) bots in play: bots that could provide information, even help surface answers hidden in a difficult to navigate website, but could not actually carry out simple tasks on behalf of the user, such as close an account or process a return.

Over the years the capabilities of virtual assistants have become much richer, deeper and more sophisticated, with task fulfilment becoming the gold standard, better conversational repair and multi-modal bots that can provide both voice and visual interactions.  

 

Good news  

Now, in 2024, not only are chatbots and voice bots viewed more favourably, but they are even preferred to a human in some instances, for example when discussing debt or late payments. The anonymity of a bot is appreciated over speaking to a person, however kind and sympathetic they might be. A survey conducted globally among retail consumers in 2023 shows some the reasons why consumers enjoy conversational commerce powered by AI: with nearly 80% of people agreeing that a virtual assistant ‘is available whenever and wherever I need it’, and around 80% like when the tool explains the reasons for recommending products(3).    

As bots are embraced by more people, there is a growing opportunity for businesses using them to engage more deeply with their customers and provide a first-rate user experience: guiding customers quickly and efficiently to information, assistance and support; whenever they want it, wherever they are.  

 

A conversational AI challenge

One of the challenges we at The CAI Company see within the conversational AI industry, and something our clients raise with us, is that bot analytics have not kept pace with the changes in complexity to CAI assistants. Teams can set up their own internal systems to monitor the many metrics available and try to use the growing amounts of data to inform decisions, but time and again it seems that the most effective way to be sure that a chat bot is behaving in the way intended is to manually review transcripts. This process is, of course, time consuming, labour intensive and impractical to do at scale. Not to mention the fact that it’s also impossible to test for every possible query and scenario or to know the extent of some of the conversational issues uncovered by manual review.  

A recent webinar from VUX World, What’s Next for AI-Powered Customer Experience in 2025, stated that quality assurance is “arguably one of the biggest challenges from a practitioner standpoint working with these models. What you need to know when you’re building with these models is: are they doing the thing that you need them to do and getting it right every single time?”     

It was highlighted that there’s a huge gap in the market when it comes to quantitative and qualitative bot analysis software. So you need to have human involvement: people to review interactions, read transcripts and make sure that the quality of the conversations is decent. However, manual review at scale is simply not feasible. It’s impossible to check every single conversation when you’re dealing with millions—if not billions—of interactions. The lack of proper tooling to support this process is a major challenge, creating a substantial gap in the industry.    

Kane Simms summed up the situation by saying “in 2024 in terms of quality assurance most companies are completely flying blind. In 2025 there needs to be some improvements made.”   

How well is your bot performing

How do you know how well your bot is doing?  

With all this in mind, just how do conscientious bot managers find a clear understanding of what’s going well – or not! – in bot conversations? Goal completion, time to resolution, sentiment analysis, fallbacks and handovers are just some examples of important indicators, and ways you might review how well your bot is doing.

But we know from talking to people in our industry that there is a huge disparity between chatbot teams. Some feel worryingly in the dark about their bots’ performance, while others have sophisticated monitoring in place looking at dozens of metrics. We know that everyone reads (or intends to read!) transcripts somewhere along the way, but when?    

We found that information to benchmark against was hard to come by, and even though the conversational AI industry is great at collaboration and information sharing, it’s not easy to find industry stats or solutions to common challenges of

  • being able to identify and quantify insights,
  • unlock trends,
  • evaluate performance,
  • monitor change,
  • prioritise development,
  • and measure the impact of changes.   

As a team, we were curious. What do CAI practitioners want to understand more clearly? Are there any aspects of understanding conversational performance that are universally agreed to be difficult or require extra effort to do? Is everyone else searching for a better solution to the time it takes to manually review conversations?  

 

Taking action 

So we decided to do something about it – and would really appreciate the thoughts of our fellow bot designers and developers. If you could give a quick 5 minutes to answer 20 brief questions on your experience monitoring bots and conversational assistants, we’ll share the final report with you.  

Please complete the survey below, or you can go to Bot Analytics Survey

(it’s just 20 questions, many are multiple choice, and shouldn’t take more than 5 minutes).

 Thanks in advance for your help, we look forward to sharing the results!

And best of luck with the Apple Watch prize draw.  

 

  Notes:

 

The prize draw to win an Apple watch was held on 20th January 2025 (congratulations to our winner, Jonathan Redsell!) by The CAI Company, Saltford House, High Street, Saltford, Bristol, BS31 3ED using a random name picker. Only emails with business domains will be entered into the draw. For the report, only anonymised aggregated results will be shared outside The CAI Company and not your email, name or any company information you provide.