Anyone in the business of monitoring conversational assistants or bots will be used to keeping a keen eye on performance metrics. Whether it’s being alert to outages or handover spikes in the moment or producing a report on past performance, the data is the key to refining and improving the conversational experience and business outcomes.
But when we look back over our bots’ stats over the past days, weeks and months, how often do we see sudden changes that aren’t easily explained by the other metrics in our dashboard?
Pipe and deerstalker at hand, bot managers become data detectives to get to the bottom of the behaviour.
There are a few ‘usual suspects’ that can significantly impact the volume and nature of customer inquiries. These indicators are well worth tracking as part of a comprehensive analytics strategy and to help shortcut some of that sleuthing. Tracked pro-actively, they may even help swerve some of the issues in the first place.
1. The weather
Image generated with Microsoft Copilot
When storms, cold snaps or heatwaves wreak havoc, people increasingly turn to companies for support, whether it’s for advice, emergency supplies, insurance claims, or utility disruptions. This uptick in demand often overwhelms traditional customer service channels, leading to longer wait times and heightened frustration among consumers.
Weather changes can also change the meaning behind user utterances.
“Is it safe to travel?” was a frequent user query for a travel company during the pandemic, the reply to which was an outline of COVID measures being taken to make travel as safe as possible.
The same query, “Is it safe to travel?” asked during a period of intense storms to the same company had a different, more urgent meaning. In this case, customers were still getting responses about COVID, which led to a spike in negative user sentiment, handovers and an unsatisfactory user experience.
To adeptly manage weather induced flux, optimizing chatbot responses becomes crucial. A tailored approach involves programming chatbots to recognise weather-related keywords or phrases, triggering context-specific assistance. If the use case is heavily weather dependent (e.g. travel) integrating real-time weather data APIs allows chatbots to offer proactive advice or solutions or remind customers of potential service delays.
If you are monitoring bot data retrospectively, tracking the weather alongside your usual metrics can be an extremely useful troubleshooting tool in your armory.
2. Business updates
A change in a business’s offering can impact automated assistance in a number of ways. A new discount, a freebie, a price increase or a feature sunset can all prompt people to seek extra support or vent.
In the worst cases, one team may have made a change without letting other teams know, leading to a scramble to get comms up to date, as the handovers to agents roll in.
Tracking business updates alongside other analytics can prompt better communication between teams, or at least encourage chatbot teams to proactively chase for updates rather than be passively impacted by them.
At the very least, keeping a record of the dates that changes are implemented can be useful for reporting and analysis.
Working on content for a customer service chatbot in a fast-changing industry, our consultants created a cross-department ‘knowledge sharing’ meeting once a week – where we could all align on when and what needed to change. We could review planned email or web content and make sure the bot carried the same information.
We also recommend communicating regularly with the agents – they get a feel for emerging topics, issues with the bot, flows that could be fine tuned. They’ll also spot out of date information because they may need to contradict the bot on occasion, which leads us to the next topic…
3. Copy changes
Conversation designers know that altering content can have far-reaching consequences on customer experience. The introduction of new phrases or modifications in tone can disrupt the familiar flow of conversation, leading to confusion or misunderstanding.
When customers interact with a chatbot, they seek quick and accurate information. However, if the messaging is constantly changing, it can disrupt the learning curve and familiarity that users develop over time with the chatbot’s interface and responses.
In cases where there is a dedicated conversational AI designer or developer on the team, careful thought is given to copy changes. However, it is still easy to accidentally introduce copy that has an unwanted or unimagined intent.
Emojis are a good example. There is a generational difference in the use of the 😊 emoji, with millennials and older generations using it to indicate friendliness or pleasure. In younger generations, the same emoji can also mean sarcasm, a forced smile, indicating the opposite. Introducing a sarcastic smiley into copy might lead to subtle changes in brand analytics, such as agreement with research statements like “Brand X understands me”.
Moreover, frequent copy alterations may introduce errors or inconsistencies in information, leading to misunderstandings about services or products. As a result, customers might receive incorrect guidance or answers that do not align with their expectations or previous interactions. Such experiences can lead to higher dissatisfaction or handovers as people seek to clarify information.
In large teams, the people monitoring the analytics or in charge of regression testing may be separate from the copywriting or design team. In this instance keeping records of copy changes and publishing dates can substantially reduce the workload across the board.
4. Warehouse issues
Image generated with Microsoft Copilot
Warehouse issues can arise in retail due to various reasons such as inventory management problems, supply chain disruptions, or logistical errors. These issues can lead to stock shortages, delayed deliveries, or incorrect orders, directly impacting the customer experience.
As a result, customers may reach out to contact centres seeking information or resolution. Proactively monitoring warehouse issues and keeping customers informed can help mitigate the impact on customer satisfaction and reduce the strain on contact centres. If you can greet an identified customer with an apology about their impacted delivery, then you may be able to allay concerns and turn a potentially negative situation into a positive customer engagement.
Keeping a record of warehouse issues can also help explain spikes and recovery times in retrospective reporting.
5. Delivery failures
Anyone who has ever had a late delivery knows how frustrating it can be, particularly if it’s a product ordered for a deadline such as a birthday. Ecommerce automated assistants should be well set up for handing delivery and fulfilment queries, but given the complexity and unpredictability of delivery logistics, things can and do go wrong.
The API to the delivery company may fail, there might be a comms outage, a traffic delay may go unreported. Whatever the issue, keeping track of it can explain away anomalies after the event. Tracking it proactively can help messaging to appease frustrated customers and keep them informed.
At the CAI Company, we worked with a client to implement a comprehensive “delivery exceptions” handling strategy. When a (fresh food) delivery is due to arrive outside the expected time due to delays, the weather, or warehouse issues, a proactive message is sent to the customer to inform them, and in some cases offer a credit. Taking this forward-thinking approach has led to a healthy drop in contact centre calls and an increase in positive customer sentiment.
6. Product or recipe changes
As with any business update, product or recipe changes can be the reason behind customers getting in touch or registering a change in sentiment.
Keeping track of these can be both helpful in pinpointing why there is an upsurge in queries or even better, proactively providing information to people so that they are informed.
7. The news
The news can also lead to surges in enquiries about specific products, services or advice. Positive news can be an unforeseen blessing for a business, such as a glowing segment about a product on a news report. Equally, a damning expose, or a negative association can bring in worried or angry queries.
When news breaks, a chatbot or VA that is promptly updated with the company’s stance or action plan shows great communication and helps avoid misinformation. Tracking the news can help teams proactively adjust their automated assistants to manage the situation. A responsive chatbot frees up human resources that may arise from the news event, so that every customer query can be dealt with.
Working on a customer service bot for a fast fashion company, we’ve seen a surge of interest in an item or style when certain celebs or royal family members are in the news wearing something that people love. Awareness of this means that you can account for inquiries about specific items and point customers in the direction of something they might want to buy, update it as an item to push in bot marketing assets (multimodal) or include it in personalised shopping recommendations.
Equally, keeping a record of when certain news items broke can help with interpreting long term analytics to understand anomalies or key milestones.
Conclusion
There are many benefits to monitoring external events and other non-metric factors. Proactive monitoring can help provide better service and improve customer satisfaction.
In addition, by keeping great records of all of the above events, the shelf-life of this knowledge is extended to anyone involved in reporting and analytics, long after the news itself has faded from memory.
