GUEST:
Researchers estimate we will speak to chatbots more than we speak to our spouses by 2020. Obviously, companies that implement chatbots are doing something right. However, businesses still have a hard time determining whether or not their bots are up to snuff. While there are plenty of effective chatbots on the market, there are also many that don’t quite meet consumers’ needs. So how do you measure the success of your chatbot?
This is the dilemma facing an increasing number of companies that use chatbots as part of their customer experience. 80 percent of businesses want to implement a chatbot by 2020, but many still face the challenge of gauging the efficacy of the technology.
Google’s Chatbot Analytics platform recently opened up to all, but it is still necessary for businesses to develop and understand their own chatbot success metrics to effectively use the platform.
The process of defining the best KPIs for your company’s bot will depend on your business goals and the functions you want your bot to perform.
Here are seven metrics of success you can use to identify opportunities for improvement in your company’s chatbot.
Revenue growth
The first question any prospective investor wants to know about a company is whether or not it makes money. Therefore, the best indicator of a chatbot’s value is its financial benefit.
There are many ways to evaluate a bot’s impact on revenue – the best one for your bot will depend on its purpose. Another interesting wrinkle is that your chatbot can have a knock-on effect on a number of areas.
For example, you can measure a customer service bot’s profitability growth by the amount of money it saves the company compared to maintaining a customer service team 24/7. But you will want to take the bot’s impact on customer service into account. If self-service rates are higher and clients are more satisfied, that will result in repeat customers and higher online sales, thus impacting top-line revenue growth.
Self-service rate
Nirvana comes for businesses the moment a user gets exactly what they want from the chatbot without any human input.
If your chatbot’s goal is to change a user’s password, you would measure success by the percentage of user interactions that end with this as a result.
The self-service rate closely correlates the cost savings aspect of revenue growth – in other words, how much money did your chatbot save?
Satisfaction rate
What better way to find out exactly how well your chatbot is doing than to ask the very people who use it?
Your chatbot can help you determine this metric by asking the key question for the Net Promoter Score – “On a scale of 1-10 how likely is it that you would recommend our chatbot to a friend/colleague?” As a lead indicator of growth, the NPS provides a crucial foundation for understanding your chatbot’s customer experience performance.
Activation rate
At this point, it’s worth reflecting on AARRR and its importance in measuring the success of your business.
The activation rate in the context of a chatbot refers to when a user responds to its initial message with a question or an answer which is relevant to your business goals.
For example, a chatbot designed to provide you with weather updates would receive an activation rate when you enter your location – thus allowing the bot to provide you with the information.
How can this KPI help? If for some reason people were not responding when the weather chatbot first reached out to them, the botmaster would be able to tinker with it to enable a more satisfactory outcome.
Confusion triggers
Unfortunately, even bots with the most robust natural language processing are unable to understand everything a user says.
These errors are a useful indicator for measuring whether or not you need to improve your chatbot’s matching.
Bear in mind there are three different triggers, each of which necessitates its own type of response.
There is first the simple confusion from the bot if it cannot understand a comment. A basic “Sorry, I didn’t understand that. Can you ask again in a different way?” response would suffice.
Second is if the user sends a number of messages which are outside the remit of your chatbot. After a couple of attempts, it would be worth programming your bot to relay a message that reminds the user of its exact purpose.
The final trigger is if the bot forces a user to speak to a customer service agent after the interaction. Each of these will tell you something different about how your chat agent is performing.
Retention rate
Once again referring to AARRR, the retention rate represents the percentage of users who return to the chatbot over a specified period of time.
This timespan would vary between the bots depending on their purposes. For example, a fitness chatbot would require daily interaction and would benefit from analyzing its 1-day retention.
Artificial intelligence/machine learning rate
How strong is the AI in your chatbot? You can use the percentage of user questions that are correctly understood to measure this.
Which leads us the million, if not billion dollar question — can my chatbot learn independently?
Chatbots with machine learning can measure progress by comparing the improvement in self-service rate over a period of time without human intervention.
An agent with robust machine learning will be able to continually run its own gap analysis to highlight potential areas of improvement.
The demand for chatbots among Millennials is clear. Consumers are asking for simple and effective customer service, but not every chatbot is capable of delivering on this promise without a few tweaks. In a market that is becoming increasingly crowded, these KPIs can help you keep your chatbot one step ahead of the pack.
Jordi Torras is CEO and founder of Inbenta, an artificial intelligence technology company.
