How to measure success rate of a Chatbot: Metrics

Defining an objective or goal is the most important step in building a successful chatbot. However measuring the right chatbot analytics and metrics, will help you improve your chatbot performance and transform your business.

A chatbot is not something that you once set and forget. Though chatbots have been there for a while now still many companies have no proper knowledge as to how a chatbot can transform their business, the use case is only limited to replace customer service. Setting up Bot is an easy task, but the most important task is to develop proper use cases and have KPI s in place to measure the success of the bots. Working with a chatbot is a continuous process; once you have integrated chatbot with the business it’s really important to measure its effectiveness and impact on your business.


According to a report published by Deloitte, “70% of businesses are rapidly leveraging artificial intelligence to change the way they interact with customers dramatically”.


Anecdotes of success of chatbot or comparing bots with live chat, emails, phone calls are very common now but if you think you just build a chatbot and throw it away then that will not be possible, it needs continuous nurturing. Like any other strategy, the chatbot needs attention and timely upgradation.


So, in this article, we hope to identify a few key KPIs that will help your business measure the effectiveness of chatbots and become a tool to support your marketing and sales activities.
Lets first look at Quantitative metrics:

1.Chatbot Activity Volume:
It helps to evaluate a number of user interactions from the time the user first asked a question until constructive communication takes place. The activity does not mean that the conversation is only limited to “Hi” or “Hello”. For example, if a user receives a promotional message from a newspaper and he/she selects “business” news as an option only then the activity metrics will be updated with the user’s interest. These metrics also help differentiate between a new user and returning user activation events. It helps to measure two things:
• Frequency of interaction
• Number of users

2.Retention Rate:
It is not enough to keep track of active users, and special attention must be given to the retention rate. Retention rate is represented by the percentage of users who return to interact with chatbot within a specific period of time. It provides a good indication of the chatbot’s relevance and its level of acceptance among your clients.

3.Interaction Rate:
Interactions per user are key metrics that show people are actually engaging with your bot beyond a simple “Hi”. If you want to measure user engagement during conversations with your chatbot, you’ll definitely want to observe this indicator. It will allow you to measure the average number of messages exchanged per conversation. It is important to observe your user-bot back-and-forth exchanges to identify the average number of interactions that lead to successful goal completion. Ideally, a bot should be as efficient as possible.

4.Fall Back Rates:
Most chatbots have fall-back answers, programmed to suit the user if he “explores” areas that are still unknown to your robot. Usually, the virtual attendant says: “he does not know how to meet that demand”.
If users are frequently asking for something that your chatbot doesn’t know, then you should either look to fill this knowledge gap or make it clear that the chatbot can’t provide this value. If we divide the number of times the chatbot has had to use a fall back response by the total number of messages, we will have the rate of confusion.
Confusion rate = number of fallback answers / total answers offered.

4.Goal Completion rate:
GCR enables you to measure the success rate of a given action performed through your chatbot, for example, clicking on a CTA button or link, filling out a form, proceeding to make a purchase, etc. It doesn’t look at how many users engage with your bot. It tracks the percentage of those users who, in fact, reach the goal you designed the bot to accomplish. It is one of the few KPIs that have a direct impact on the bottom line.

5.Session Length:
This KPI is important to assess the effectiveness of the chatbot regarding its ability to carry out a meaningful conversation with users. While the ideal session length will vary based on case to case and the context of the conversation, short session lengths are often indicative of some form of failure unless the chatbot can resolve user inquiries almost immediately. It is necessary to use some sort of timeout, so that session lengths are not inflated by idle periods. This will indicate the effectiveness of your bot and its ability to carry out a meaningful conversation.

6.Bounce Rate
The Bounce Rate corresponds to the volume of user sessions that fail to result in the intended “specialized” use of your chatbot. An increased rate indicates that the bot is not being consulted on subjects that are relevant to its area of expertise. This indicates that it’s time to update the content and rethink its placement in the customer experience, or both. This indicator should be observed closely.

Qualitative Measures:
1.User Satisfaction:
User satisfaction metrics can be measured through exit surveys. Customers or the people engaging with the chatbot can rate their experience with the bot which can help companies achieve further product excellence. This can be implemented as a binary variable such as “did the bot meet your requirement? – Yes, or No” or we can create more complex evaluation forms to rank and provide points for each different category, we can also include NPS in the exit form to check loyalty among our customers. This metric would capture the overall effectiveness of the bot from the user experience point directly provided by the user.

2.Self-Service Rate
This metrics represents the number of users whose queries were resolved through the responses given by your chatbot, without having to call Customer Service. It is calculated based on the percentage of sessions that were successfully completed through an interaction with your bot without being redirected to a live operator. In the process, it enables you to evaluate the level of client satisfaction. This is the equivalent of a call center’s First Call Resolution (FCR) rate, the percentage of problems that are resolved through a single phone call. This indicator is very important for analyzing the ROI of your chatbot project.

In addition to the mentioned metrics, measuring bot’s impact on your business directly correlates with the use-case, business vertical that it is been used for. Defining an objective or goal is the most important step in building a successful chatbot. However measuring the right chatbot analytics and metrics, will help you improve your chatbot performance and transform your business.