We have all had both good and bad customer care experiences. The way the agent converses often goes a long way in customer having a satisfying or frustrating experience. Even when an agent is unable to provide a solution, an astute agent can make the customer feel appreciated and drive pleasant interaction, rather than frustrating the customer. Using a data-driven approach, we wanted to understand what attributes of an agent’s persona impact user satisfaction, if any. To do so, we conducted an empirical study using customer care conversations from six brands.
In a nutshell, we found that agents who show conscientiousness, agreeableness and emotional range, while avoiding characteristics of anxiety, self-consciousness and immoderation, are more likely to provide a satisfying user experience. Further, personas that indicate that the agent values self-enhancement and hedonism, are detrimental to user satisfaction.
Given we now know what personas elicit positive customer service experiences, what does it mean for you? If you are in business of providing customer care, it is important to ensure that your agents, besides providing the right solution, are also projecting that persona. This will increase the likelihood that your customers are getting a positive experience. If you are considering bringing in automated agents (bots/conversations) to help alleviate human labor, it is important to ensure that they don a persona that is correlated to positive user experiences. This study will provide you pointers on what traits make up “good” personas and also how you might assess the persona.
Read on to find how we collected and prepared the data, as well as details on the analysis.
Now-a-days many brands are providing customer service on social platforms such as Facebook and Twitter. For example, see below for a sample Twitter conversation between a user and a telecom company agent.
We did an analysis of such customer-care conversations happening on Twitter to answer the question does agent personality impact whether customers will be satisfied with the interaction?
We collected Twitter conversations from six well-known US brands. Three of these are large retail/e-commerce business, two of these are telecom/broadband providers and one is a large courier delivery company. We cleaned up the conversations to only include dialog that satisfied following constraints:
- Conversation between one agent and customer only. Some twitter conversations have multiple participants. The conversation has the following flow – customer->agent->customer and so on.
- Personalized identifier on responses (^ThomasC here) that allow us to extract who the specific agent was that handled the conversation.
Next, we needed to identify conversations where customers were satisfied versus dissatisfied. Our initial thought was to mark a conversation that received a like, as a positive interaction experience. However, there were two challenges with this approach – Very few customers came back to like a conversation even though they were satisfied, and it is not necessary that the user who faced the problem gave the like. After manually inspecting the data, we realized that in most cases with a satisfied interaction, the customer was grateful in the last or penultimate utterance. Hence, we used the following heuristic:
- If the last utterance in the conversation is by customer and (s)he expresses gratitude, then interaction was positive.
- If last utterance in the conversation is by agent, then we check the penultimate utterance, which was by customer for gratitude.
After applying these criteria, we had a filtered list of 21825 conversations across the 6 brands, with 279 unique agents. Out of those, 50.4% of conversations (i.e. 10997) were identified as conversations where interaction experience was positive. Below, we show the distribution of conversations where users were satisfied versus dissatisfied, across all six brands.
Our hypothesis was that the persona of an agent does have an impact on user satisfaction. In other words, agents with certain kinds of personas will produce more conversations where users are satisfied. In order to detect the persona of each agent, we used Watson Personality Insights. Prior studies have shown that words an individual uses directly correlate to the individual’s personality and impact how (s)he will be perceived. Watson Personality Insights builds upon this science to provide a model, which takes text written or spoken by an individual as input, and provides a personality map based on personality traits such as Big5 and values as output.
- “Big5” captures aspects of an individual’s personality that show how the individual engages with the world.
- “Values” captures aspects of the personality that relate to the motivational factors that influence the individual’s engagement with the world.
Proving the Hypothesis
We designed the following evaluation to prove our hypothesis. For each agent in our data, we calculated the percentage of satisfying versus dissatisfying conversations (P). This is the dependent variable in our analysis. We then took all the utterances for each agent and used Watson Personality Insights to calculate each personality trait. The personality traits for each agent were our independent variables (traits).
We started by calculating the correlation between the variable P (proxy for user satisfaction) and traits. In the table below, we list the traits that were significantly correlated to the variable P ( < 0.05 p-value) and had significant correlation (< -0.5 or > 0.5).
Facet of Conscientiousness
Facet of Agreeableness
Facet of Agreeableness
Facet of Extraversion
Facet of Emotional Range
Facet of Emotional Range
Facet of Emotional Range
As expected, user satisfaction is positively correlated to agent personality traits such as conscientiousness and agreeableness. Conscientiousness is an agent’s tendency to act in a thoughtful and organized way. Agreeableness captures their tendency to be compassionate and thoughtful. Further, user satisfaction is also positively correlated to sub-traits for these dimensions such as sincerity and modesty. Extraversion is another Big5 trait that measures a person’s tendency to enjoy other’s company. While Extraversion itself does not come up as a significant distinguishing trait, one of its sub-facets, Enthusiasm, is positively correlated.
Emotional Range came up as another Big5 attribute that is significantly positively correlated to user-satisfaction. This trait measures the extent to which a person’s emotions are sensitive to the individual’s environment. This proves our intuition from real engagements i.e. if agents tune their messaging based on a user’s tone, they are more likely to be perceived as effective. However, while showing emotions is important for an agent, it is also important not to show certain facets such as anxiety, immoderation and self-consciousness. All of these are sub-traits for emotional range and as we can see from the table they are significantly negatively correlated.
Two other traits that are negatively correlated to user-satisfaction are self-enhancement and hedonism. These two traits come from the value psychology model, which attempts to capture motivational values that guide how an individual interacts with the world. Self-enhancement indicates that the individual’s actions are guided by personal success and hedonism indicates that the individual seeks personal gratification.
Further, we then trained a linear regression model (LM) that tried to predict the percentage of successful conversations based on personality traits. In addition, we included the brand as a feature to account for the influence of domain in the model. With these parameters, the LM model had a correlation of 0.81. The agents were split into 2 classes based on percentage of conversations they participated in where the user was satisfied, and then a logistic regression model was trained to predict the 2 classes (see the figure below for distribution of agents across these classes). The weighted precision was 0.69 and recall was 0.68.
The fact that we were able to train models with good predictive power further confirms the hypothesis that user satisfaction is indeed related to the persona of an agent.
In conclusion, in this study we analyzed Twitter conversations to determine how words spoken by an agent can be used to derive personality, which in turn is significantly correlated to customer satisfaction. If you’ve reached this point in reading this article, the key take-away is that customer care conversations are not just about content – expression is also important. It is critical not ignore how your agents, be it humans or bots, are expressing themselves in front of your customers!
Please feel free to contact us, if you want to assess the personality of your bot and/or agents and infer customer satisfaction (using above data as baseline).