We have all had both good and bad customer care experiences. The way the agent converses often goes a long way in customer having a satisfying or frustrating experience. Even when an agent is unable to provide a solution, an astute agent can make the customer feel appreciated and drive pleasant interaction, rather than frustrating the customer. Using a data-driven approach, we wanted to understand what attributes of an agent’s persona impact user satisfaction, if any. To do so, we conducted an empirical study using customer care conversations from six brands.

In a nutshell, we found that agents who show conscientiousness, agreeableness and emotional range, while avoiding characteristics of anxiety, self-consciousness and immoderation, are more likely to provide a satisfying user experience. Further, personas that indicate that the agent values self-enhancement and hedonism, are detrimental to user satisfaction.

Given we now know what personas elicit positive customer service experiences, what does it mean for you? If you are in business of providing customer care, it is important to ensure that your agents, besides providing the right solution, are also projecting that persona. This will increase the likelihood that your customers are getting a positive experience. If you are considering bringing in automated agents (bots/conversations) to help alleviate human labor, it is important to ensure that they don a persona that is correlated to positive user experiences. This study will provide you pointers on what traits make up “good” personas and also how you might assess the persona.

Read on to find how we collected and prepared the data, as well as details on the analysis.

Data Preparation

Now-a-days many brands are providing customer service on social platforms such as Facebook and Twitter. For example, see below for a sample Twitter conversation between a user and a telecom company agent.
useimage1
We did an analysis of such customer-care conversations happening on Twitter to answer the question does agent personality impact whether customers will be satisfied with the interaction?

We collected Twitter conversations from six well-known US brands. Three of these are large retail/e-commerce business, two of these are telecom/broadband providers and one is a large courier delivery company. We cleaned up the conversations to only include dialog that satisfied following constraints:

  • Conversation between one agent and customer only. Some twitter conversations have multiple participants. The conversation has the following flow – customer->agent->customer and so on.
  • Personalized identifier on responses (^ThomasC here) that allow us to extract who the specific agent was that handled the conversation.

Next, we needed to identify conversations where customers were satisfied versus dissatisfied. Our initial thought was to mark a conversation that received a like, as a positive interaction experience. However, there were two challenges with this approach – Very few customers came back to like a conversation even though they were satisfied, and it is not necessary that the user who faced the problem gave the like. After manually inspecting the data, we realized that in most cases with a satisfied interaction, the customer was grateful in the last or penultimate utterance. Hence, we used the following heuristic:

  • If the last utterance in the conversation is by customer and (s)he expresses gratitude, then interaction was positive.
  • If last utterance in the conversation is by agent, then we check the penultimate utterance, which was by customer for gratitude.

After applying these criteria, we had a filtered list of 21825 conversations across the 6 brands, with 279 unique agents. Out of those, 50.4% of conversations (i.e. 10997) were identified as conversations where interaction experience was positive. Below, we show the distribution of conversations where users were satisfied versus dissatisfied, across all six brands.

Analysis

Our hypothesis was that the persona of an agent does have an impact on user satisfaction. In other words, agents with certain kinds of personas will produce more conversations where users are satisfied. In order to detect the persona of each agent, we used Watson Personality Insights. Prior studies have shown that words an individual uses directly correlate to the individual’s personality and impact how (s)he will be perceived. Watson Personality Insights builds upon this science to provide a model, which takes text written or spoken by an individual as input, and provides a personality map based on personality traits such as Big5 and values as output.

  • “Big5” captures aspects of an individual’s personality that show how the individual engages with the world.
  • “Values” captures aspects of the personality that relate to the motivational factors that influence the individual’s engagement with the world.

Proving the Hypothesis

We designed the following evaluation to prove our hypothesis. For each agent in our data, we calculated the percentage of satisfying versus dissatisfying conversations (P). This is the dependent variable in our analysis. We then took all the utterances for each agent and used Watson Personality Insights to calculate each personality trait. The personality traits for each agent were our independent variables (traits).

We started by calculating the correlation between the variable P (proxy for user satisfaction) and traits. In the table below, we list the traits that were significantly correlated to the variable P ( < 0.05 p-value) and had significant correlation (< -0.5 or > 0.5).

Trait

Type

Correlation

Conscientiousness

Big5

0.7198312

Agreeableness

Big5

0.6805151

Emotional Range

Big5

0.6454228

Achievement striving/driven

Facet of Conscientiousness

0.6587996

Sincerity

Facet of Agreeableness

0.6870826

Modesty

Facet of Agreeableness

0.6182671

Energy /Enthusiasm

Facet of Extraversion

0.6830167

Self Consciousness

Facet of Emotional Range

-0.6684006

Anxiety

Facet of Emotional Range

-0.6646196

Immoderation

Facet of Emotional Range

-0.6616996

Self enhancement

Value

-0.6163081

Hedonism

Value

-0.6720199

 

As expected, user satisfaction is positively correlated to agent personality traits such as conscientiousness and agreeableness. Conscientiousness is an agent’s tendency to act in a thoughtful and organized way. Agreeableness captures their tendency to be compassionate and thoughtful. Further, user satisfaction is also positively correlated to sub-traits for these dimensions such as sincerity and modesty. Extraversion is another Big5 trait that measures a person’s tendency to enjoy other’s company. While Extraversion itself does not come up as a significant distinguishing trait, one of its sub-facets, Enthusiasm, is positively correlated.

Emotional Range came up as another Big5 attribute that is significantly positively correlated to user-satisfaction. This trait measures the extent to which a person’s emotions are sensitive to the individual’s environment. This proves our intuition from real engagements i.e. if agents tune their messaging based on a user’s tone, they are more likely to be perceived as effective. However, while showing emotions is important for an agent, it is also important not to show certain facets such as anxiety, immoderation and self-consciousness. All of these are sub-traits for emotional range and as we can see from the table they are significantly negatively correlated.

Two other traits that are negatively correlated to user-satisfaction are self-enhancement and hedonism. These two traits come from the value psychology model, which attempts to capture motivational values that guide how an individual interacts with the world. Self-enhancement indicates that the individual’s actions are guided by personal success and hedonism indicates that the individual seeks personal gratification.

Further Analysis

Further, we then trained a linear regression model (LM) that tried to predict the percentage of successful conversations based on personality traits. In addition, we included the brand as a feature to account for the influence of domain in the model. With these parameters, the LM model had a correlation of 0.81. The agents were split into 2 classes based on percentage of conversations they participated in where the user was satisfied, and then a logistic regression model was trained to predict the 2 classes (see the figure below for distribution of agents across these classes). The weighted precision was 0.69 and recall was 0.68.
useimage3

The fact that we were able to train models with good predictive power further confirms the hypothesis that user satisfaction is indeed related to the persona of an agent.

Summary

In conclusion, in this study we analyzed Twitter conversations to determine how words spoken by an agent can be used to derive personality, which in turn is significantly correlated to customer satisfaction. If you’ve reached this point in reading this article, the key take-away is that customer care conversations are not just about content – expression is also important. It is critical not ignore how your agents, be it humans or bots, are expressing themselves in front of your customers!

Please feel free to contact us, if you want to assess the personality of your bot and/or agents and infer customer satisfaction (using above data as baseline).

14 comments on"Personality of your agent matters – An empirical study on Twitter conversations"

  1. Great read!! Curious about how the role of industry plays in the purpose of an agent – comparing where content potentially is stronger driver v/s personality. We’ll be taking this experiement up in our learning-focused bot, where content might have to use personality as a driver to “understanding”…

    Thanks for this amazing post!

    • Niraj – glad you liked the article. Would be really interested in knowing about your experiment once you are done with it. Please do share.
      Industry definitely plays a part in purpose of agent. However, with our study what we were able to establish was that independent of industry, personality of agent does impact customer satisfaction, and there are some traits that exhibit strong correlation to satisfaction.

  2. Luiz Siqueira November 09, 2016

    In my opinion, the part of verbal comunication of neurolinguistic programming would be very helpful in this study. Initially, to make the first contact with the custumer, you need to use words with the same weight of bad feelings as the one put by the custumer. In this way you empathize with his suffering/feelings. After that, you start building the solution, every time weighing the good or bad feelings in each expression of the customer, trying to mirror it. Improving the good humor just after the customer has already improved.

  3. Excellent. Thank you.

  4. 🙂 OK !

Join The Discussion

Your email address will not be published. Required fields are marked *