Category Archives: Metrics

Metrics In The Eye Of The Beholder


A previous post on this blog discussed several reasons for not using case life as a goal for first line support engineers. I was reminded of that post, and the reasons for writing it, while hearing someone state that organizations should “only measure outcomes, not activities”. It’s easy to agree with this statement, after all, outcomes are the only thing that matters in business, right?

Let’s take a moment and try to understand how these outcomes are realized. The obvious question to start with is “what’s the organization’s deliverable to the company?” Over the years, through my own and others’ experiences, I have seen a variety of answers to this question as it concerns enterprise support organizations. Goals ranged from operational objectives through customer satisfaction all the way to maintenance renewal rates and revenues. Surely the organizational deliverable is an important metric to track and report on, but is it the only thing that matters? And more importantly, does it apply to every member of the organization?

Similar to case life mentioned earlier, or to the goals listed above, outcomes are trailing indicators. They, almost by definition, represent an aggregation of a variety of activities and inputs. Making any improvement to the operation requires us to understand the individual components our deliverables are composed of and build a system of metrics that allows us to understand the way we perform on each, identify weaknesses and measure improvements. We should also measure inputs, from the number of cases to the growth in the number of customers and track the way those inputs influence our performance, all the way to delivery on our goal. In essence, we should design a metrics structure that breaks down the goals of the organization into departmental goals supporting it, and into the activities that these goals are composed of

We should also keep in mind that metrics can serve different roles depending on the audience. One group’s outcome is another’s input. Activities represent a major investment for the organization and ensuring their efficiency and effectiveness is a primary concern for anybody running a process driven operation. Any measurement system we design will have to take into account the organizational deliverables for the short and longer term future, the investments it has to make into producing those deliverables, the inputs it requires in order to do that, and the way these measurements apply to every function within the organization. On that in our next post

No Free Lunches, or How To Reduce Case Life

Clock and calendar

A few weeks ago I had an interesting discussion with a senior executive in an enterprise technology company. That person started his career as a support engineer, and at that time one of his main measurable objectives was case life – the average amount of elapsed time between opening and closing of cases in his ownership. His impression, at the time and even now, was that the only way in which he could impact that goal was to close cases prematurely. Presumably this was not the intended result of this goal.

That discussion opened the door for two questions: can case life be a valuable metric for managing support organizations, and if so, how can it be used in a meaningful and productive manner?

Case life, when we think of it, is an aggregate metric combining a variety of activities taking place throughout the case life, and many times more than once. Each activity has, therefore, a double impact on case life, first by its elapsed time, and second through repetition which, in turn, offers opportunities for improvement exist both by reducing the time each element takes, and by eliminating as many repetitions as possible.

When we examine the most common elements of a case life and analyze the influences on elapsed time and drivers of repetition we find that most of the time spent on a case is comprised of waiting for one of three activities to take place:

  • Producing problem documentation
  • Investigating the problem
  • Implementing a fix and verifying its effectiveness

We also know that for each of these activities we can reduce the time they take to perform, and the number of iterations required to bring them to completion. For example, producing the documentation required to investigate a problem will be much faster and require fewer repetitions when the documentation is generated automatically when the problem occurs for the first time. Having to wait for the problem to recur and then rely on verbal instructions from the support engineer will invariably create errors and require repeated attempts to get right. But, high impact changes require higher investment in tools or in the product.

If we map the options for reducing case life according to their complexity and anticipated impact, we’ll see something similar to the following chart, where the activities that impact case life the most are also those that requires the largest investment and involve higher levels of the organization:

Case Life Elements2

Taking all this into account, we can conclude that the ability of the support engineer to influence case life is relatively small and depends on their ability to manage the customer interaction efficiently and effectively. But, the bigger impact will be driven by higher investment and greater organizational focus on the various drivers.

Having reviewed this we can now answer the questions we initially posed:

  1. Is case life a valuable metric for the support organization?
  2. Should it be used to measure the support engineer?

The answer to the first question is a resounding yes. Case life, the way it develops over time and its composition gives us unique insight into the performance of the organization, helps us gauge the success of past actions and outline future development plans. On the other hand, measuring support engineers on case life is probably unproductive, and is likely to drive the behaviors discussed in the first paragraph. It is better to measure engineers on the specific element each needs to improve, and especially those they can directly influence

Basic Concepts of Six Sigma

Six Sigma Blue Stripes Horizontal

I recently came across a post titled Using Six Sigma to Improve Customer Experience and Service. As it touches on several topics close to my heart I read it with great anticipation and sadly even greater disappointment.

Since I feel the author misses the basic concepts of six sigma and the many improvement opportunities it offers support and services organizations I decided to attempt and correct some of the misconceptions and offer a different perspective to some of the points discussed.

First and most obviously missing is the fact that six-sigma is an iterative improvement process. DMAIC, therefore, is circular as shown in the chart attached to the original post rather than being a one time linear activity, at the end of which is the six-sigma nirvana stage of 3.4 defects per million.

Second, the statistical concepts behind six-sigma are never mentioned, nor is the meaning of 3.4 defects per million opportunities (known as DPMO). Here is a brief explanation. Sigma is a measure of spread of a normally distributed population, and measures the number of standard deviations fitting within a certain range. That range, in turn, is the acceptable performance range, so performance within that range is considered good, and outside of it is bad. To achieve one sigma, for example, about 68% of the population must be within the acceptable performance range, for two sigma, 95.5%, and so on, as shown below:

Sigma ValuePercentage Within Range

The last item I’d like to touch on is the need to make quality improvement, six sigma or otherwise, an inclusive initiative. It should ensure each and every employee understands the improvement process and expected results, and is able to make contributions.

Obviously, it is not possible to increase sigma levels by going through the phases of DMAIC without transforming service delivery, and it is extremely unlikely that a single journey through DMAIC will take you into six sigma performance levels. However, the benefits of improvement, even from one sigma to two sigma are enormous. So, the value in six sigma is in the journey and in creating the continuous improvement culture rather than reaching the elusive ultimate destination.

How To Follow Up On Your Survey Results

Fill in the customer satisfaction survey

Recently I have seen a number of discussions and posts discussing ways to follow-up on NPS surveys. Strangely, the writers seem to focus on transforming NPS, which is a quantitative survey with rigorous interpretation methodology into a qualitative interview, where insights are gained from reading customers’ comments or follow-up interviews to ‘drill down’ for the reasons behind the ratings. This accentuates the challenge with NPS results being non-actionable, and poses several additional problems we must think about and offer other methods to supplement the ratings.

First, let’s discuss the reasons for not using comments and interviews for extra insights from surveys:

  1. Sample size is always a concern for surveys. Comments are not always filled by customers, therefore the sample size is further reduced
  2. The manual nature of interviews will inevitably make scalability and cost a concern, further reducing sample size
  3. With global operations, language and time zones may eliminate a portion of the customers due to your inability to conduct interviews or correctly interpret comments
  4. Confirmation bias, where interviewers and comment readers only account for responses that confirm existing concepts, may pose a significant threat to the success of the survey program
  5. Discrepancies between comments and actual drivers of dissatisfaction are well documented. Relying on comments only will prevent you from confirming the comments via metrics

Now, should you read survey comments, or interview customers who rate you poorly? Absolutely, but do not confuse that for your main insights. Comments and interviews provide illustration to the broader conclusion you derive from researching the details of the survey and correlating them with your operational and demographic information.

So, how should you go about analyzing the responses from your NPS survey in greater detail?

First, by all means, call your customers back to follow up on survey results. Call those who rate you poorly, as well as those who rate you well. But, also correlate the results with demographic and operational data. For example, how does score vary across regions or industries? Do customers using a certain feature or function rate better or worse than others? Does your score vary based on support case count or their duration? How do events over time impact customers’ score? Last, and equally important, remember that non-responsive customers do that for a reason as well. Can you identify different factors that drive customers’ response rate? Does any of that indicate their propensity to renew, or churn?

In conclusion, go ahead and experiment with your customer survey results and the drivers behind them. Do not assume that what works for others will necessary work for you. If you have access to a person with statistics knowledge seek their help in building a regression model that identifies the impact of each factor on customer satisfaction. If you don’t, there’s much you can do on your own to analyze the results and understand your company’s specific environment and reach conclusions on what to improve next.

Support Basics: Case Lifecycle Measurement

Total Lunar Eclipse

A few days ago I had a very interesting conversation with an executive in the enterprise technology industry. We spoke about identifying the root cause for case delays in particular and ways to analyze case traffic in general. I was very surprised to learn that the concept of case lifecycle and the ability to track that for time as well as failed attempts was completely new to him. In this light, here’s a case lifecycle primer and a some metric guidelines associated with it.

If we think about a typical case, in most companies it would transition between multiple entities handling it for multiple reasons. Those may be the customer, the support team, an escalation team and possibly the engineering team. The case will spend some time waiting for a person or a technology resource to become available, be worked on and then transition to the next phase. As we well know, the case may spend varying amount of time in each phase and can transition between those phases repeatedly. For example, a case is opened, assigned to a support engineer, who analyzes it, proposes a resolution to the customer. The resolution does not work, the customer returns the case to the support engineer, who researches some more, requests additional information, etc.

The discussion we had concerned support managers’ ability to analyze information from case life cycle and associate the phases with improvement opportunities on one hand and impact on customer satisfaction on the other. To give us the level of detail needed, we must know, for every case, what state it is in. This state needs to include the entity that owns the next action (customer, support, engineering, etc) and details of that action. For example, a case status may be “Customer – Installing Fix”, “Customer – Collecting Information” or “Engineering – Fixing Bug”. Now, in order for this information to have any value beyond the immediate case status, we need to be able to map the entire case journey along a timeline, and do it for our entire caseload.

So, how do we do that? Enter the audit trail.

Most case management systems keep an audit trail, some will log all changes to the case, others only specific ones. In any case, recording status changes and their timestamp is the first step we need to take if we want to analyze our case load. Once we do that, we can begin to understand where our cases spend their time which will enable us to investigate the reasons for the delays. More adventurous managers will want to attempt regression analysis of various case parameters to determine the operational drivers of customer satisfaction, from case life to repeated requests for additional documentation.

Have you ever attempted to analyze your caseload this way? Have you been able to come to any meaningful insights and drive improvements?

The Only Question You’ll Ever Need, or NPS, But Carefully.

Question mark

David Kay has an excellent post discussing Net Promoter Score surveys. While I agree with David on many of the points he makes I also have a few reservations about NPS and especially its usability in enterprise technology environments and I’d like to share both in this post

To those unfamiliar with NPS, it is a methodology developed by Frederick Reichheld to measure customer loyalty using the response to a single question: “How likely are you to recommend [product or company] to a friend or a colleague?” It was introduced in the December 2003 issue of Harvard Business Review, in an article titled “The One Number You Need to Grow” (subscription required). According to the article, it is the single best predictor of future revenue growth

Over the years I had multiple exposures to NPS surveys in enterprise technology environments and found a number of things I like about it. Unsurprisingly, those are similar to what David lists in his post:

  • NPS offers companies a single point of focus to present to the different interested parties with a simple story
  • Separation of the respondents population into promoters, detractors and neutrals, and the single index created to represent progress across the spectrum
  • A clear and concise methodology that is easily understandable and relatively easy to follow

However, there is a number of things I do not like about many NPS deployments. First is the confusion between “the only number you need to grow” and “the only question you need to ask”. Second is the inability to derive actionable data from the would recommend question even in the simplest of customer experiences, let alone in a complex, multi-faceted environment such as enterprise technology. Namely, NPS maybe a leading indicator to corporate performance, but it certainly is a trailing indicator to many decisions and actions which need to be monitored and tweaked to produce the desired results

To demonstrate the actionable data point let’s imagine a simple consumer product where the expectations are clear and the likelihood of failure very low. I am sure every one of us, with a little effort can imagine half a dozen points which would cause them to not recommend the product (price, availability, functionality come to mind immediately). Each of those would require a different action from the vendor, except there is no way to determine the reason and subsequent action from the response to the NPS question without additional data and analysis. Now how many more different interaction types and touchpoints does an enterprise technology provider have with its customers?

Thinking about the suitability of NPS to the enterprise technology environment, we add to the complexity of assessing which segment of the performance spectrum of the product we encounter additional complexities. Interestingly, one of them was made in the the original HBR article: “The ‘would recommend’ question wasn’t the best predictor of growth in every case. In a few situations, it was simply irrelevant. In database software or computer systems, for instance, senior executives select vendors, and top managers typically didn’t appear on the public e-mail lists we used to sample customers. Asking users of the system whether they would recommend the system to a friend or colleague seemed a little abstract, as they had no choice in the matter.” The extension to this argument is that growing the NPS number for all respondents regardless of role maybe a futile, unfocused effort that discounts the opinions of the decision makers due to sheer scale. After all, most companies have only one CIO and very few decision makers overall at a strategic level. Also, from a support executive perspective, our ability to control all those touchpoints is limited. How many of you have ever tried to change the structure of an invoice your company sends to its customers? How successful were you?

So, should you implement NPS?

My recommendation is that you should, but you should do it very carefully. There are several things to keep in mind when doing that, and the bigger and more complex your engagements are, the more critical these points become. First, ensure every stakeholder understands that way individual actions and decisions drive NPS and it is not a number you can grow per se. Second, understand the critical touchpoints, their influence on the individuals within your customer organizations and their influence the overall desired business outcome. Third, understand your respondent population and the impact they would have on your company. Last, but not least, use NPS as a banner number to rally the troops, but be aware of the complexities behind it and be ready to tell a more nuanced story than The Only Number You Need to Grow

Thank you for reading through this long post. I’d love to hear some opinions about this very loaded subject. If you have experiences to share having implemented NPS successfully, or those implementations that did not achieve the desired results.

On Metrics and Employee Confusion

There’s an excellent post from Bob Champagne discussing the misalignment of strategy and metrics. I frequently seen this practice in employee objectives and evaluation as well as in job postings. It is not uncommon to see an enterprise software company talk on one hand about concepts such as knowledge creation and future call avoidance and on the other use low-complexity call center metrics, such as after call work and average call length. In this situation, employees will always veer towards the objectives that pay them more, either financially or in management praise or both. Conclusion? Take time to develop your metrics based on your strategy and objectives. Ask yourself how you would behave if you were faced with the choice and do not assume your employees will act differently.

Support Managers: How much do you owe?

Agile practitioners have been discussing technical debt for a while, using this term for the gap between the investment in software development and the business value derived from that same software.

After being introduced to the term and the concept behind it I kept wondering about the best way to implement a similar concept in the customer support environment.

If we accept the concept that customer support is responsible for generating or protecting customer value, then we can come to the conclusion that outstanding support debt at any point in time can be defined as “the total amount of customer value that is unrealized due to unresolved cases” at that specific point in time.

When we look at our open cases (aka backlog) we can reasonably assume that a case that’s been open for a longer period has a bigger negative impact on customer value than a case that’s been open for a shorter period. Therefore, we can safely accept the amount of time a case has been open as an approximation to its negative impact on customer value, and the total impact support has on total customer value can then be represented by the total age of all open cases (regular blog readers will note that this number is similar to the pain report, but has no weighting for the severity of the case).

So, now that we have this metric defined, what do we do with it?

First, we have to understand that this metric is a trailing indicator, showing the success we had in accomplishing two goals that are common to most support organizations:

  1. Reducing the number of cases opened
  2. Processing cases faster

And like most other metrics, its main value is in the way it changes over time and how it is associated with changes in business volume. The best option I have found for that is dividing the change in support debt by the change in revenue over a certain period. If the resulting number is lower than 1, then the organization is improving its performance, if it is greater than 1, it is not managing its workload well.

In a future post I will talk about the relationship between this metric and other commonly used ones. Stay tuned.

An open question remain. In what way ca we relate this number to financial value? Any ideas?

The Importance of Trends

In a comment to the previous post, titled “The Pain Report”, my long time friend and colleague David King made an excellent point that is worth repeating. The importance of any metric is not in a single value, but in the way it develops over time. This is how improvements or deteriorations are determined and this is how support organizations can preempt potential high visibility situations before they become so.

I have written about metrics and benchmarking in the past, that post and especially the links it contains are well worth visiting.

The Pain Report

One of the people I worked with in the past used to produce what he called “Pain Report”. This report attempted to predict the customers (or sales people) that would explode next and give support managers a way to proactively address problems.

The report was basically a weighted sum of the amount of time all cases were open. Here’s how it works:

  • Assign a weight for every severity level you have. The more severe the case, the higher the severity. For example, with a four level severity scale a possible a possible scale can be:
  • Now multiply the number of days each of the customer’s cases have been open, and what you have is the weighted number of days for every case
  • Sum the numbers you received, and you receive the customer’s pain index

The weight numbers in the table above are a way to assign relative urgency to the different severities, play with them and determine the weighting that works for you. Here is a sample of two customers’ pain report that demonstrates the influence of higher severity cases on the pain index:

Both customers have an identical number of cases open, and the total number of days is similar. The only difference between the reports are two higher severity cases that customer “A” has.

Now, this report does not attempt to predict the next angry phone call our CEO will receive. There are more variables at play here, from a pending deal to total lack of good will on the customer’s side. But, it does give us an additional perspective on the caseload and the way it is broken down by customer, and helps us diffuse potential trouble spots before they materialize. It will probably be far more useful in very high complexity environments, where customers have multiple cases open at any given time and support management needs to find a way to make sense of their workload.

Have you implemented such a report? Do you think it will useful in your environment?