A five step framework for business oriented metrics

A practical look at why some metrics programs fail while others are successful, along with some tips you can use to kick your metrics up a notch.

Introduction

I was math-challenged as a child and hatred of anything having to do with numbers followed me into adulthood. This hatred remained with me until I became a manager and needed to begin proving the work my team was doing or understanding where we were failing. Actually, the turning point may have been the now-overused adage “you can’t move what you don’t measure,” a powerful concept that has a lot to do with the metrics programs I’ve created over the years. I’ve worked hard at this, mainly because of my math aversion. While Excel certainly helps, it’s all still “funny math” and through practice I’ve learned how to justify any story I want to tell using the numbers available from the IT Management tools my organizations have used.

Ultimately, if you can a story with metrics, how do you decide what is the right story? That’s the focus of this blog: determining the story to tell and to whom. 

Building a Business Oriented Metrics Framework 

Ultimately, if you turn to ITIL for help with metrics, you can be led astray pretty easily unless you read all of the books (or at least Service Strategy (SS) and Continual Service Improvement (CSI)). This is because at the end of each process described, there is a list of Critical Success Factors (CSF’s) and Key Performance Indicators (KPI’s). These are great sample metrics for the process you might be implementing and are critical for measuring that process’ success, however providing them to business partners will have you producing the same type of metrics IT’s been producing for years, the type that are not of real interest to the business. You’ll also be led into a false set of security because they came from ITIL, didn’t they? YES!

While these process metrics are one of the three types of metrics ITIL recommends you produce (Process, Service and Technology) and while they are important metrics to produce, they’re of little or no interest to business partners outside of IT because they don’t tell you how well IT is doing at delivering on the key strategic initiatives of the business.

To craft a metrics program that is of interest to the business, you need to start with the business. To help you get started, you can use the informal framework for building business-based dashboards and scorecards presented here (If it seems familiar, it is. It’s based on ITIL’s Continual Service Improvement approach):

Linium-Metrics

This framework is very simple:

  • Know the vision of the organization or line of business
  • Document the goals that support this vision
  • Discover those Critical Success Factors (CSF’s) the organization feels are needed to be successful
  • Create Key Performance Indicators (KPI’s) or measurable indicators of the Critical Success Factors. Include target levels for these, so success is clearly shown.
  • Organize them into dashboard views for each audience that may be viewed live (on-line).
  • Develop scorecards that may be used for trending, historical reporting.

Three Steps to Using this Framework

This framework can be delivered using five basic sets of activities or steps, which are described below.

In addition to these steps however, some of this can only be demonstrating using examples. For these, let’s use a sample organization that is expanding into web-based sales to demonstrate the concepts. In this organization, the new Web Sales department and the Audit/Control group are tasked with delivering on three goals that support the organization’s vision of “providing the best shopping experience on the web.”

These goals include:

  • Providing Customers with an Excellent Web Shopping Experience
  • Giving Customers the ability to do shop any time of day (or night)
  • Guaranteeing credit card security

With this in mind, let’s look at the five steps:

Step 1Create a Focus Group

To ensure alignment, create a focus group consisting of key stakeholders from several lines of business and a few IT Managers. For the organization in the example, this would include managers from Web Sales, Audit/Control and the IT teams tasked to develop and deliver the website.

Step 2: Understand the vision, goals of the organization

With the focus group, take a look at the organization’s strategic plan. Typically the strategic plan includes a set of initiatives designed to support the organization’s vision, similar to the web sales initiative. These are often stated as goals so review the business goals associated with the initiatives and define the ways in which IT supports these goals.  Think of the goals as the pillars that support the organization.  This will ensure your program aligns with these goals and the strategic initiatives.

To move to the next step you will need the vision and goals, similar to the ones provided for the sample organization.

Step 3: Identify your audiences and their contribution

Next, working with the focus group, create a matrix to document the goals and critical success factors for each of the organizations to which you’ll be reporting. This matrix will be used to plan the dashboards and scorecard measures you need. Using the sample organization, the matrix would look like the one that follows.

Audience

Goals

CSF

KPI

(with target)

Web Sales department (1) Excellent Web Experience(2) Ability to do shop anytime
Audit/Control (1) Confidence when using credit cards
IT (1)    Service Operations Excellence(2)    “Fort Knox” security

Step 4: Make the goals measurable

To quantify the goals, you’ll need to work with your focus group to determine the Critical Success Factors that will demonstrate the fulfillment of their goals. The best Critical Success Factors (CSF’s) will be: “SMART”: Specific, Measurable, Attainable, Realistic and Timely.

Once your and the focus group have agreed on the CSF’s, you’ll be able to develop Key Performance Indicators, or measures that support the CSF. It’s extremely beneficial to develop KPI’s along with targets, so you and your business partners are clear on whether you’re successful in delivering on each of the goals. The best part about this approach is that when IT and the business agree on measures and targets, it’s easy to tell when IT has delivered or when IT is not meeting the needs identified by the business.

The ITIL books demonstrate this process clearly at the end of each process documented. The last section of the process description includes a list of Critical Success Factors for the process and Key Performance Indicators that support them.

For example, the Incident Management process (ITIL Service Operation 2011, p. 109) has a Critical Success Factor to “minimize the impact to the business of incidents that cannot be prevented.”

This is not measurable by itself, but four Key Performance Indicators follow it:

  1. The number of known errors added to the Known Error Data Base (KEDB)
  2. The percentage of accuracy of the KEDB
  3. Percentage of incidents closed by the service desk without reference to other levels of support and
  4. Average incident resolution time for those incidents linked to problem records

At the end of this stage, your matrix will be complete, similar to the one which follows for the sample organization:

Audience

Goals

CSF

KPI

(with target)

Web Sales department (1) Excellent Web Experience(2) Ability to do shop anytime (1)    Customers are satisfied with the website design and functionality(2)    Web site is available 24×7 (1) 85% of customers give the site a 5-star rating on exit(2) Web site is 100% available
Audit/Control (1) Confidence when using credit cards (1)    Web site is PCI compliant(2)    Security patches are up to date (1) 100% PCI Audit pass rate(2) 90% of patches applied within 24 hours
IT (3)    Service Operations Excellence(4)    “Fort Knox” security (1)    Web site is available 24×7(2)    Web site is PCI compliant(3)    Security patches are up to date (1)    100% site availability SLA(2)    99% performance SLA(3)    100% PCI Audit SLA(4)    No Security Breach SLA(5)    90% on-time patch SLA

You might notice several things when reading this list:

  • A qualitative measure (5-star rating by customers) is used to determine the customer’s view of the website. This is a critical measure as the CSF points to the customer’s experience.
  • The quantitative measures that sound like IT performance measures are translated to SLA’s for reporting purposes under the IT list of KPI’s. When creating the dashboards and scorecards in the IT Service Management tool, these SLA’s may be configured to demonstrate IT’s achievements against the business KPI’s.
  • Most of them sound like technology metrics. While this is true, these are a short list of technology metrics that these audiences care about. Notice some frequently reported, but missing measures: average speed of answer at the service desk, mean time to restore service etc. These would be IT metrics that support teams would need, but not IT management or the business, unless IT is failing to deliver on the metrics listed in the matrix and management wants to dig down to discover the reasons.

Step 5: Build the dashboards and scorecards

Once the matrix is agreed on and the method of measuring each KPI is defined, documented and agreed on by the focus group, the final step is to design dashboards and scorecards that represent these KPI’s. These are both graphical views of the Key Performance Indicators listed above, showing the result in comparison to the target. The main difference between the two is in the delivery:

  • Dashboards are dynamic: live representations of the data, often provided via a web portal that is integrated either to a measurement tool or directly into an ITSM tool.
  • Scorecards are static: they provide a historical look at the data including trending over a period of time.

Sustaining Success 

There are two final aspects of using this framework:

  • Continual improvement
  • Measurement retirement

As these dashboards and scorecards are used by the business, it’s important to come back to the focus group to evaluate the results. This may lead to creating new KPI’s or tweaking the ways in which they are measured, depending upon the focus group’s satisfaction with performance. In the case of the sample organization, it’s possible that the business is not meeting their objectives and may initiate changes to their critical success factors that will drive a need to change the measures. The point here is that you should not build the dashboards and scorecards then forget about them. Rather, you should meet with the focus group quarterly to review the metrics programs and IT’s achievements. This is a great opportunity to talk about service improvements that the business might need to support the initiatives as well.

Knowing when to stop delivering a dashboard or scorecard report is the last critical piece to a successful program. Once IT is reliably meeting the targets set by the business for a particular goal, it’s a good idea to discuss this result with the focus group during the quarterly review.  In this case, you’re not looking at changing CSF’s and KPI’s to address a business need, but rather you’re reviewing the KPI’s to see if the business still needs to see them continually and if any of the targets need adjustment.

Bear in mind that once you are achieving targets reliably, the business might want to work with IT to “up the game.” So in the sample organization, once the security patches and PCI audit result SLA’s are being met consistently, the business might want an shorter SLA for deployments of new features to the website. Thus, the matrix would be adjusted and the appropriate changes made to the dashboards and scorecards.

Benefits of the program

Providing metrics that are responsive to your business’ needs rather than the same stale set of IT metrics they don’t really care about will have a significant impact on the relationship between you and the rest of the business. Looking back at the reasons to measure, you can expect the following results:

  1. Direct: Live dashboards also provide the ability to determine the activities needed to drive success of an initiative and whether these activities are providing the expected result,
  2. Validate: You and your stakeholders are able to use the metrics you provide to validate whether IT’s performance is contributing to the business’ ability to meet their goals and objectives,
  3. Justify: IT is able to produce metrics that support a business case for infrastructure or development projects related to the delivery of a service,
  4. Intervene: Live dashboards provide IT and the Business to know when there is a performance issue and they can intervene immediately to turn the problem around.

This helps an organization move from a purely reactive mode to a more proactive approach that is integrated with the success of the business’ initiatives in mind.

Linium’s 5 Box Model – You Cannot Manage What You Cannot Measure from Linium on Vimeo.

Interview: Simon Morris, 'Sneaking ITIL into the Business'

Ignoring the obvious may lead to a nasty mess

I found Simon Morris via his remarkably useful ITIL in 140 app. Simon recently joined ServiceNow from a FTSE100 Advertising, Marketing and Communications group. He was Head of Operations and Engineering and part of a team that lead the Shared Services IT organisation through its transition to IT Service Management process implementation. Here, Simon kindly shares his experiences of ITSM at the rock face.

ITSM Review: You state that prior to your ITSM transformation project you were ‘spending the entire time doing break-fix work and working yourselves into the ground with an ever-increasing cycle of work’. Looking back, can you remember any specific examples of what you were doing, that ITSM resolved?

Simon Morris:

Thinking back I can now see that implementing ITSM gave us the outcomes that we expected from the investment we made in time and money, as well as outcomes that we had no idea would be achieved. Because ITIL is such a wide-ranging framework I think it’s very difficult for organisations to truly appreciate how much is involved at the outset of the project.

We certainly had no idea how much effort would be spent overall on IT Service Management, but we able to identify results early on which encouraged us to keep going. By the time I left the organisation we had multiple people dedicated to the practice, and of course ITSM processes affect all engineering staff on a day-to-day basis.

As soon we finished our ITILv3 training we took the approach of selecting processes that we were already following, and adding layers of maturity to bring them into line with best practice.

I guess at the time we didn’t know it, but we started with Continual Service Improvement – looking at existing processes and identifying improvements. One example that I can recall is Configuration Management – with a very complex Infrastructure we previously had issues in identifying the impact of maintenance work or unplanned outages. The Infrastructure had a high rate of change and it felt impossible to keep a grip on how systems interacted, and depended on each other.

Using Change Management we were able to regulate the rate of change, and keep on top of our Configuration data. Identifying the potential impact of an outage on a system was a process that went from hours down to minutes.

Q. What was the tipping point? How did the ITSM movement gather momentum from something far down the to do list to a strategic initiative? 

If I’m completely honest we had to “sneak it in”! We were under huge pressure to improve the level of professionalism, and to increase the credibility of IT, but constructing the business case for a full ITSM transition was very hard. Especially when you factor in the cost of training, certification, toolsets and the amount of time spent on process improvement. As I said, at the point I left the company we had full time headcount dedicated to ITSM, and getting approval for those additional people at the outset would have been impossible.

We were lucky to have some autonomy over the training budget and found a good partner to get a dozen or so engineers qualified to ITILv3 Foundation level. At that point we had enough momentum, and our influence at departmental head level to make the changes we needed to.

One of the outcomes of our “skunkworks” ITIL transition that we didn’t anticipate at the time was a much better financial appreciation of our IT Services. Before the project we were charging our internal business units on a bespoke rate card that didn’t accurately reflect the costs involved in providing the service. Within a year of the training we had built rate cards that both reflected the true cost of the IT Service, but also included long term planning for capacity.

This really commoditised IT Services such as Storage and Backup and we were able to apportion costs accurately to the business units that consumed the services.

Measuring the cost benefit of ITSM is something that I think the industry needs to do better in order to convince leaders that it’s a sensible business decision – I’m absolutely convinced that the improvements we made to our IT recharge model offset a sizeable portion of our initial costs. Plus we introduced benefits that were much harder to measure in a financial sense such as service uptime, reduced incident resolution times and increased credibility.

Q. How did you measure you were on the right track? What specifically were you measuring? How did you quantify success to the boss? 

Referring back to my point that we started by reviewing existing processes that were immature, and then adding layers to them. We didn’t start out with process metrics, but we added that quite early on.

If I had the opportunity to start this process again I’d definitely start with the question of measurements and metrics. Before we introduced ITSM I don’t think we definitively knew where our problems were, although of course we had a good idea about Incident resolution times and customer satisfaction.

Although it’s tempting to jump straight into process improvement I’d encourage organisations at the start of their ITSM journey to spend time building a baseline of where they are today.

Surveys from your customers and users help to gauge the level of satisfaction before you start to make improvements (Of course, this is a hard measurement to take especially if you’ve never asked your users for honest feedback before, I’ve seen some pretty brutal survey responses in my time J)

Some processes are easier to monitor than others – Incident Management comes to mind, as one that is fairly easy to gather metrics on, Event Management is another.

I would also say that having survived the ITIL Foundation course it’s important to go back into the ITIL literature to research how to measure your processes – it’s a subject that ITIL has some good guidance on with Critical Success Factors (CSFs) and Key Performance Indicators (KPIs).

Q. What would you advise to other companies that are currently stuck in the wrong place, ignoring the dog? (See Simon’s analogy here). Is there anything that you learnt on your journey that you would do differently next time? 

Wow, this is a big question.

Business outcomes

My first thought is that IT organisations should remember that our purpose is to deliver an outcome to the business, and your ITSM deployment should reflect this. In the same way that running IT projects with no clear business benefit, or alignment to an overall strategy is a bad idea – we shouldn’t be implementing ITIL just for the sake of doing it.

For every process that you design or improve, the first question should be “What is the business outcome”, closely followed by “How am I going to prove that I delivered this outcome”. An example for Incident Management would be an outcome of “restoring access to IT services within an agreed timeframe”, so the obvious answer to the second question is “to measure the time to resolution.”

By analysing each process in this way you can get a clearer idea of what types of measurement you should take to:

  • Ensure that the process delivers value and
  • Demonstrate that value.

I actually think that you should start designing the process back-to-front. Identify the outcome, then the method of measurement and then work out what the process should be.

Every time I see an Incident Management form with hundreds of different choices for the category (Hardware, Software, Keyboard, Server etc.) I always wonder if the reporting requirements were ever considered. Or did we just add fields for the sake of it.

Tool maturity

Next I would encourage organisations to consider their process maturity and ITSM toolset maturity as 2 different factors. There is a huge amount of choice in the ITSM suite market at the moment (of course I work for a vendor now, so I’m entitled to have a bias!), but organisations should remember that all of vendors offer a toolset and nothing more.

The tool has to support the process that you design, and it’s far too easy to take a great toolset and implement a lousy process. A year into your transition to ITSM you won’t be able to prove the worth of the time and money spent, and you have the risk of the process being devalued or abandoned.

Having a good process will drive the choice of tool, and design decisions on how that tool is configured. Having the right toolset is huge factor in the chances of a successful transition to ITSM. I’ve lived through experiences with legacy, unwieldy ITSM vendors and it makes the task so much harder.

Participation at every level

One of the best choices we made when we transitioned to ITSM was that we trained a cross-section of engineers across the company. Of the initial group of people to go through ITILv3 Foundation training we had engineers from the Service desk, PC and Mac support, Infrastructure, Service Delivery Managers, Asset management staff and departmental heads.

The result was that we had a core of people who were motivated enough to promote the changes we were making all across the IT department at different levels of seniority. Introducing change, and especially changes that measure the performance of teams and individuals will always induce fear and doubt in some people.

Had we limited the ITIL training to just the management team I don’t think we would have had the same successes. My only regret is that our highest level of IT management managed to swerve the training – I’ll send my old boss the link to this interview to remind him of this!

Find the right pace

A transition to ITSM processes is a marathon, not a sprint so it’s important to find the right tempo for your organisation. Rather than throwing an unsustainable amount of resource at process improvement for a short amount of time I’d advise organisations to recognise that they’ll need to reserve effort on a permanent basis to monitor, measure and improve their services.

ITIL burnout is a very real risk.

 

Simon Morris

My last piece of advice is not to feel that you should implement every process on day one. I can’t think of one approach that would be more prone to failure. I’ve read criticism from ITSM pundits that it’s very rare to find a full ITILv3 implementation in the field. I think that says more about the breadth and depth of the ITIL framework than the failings of companies that implement it.

There’s an adage from the Free Software community – “release early, release often” that is great for ITSM process improvements.

By the time that I left my previous organisation we had iterated through 3 versions of Change Management, each time adding more maturity to the process and making incremental improvements.

I’d recommend reading “ITIL Lite, A road map to full or partial ITIL implementation” by Malcolm Fry. He outlines why ITILv3 might not be fully implemented and the reasons make absolute sense:

  • Cost
  • No customer support
  • Time constraints
  • Ownership
  • Running out of steam

IT Service Management is a cultural change, and it’s worth taking the time to alter peoples working habits gradually over time, rather than exposing them to a huge amount of process change quickly.

Q. Lastly, what do you do at ServiceNow?

I work as a developer in the Application Development Team in Richmond, London. We’re responsible for the ITSM and Business process applications that run on our Cloud platform. On a day-to-day basis this means reviewing our core applications (Incident, Problem, Change, CMDB) and looking for improvements based on customer requirements and best practice.

Obviously the recent ITIL 2011 release is interesting as we work our way through the literature and compare it against our toolset. Recently I’ve also been involved in researching how best to integrate Defect Management into our SCRUM product.

The sign that ServiceNow is growing at an amazing rate (we’re currently the second fastest growing tech company in the US) shows that ITSM is being taken seriously by organisations, and they are investing money to get the returns that a successful transition can offer. These should be encouraging signs to organisations that are starting their journey with ITIL.

@simo_morris
Beer and Speech
Photo Credit