The learning industry’s well-known four-level model of evaluation, introduced by Donald Kirkpatrick, celebrates its 60th anniversary this year, so it’s time to reflect on just how far the industry has come during that time. Industry reports, including 2016’s “The State of Learning Measurement” from Bersin by Deloitte and “Measure Up or Shut Down: The Importance of Measuring Business Impact and ROI” by Training Industry, show that learning leaders believe they are quite adept at measuring enrollments, completions, learner satisfaction and test scores. Yet, after all this time, learning organizations still struggle with determining job and business impact. Why, after 60 years of effort, are learning and development professionals still stuck? Why are people still struggling with measuring impact?

A recent survey by LEO Learning and Watershed, “The Pressure Continues to Rise: Measuring the Business Impact of Learning in 2019,” found that 96 percent of learning leaders want to measure impact, and 67 percent are feeling pressure to do so. The study asked leaders why they are not measuring the effects of learning, and results revealed that leaders struggle with competing priorities, not knowing how to start and not having access to data.

These reasons share a common root cause: a lack of confidence in the ability to measure impact. The traditional measurement framework does a good job of providing a structure, but it is time to expand the framework to enable L&D professionals to build confidence in their ability to measure impact. It is time to add a foundational element to the model: alignment.

It Starts With Alignment

There is buzz in the learning community about the need to “align with the business.” Yet much of the buzz remains simply noise because what that alignment looks like often remains fuzzy and ill-defined. All too often, L&D teams fail to clearly define the link between training and business goals. Instead, training gets developed and everyone hopes it makes an impact. What is needed is a repeatable process L&D can use to explicitly describe the links between training and business goals. It means adding “alignment” as the first step in the measurement process.

Building a Measurement Map is a great way to illustrate alignment and hypothesize about the causal chain of evidence between a learning program and business goals. L&D and business stakeholders collaboratively engage in a measurement-mapping process and create a map that logically depicts, in measurable terms, how a learning program is expected to affect the business.

What Is a Measurement Map?

The format of a Measurement Map is intentionally straightforward. Its goal is to clearly communicate alignment, typically on one sheet of paper. It consists of four logically connected sections.

Investment: This is the training program or other initiative intended to drive business impact. A Measurement Map defines what success would look like.

Leading indicators: These are nonfinancial measures that provide insight into whether the investment is on track. Leading indicators often include evidence of behaviors and outputs. The logical argument posits that if these indicators are moving in the right direction, then the investment is on track to realize desired business results. A Measurement Map typically includes several progressive leading indicators, starting with such measures as adoption or training penetration rates and satisfaction scores, and leading to such things as employee engagement scores, number of appointments, number of new accounts and so on.

Business results: These measures carry a financial value, expressed either in dollars or easily translated into financial terms by the business. Common examples include employee turnover rates, sales volume, production run-rates and workers’ compensation costs.

Strategic goals: These represent the organizational imperative that the initiative is trying to impact, such as profitability, market share or customer loyalty.

The map uses the terms “leading indicators” and “business results” to provide a common lexicon for thinking about alignment and measuring impact. An organization may use different terminology for their metrics, but the concept of progressive metrics resonates across organizations. In a map, these metrics are all key performance indicators.

How Does a Measurement Map Look?

Before getting into how to build a Measurement Map, consider a few examples that help illustrate the value of the map in creating alignment. The straightforward example shown in Figure 1 depicts the causal chain of evidence between sales training and improved financial performance.

Suppose a sales department wanted to improve bottom-line results and believed that improving the selling skills of its salesforce could contribute to that goal. Before building any training, the L&D team asked the vice president of sales how she measured success. She answered, “Improved market share and profitability.” The L&D team probed further, asking specifically how the salesforce would be expected to contribute to that goal. The VP explained that her organization holds salespeople accountable for sales volume, which leads to market share, and for gross profits, which lead to profitability. Other members of the sales team quickly pointed out that sales are tougher to new customers than among repeat and referral customers, and profit margins are different between the groups. They also have different quotas for repeat versus referral customers.

The L&D team kept digging, asking what evidence might indicate if a salesperson is making progress toward the goal. After identifying a closing ratio, the team asked the VP how she knows if a salesperson is on track to improve their closing ratio. The answer revealed that appointment show rate and product and proposal presentations provided solid indicators of closing ratio. The VP quickly pointed out that the first things they wanted to see from their salespeople included more customer contacts and definitely more appointments.

In the discussion of repeat and referral customers, the VP of sales voiced the importance of a good customer experience — without it, there would be no repeat or referral sales. She pulled out a recent customer satisfaction report and pointed to two key elements she holds the salesforce accountable for: product knowledge and building rapport with the customer.

As the map in Figure 1 depicts, several layers of leading indicators and business results exist. The former provides quantitative evidence of early progress toward the goal, but not financial metrics. Sales volume and gross profits are readily recognized as financial measures, and thus are business results. The map arranges them to show the logical link between the performance objectives of a new sales training program and an organization’s market share and profitability goals. As the map shows, the L&D team clearly knows what the new program is expected to deliver — and how its success will be evaluated. An important aspect of this map is that the sales department was involved in creating it and defining success in well-articulated business outcomes, generating buy-in to the causal model. All elements on the map are measurable, and with an engaged stakeholder, gaining access to this business data becomes much easier.

The sales example illustrated building a Measurement Map from right to left. The map also works from left to right. A second example, shown in Figure 2, tackles a more nebulous investment: mentoring. In this case, the initiative is already in place and the organization is questioning its value. In other words, “Is it working?”

The L&D team pulled together the human resources sponsor and key business stakeholders who had employees participating in the mentoring program. HR described the objectives behind the program: developing and retaining employees and improving both individual and company performance.

L&D started by clarifying the ultimate business goal and then took stakeholders back to the objectives, pushing the group to describe the evidence they would expect to see if the mentoring program were successful.

The team quickly noted the importance of having enough mentors for all the hand-raising mentees. From there, they moved to satisfaction with the relationship and value of mentoring, noting they would need to develop a survey to gather some of this data. Stakeholders agreed that re-engagement of mentors — and mentees becoming mentors — would speak volumes to the value employees find in the program. They then posited that if employees had a good mentoring experience, performance would improve. L&D pushed on this one, asking how performance should be measured; after all, mentoring was happening across departments, each with its own business results.

With a good understanding of the company’s performance management process, stakeholders landed on ratings and pay increases. They continued building out the logic that these, plus a good mentoring experience, would lead to more internal promotions and better employee engagement. These in turn would lead to reduced recruiting costs and turnover. By impacting these costs, the mentoring program would be positively contributing to the company’s financial performance.

These two examples demonstrate how a Measurement Map guides the deconstruction of a strategy down to observable and measurable inputs, activities and outputs. Whether starting at the left or the right, the map provides the logical framework to align people strategies with business strategies.

Engage Stakeholders to Build a Map

The best format for building a Measurement Map involves a stakeholder meeting with L&D and business partner representatives. Stakeholders need to be sold on the value of the process, as they may traditionally come to L&D simply requesting new training programs. The intent of the measurement-mapping session is to align all parties around the business problem to be solved, determine what success would look like and explain how a learning intervention can support that. In other words, the session defines the expectations of the learning intervention in observable, measurable business terms.

This roughly two-hour facilitated session works a bit like a brainstorming activity yet follows a structured measurement-mapping process. It begins with discussion and refinement of the business problem, the potential intervention and the intended audience for the solution. This flows naturally into outlining some hypotheses of what stakeholders hope the intervention will achieve.

After laying this groundwork, the session shifts focus to the metrics — the observable, countable evidence of what early and ultimate success looks like. This part can get challenging and a bit uncomfortable while also being thought-provoking, as participants likely have not thought about learning and outcomes in this fashion. By asking a series of probing questions, the facilitator drives for observable outcomes, challenging participants to push out of their comfort zone and think about workplace performance, looking for evidence of the learning taking hold. During this discussion, no distinction is made between leading indicators and business results, as that can stall the brainstorming. Making that distinction comes next with the sequencing of all KPIs into the map.

At the completion of the brainstorm, each KPI gets written on a Post-it. As a group, participants start arranging the notes on a wall, building their map and creating a causal chain of evidence. To test the logic, participants practice telling the story from right to left and from left to right. When business stakeholders give the map a thumbs-up, L&D knows they have alignment. They also likely have the attention of the business in designing a sound solution and measuring its impact.

A participant summed up the process well: “By engaging the business in this process, there has been a huge shift in how they think about ‘training’ and the value of appropriate, instructionally sound solutions.”

Filling the Void

Learning leaders struggle with measuring business impact, often because they do not know how to start. Using a tool like the Measurement Map fills a void in learning’s traditional measurement framework by bringing alignment into focus. The map also provides the foundation for designing and executing a credible measurement plan. One learning leader commented, “This is a truly tangible tool that takes the ‘scary’ out of tying back to business impact.”

Creating a Measurement Map is both art and science. No two maps will be alike. As long as all parties agree to the logical argument and all items are measurable, it’s a good map. Start mapping, and enjoy the process!

The post So you want to measure impact. Now what? appeared first on Chief Learning Officer – CLO Media.