So I saw a twitter thread pointing to an argument about how ROI is dead. And, well, that’s largely okay with me. However, the trigger for the post was from the results of Chief Learning Officer 2020 State of Learning report. And, when I saw them, I saw some problems. The question is whether we’re measuring impact, or not. I’d like to go through them and evaluate each.

(Back to my usual prose, as I need visual support for this. 😉

So, in the report, they indicated that the respondents indicated the demonstrated impact of training in these ways:

• General training output data
• Training output data aligned with corporate initiatives
• Learner satisfaction with training
• Employee satisfaction with training availability
• Employee engagement
• Business impact
• Employee performance data
• Planned to actual budget, expense, revenue data for training group
• Stakeholder satisfaction with training data
• ROI measures
• Net promoter score
• Employee satisfaction with company data

Yikes!  Some of these are problematic at best.  Let’s look at why some of these might not be good measures of impact. And, let’s be clear; impact should be about positively affecting the organization in a meaningful way. Moving needles like fewer errors, more revenue, reduced costs, happier employees and customers, etc.

So, first, what is general training output data? If it’s like what I saw in (then) ASTD’s State of the Industry report, it’s metrics like employees served per L&D employee or cost/seat/hour for training. Which might a useful measure of efficiency, if you can come up with a principled basis for what a good number would be, and then see if you’re above or below that. Unfortunately, what people do is just compare themselves to the industry average. Is that a good indicator? How do you know? Do you want to be just ‘better than average’?

Then, training output data aligned with corporate initiatives. Again, hard to say what this means (and I can’t seem to find the report). However, it sounds like it’s still efficiency, just doing that for things the business thinks are important.

And we go worse: learner satisfaction with training? Er, research I’ve read and heard cited (I think it’s from Salas, et al, but memory fails) says that’s not valid. There’s a .09 correlation between what learners think of learning impact, and it’s actual impact. That’s zero with a rounding error. That’s all about making learning ‘fun’ (instead of ‘hard fun’). Yes, you do want them to think it’s also been a good experience if you’re focusing on LXD, but that’s secondary.

Similarly, with satisfaction with training availability. What’s that matter? That’s not impact!

Some good things buried here: employee engagement should be good; more engaged employees is a good thing. As long as it’s not at the cost of something else, like, say, impact? And business impact is obviously good, as is employee performance data. Presumably positive business impact, and employee performance improving.

Planned to … stuff is all about efficiency again. And that’s ok, but only after impact. Otherwise, well, we’re not costing too much…?!?

Satisfaction again not good.

And, to the original point of the article. ROI?  Yes, what it costs you to move a needle should be less than the cost of what the needle was costing you. However, I could be doing things that return the biggest ROI without doing the most important things. They can be different (e.g. a small program with a better ROI but less overall impact). So it’s only secondary.

Finally, employee satisfaction with company data? I have no idea what that means? But, again, ‘satisfaction’ isn’t really meaningful unless it’s based on real impact.

I’ve complained before about L&D measurement. Here it is, right in front of us. The answer to the question of whether we’re measuring impact or not appears to be ‘mostly not’.  We’re still (largely) measuring the wrong things. And we wonder why we don’t have credibility. Please, please, start designing to improve measurable gaps, and then actually improve the outcome. Otherwise, you’ve no idea whether that bum in that seat for an hour is doing the organization any good, versus just not costing too much relative to the industry average.

The post Measuring Impact (or not) appeared first on Learnlets.