Why is ROI not a commonly used metric for success in learning?
Over the past 4 years, I have come to know many learning professionals. I’ve met ones who specialize in content creation, instructional design, learning technology and general instruction. When I ask the question of how they measure the success of their programs, ROI (Return on Investment) are three letters that are not often heard.
As I have continued to dig deeper into why ROI is not common in the learning space, it is clear the people who serve the learners are not the problem… so then what is it? I have concluded that there are 3 parts to this problem – and the good news is that all of them are solvable.
Problem 1: The Shortage of Code Wizards
Let’s start with the technology that serves the learning community, the LMS (Learning Management System), which is the most common piece of infrastructure deployed at companies today. This technology was built to manage and measure the delivery of learning programs across the organization and it does this job very well. When the LMS first hit the market, it was built to support SCORM (Shareable Content Object Reference Model, or a set of technical standards for e-learning software products) as a measurement tool. As a 2004 technology, the LMS helps companies capture completion and user engagement data but not much else.
Once you get past the LMS, all types of software tools have now become part of the learning experience. There are great authoring tools such as Adobe Captivate and Articulate, simulation tools such as WalkMe, and don’t forget about custom development as I don’t want to leave our code wizards behind. These tools were all built with the intent to enable developers to build great learning experiences and allow designers lots of freedom to create them.
But the train didn’t stop there as the industry recognized the shortcomings of SCORM and the need to generate rich data experiences. So, welcome X-API to the world (which is a rebirth of the failed TinCan API from a few years back). While new to the market, everyone got excited about X-API because it helped move the ball further down the road towards measuring ROI from learning.
However, this new technology requires “real” coding skills and as we all know, good developers don’t grow on trees. So, while there is promise, there are still opportunities ahead.
Problem 2: Third Party Data
Let’s say you work at a progressive company such as Amazon and have talented developers like @melmilloway on your team who can make awesome learning experiences that leverage X-API.
Is that alone enough to measure ROI?
The simple answer is no.
While marketing, sales, finance, and operations have easy access to performance data, the learning world does not. In order to generate a true ROI for your learning investment you need to measure the impact on the business and that requires an integration with systems such as your ERP, Salesforce, etc…. Each of these systems has their own unique data schema but they also all have API’s which would make the problem easier to solve if more learning was not delivered in the model of SCORM compliance but in the new X-API format instead. Sounds a bit complicated, I know. Stick with me.
So why are companies not requiring all projects to be X-API compliant? There are a few answers to that question but from what I can tell, it is a combination of:
- lack of X-API developer talent
- legacy LMS’s which don’t support X-API
- (and to be candid) this formula is missing a key stakeholder i.e. the CFO.
Problem 3: The CFO
The Chief Financial Officer of every company has one primary responsibility: management of the financial health of the enterprise. While this might not be their only job, it is the one that keeps them up at night and certainly what is expected by the CEO. As part of their duties, it is often the place where business cases get presented for funding, budgets get submitted for approval, and accountability on performance is measured by numbers not feelings.
CFO’s often look for opportunities to hold business unit leaders accountable for performance by using data as their driving force. Years ago CFO’s used to struggle to make this happen in areas such as marketing. The reason being that the data generated were from such ambiguous sources such as the number of impressions an ad or billboard generated. That all changed with the internet and the addition of marketing automation technologies such as Pardot, Infusionsoft, and Marketo, that delivered analytics on every detail of a campaign. Once the CFO gained the power of knowledge, the move to hold CMO’s responsible for their investment became a regular exercise and changed the face of marketing forever.
In today’s world of learning the CFO is simply a bystander to accountability. It’s in their DNA to measure and analyze. How do completion, participation and smile surveys turn into data?
I think the time has come for the CFO’s to hold learning departments accountable for performance beyond the feel-good measures above.
There is simply no reason that a learning program – be it in compliance, onboarding, sales, or leadership – can’t be measured and tied off against internal systems that drive true business KPI metrics. As the spend and innovation continue to grow in EdTech it is important that companies take the time to think about what success looks like for them:
If a company’s goal is to grow great leaders and invests in leadership training, they should take the time to determine what their ROI is for this investment.
Do executives expect every employee to be a manager who enters the leadership program? If so, what should their performance metrics be? How about a program launched to solve an ethics or compliance issue? Does a checkbox of completion really protect the company from potential fines, lawsuits and customer loss due to the violation of these issues? Hopefully, you are starting to get the picture.
Imagine this simple scenario: if this was your money and you had to risk it based upon the success of a training program, would you go forward with the standard approach, or would you want to generate measurable data?
The real data to track ROI is worth the change. Besides being able to track a return, accessing data to show training strength and weaknesses allows for training reconstruction, resulting in a better overall program. Food for thought.