MEASURING AND EVALUATING LEARNING IS ONE OF THE MOST
CRITICAL ISSUES IN THE LEARNING AND DEVELOPMENT
FIELD.
BY DR. JACK PHILLIPS AND HOLLY BURKETT
The value of learning is inherent in today’s
business environment where leaders understand that
it takes a trained workforce to drive business
results in an increasingly competitive, global
environment.
As learning becomes more integrated with business
strategy, human capital is seen as a key
differentiator and driver of growth, transformation
and productivity. In this knowledge-based economy,
investments in employee learning and development
continue to accelerate, with technology-based
training solutions penetrating all areas of
corporate education and training. At a cost over $3
billion a year, high-quality training courses,
tutorials, and multimedia simulations that keep up
with the pace of change and rapid technological
advancement are
intrinsically expensive and time-consuming to
create.
While most executives intuitively realize learning
is a basic necessity and that e-learning can provide
greater flexibility for workplace learning to occur,
there is increased frustration from the lack of
evidence to show that learning programs of any
modality really work. As executives and managers
watch budgets grow without appropriate
accountability, return-oninvestment (ROI)
calculations and evidence of bottom-line value are
now being routinely required from learning
departments and e-learning functions. Sponsors need
to see how major investments of time, money and
resources are paying off and contributing to
strategic business goals.
In today’s dynamic, cost-competitive,
resource-constrained business climate, measuring and
evaluating learning has earned a place as one of the
most critical issues in the learning and development
field and is largely driven by the needs of the
clients and sponsors of learning projects, programs,
initiatives and solutions.
THE E-LEARNING PERSPECTIVE
Despite heightened interest in measuring the value
of learning, there seems to be much confusion about
measuring the ROI of e-learning methods. From the
practical perspective, e-learning is simply another
method of instructional delivery and that the
process of calculating ROI is the same, regardless
of the delivery method.
However, in far too many cases, e-learning
investments are justified by simply comparing the
cost of the instructional-led approach to the
e-learning approach. This can result in radically
different cost amounts because of the savings
generated with travel, lodging, classroom facilities
and instructor cost. For such a comparative analysis
to be accurate, the outcomes of the instructor-led
learning must be the same as those generated with
e-learning — and that is not always the case.
THE RECOMMENDED APPROACH
An accurate ROI calculation of an elearning program
requires data collection at four levels: reaction,
learning, application and impact. The impact data is
isolated from other influences and converted to
monetary value. This monetary value is then compared
to the cost of the elearning program. The result is
an incredible ROI study.
To fully compare the value of an e-learning program
to the value of an instructor-led program requires
two comparable studies with the same calculation
process in which a string of benefits and costs is
developed for both scenarios. If the ROI is higher
for the e-learning delivery alternative, then it can
be proven to be a good approach from the economic
perspective.
AN ALTERNATIVE APPROACH
Due to the extra effort to capture four levels of
data and conduct a comprehensive ROI study, it is
not always possible to complete an instructor-led
comparison. Consequently, the ROI based on cost
savings alone may be appropriate. The key issue is
determining program success with targeted learning
outcomes.
For instance, if there is data from the
instructor-led program that shows the extent of
desired learning gains, then the same measures can
be used to compare learning gains from the
e-learning perspective. In other words, if the
participants learn just as much or more with the
e-learning delivery method as they do with an
instructor-led option, then it may be proper to
compare the two projects on cost savings alone —
with maybe one more adjustment. At level three, when
the evaluation follows what participants do back on
the job with what they have learned, our studies
have shown that participants in e-learning programs
are less likely to follow through than in an
instructor-led program. In an instructor-led process
there is often a commitment made between the
participant and the instructor that might increase
the likelihood of participants applying what was
learned and increase the likelihood that performance
improvement and impact data is provided in a
follow-up survey. An e-learning participant is
generally disconnected from the instructor and can
conveniently ignore the computer, which reduces
application of learning. This factor will then tend
to reduce the monetary value or business impact of
an e-learning option, because the monetary benefits
will not be the same as those from an instructor-led
solution. In this case, the complete ROI study would
be conducted using benefits and costs for the
e-learning, along with the cost savings generated by
the cost differential of presenting the program two
different ways.
This should also include a footnote suggesting that
the e-learning results might not be quite as
impressive, because there will typically be less
application. Essentially, the payoff is expressed as
a difference in the cost two different ways, but
with a qualifier statement.
SUMMARY AND NEXT STEPS
The complexities of this situation have caused some
professionals to steer ROI calculations away from
cost comparisons alone.
For example, in a recent article in
Training
magazine that focused on ROI in e-learning, readers
were asked to submit ROI studies for inclusion in a
future article. The request included a notation that
studies must not be based on cost savings alone, but
must also show the full ROI calculation. This was
intended to underscore the importance of a complete,
accurate analysis, as described here.
For designers and developers, there are some lessons
on conducting credible ROI studies. Showing the
cost-versus-benefit of an e-learning approach is no
different than any other delivery method. Many
e-learning programs are developed on the assumption
that converting facilitator-led programs to
e-learning platforms are more cost-effective and
practical. To ensure that this is the case, the ROI
of the facilitatorled program should be compared
with the ROI of the e-learning version to determine
its true cost savings and utility.
Essentially, running the two programs parallel may
be the most preferred way to gather this data. The
good news is that cost reductions are so significant
that even if the instructor-led and e-learning
benefits are not always on the same level, the
e-learning approach may still result in a much more
positive ROI. In fact, many organizations have shown
that investments in learning technologies are paying
off with clear cost savings, efficiency gains,
increased content reuse, and decreased costs of
learning development and delivery. The key is to
ensure that the e-learning project or solution is
connected to one or more business measures that will
reflect its overall success.
In reality, most learning or e-learning projects are
undertaken to deliver business value. It is
important to remember that positioning an e-learning
project to add value requires a more broad-based
view than a simple ROI calculation.
Using the ROI process to collect results data at
multiple levels of impact and show how results are
linked to targeted business measures is an effective
way to demonstrate the worth of e-learning projects
and add credibility to the e-learning function.
Since e-learning is an investment that impacts
multiple areas of an organization, a strategic
“line-of-sight” on the organizational use of
e-learning is critical. Key questions include:
>>
How does your e-learning function provide strategic
value?
>>
How does your strategic activity translate into
business results?
>>
Do your e-learning programs make you more responsive
to your customers?
>>
Is your organization able to innovate more
effectively as a result of e-learning investments?
>>
Do those who need information have greater access to
it?
>>
Do e-learning services help your organization
achieve its core capacities more effectively? More
evaluation work is needed in the e-learning arena,
so give us your input, conduct your own impact
studies, share lessons learned and best practices …
and stay tuned.
—Jack Phillips, Ph.D., is chairman of the ROI
Institute and a world-renowned expert on
accountability, measurement and evaluation. Using
more than 20 years of professional experience, he
has developed a sound, tested methodology for
calculating true financial ROI for training,
leadership, marketing, e-learning and performance
improvement initiatives. Visit the ROI Institute at
www.roiinstitute.net.
—Holly Burkett, MA, SPHR, CPT, Principal of
Evaluation Works, is a certified ROI professional
with more than 18 years experience in performance
improvement and measurement. Editor-in-chief of
ISPI’s
Performance Improvement Journal,
she recently co-authored “The ROI Fieldbook” with
Jack and Patti Phillips and Ron Stone (2006). Reach
her via e-mail at
burketth@earthlink.net
|
You’ve heard the traditional benefits of how
organizations are harnessing the power of
e-learning to scale to global audiences,
reduce travel costs and deliver just-in-time
training. But as e-learning itself has
evolved, so have the ways in which it is
used. From driving organizational change to
addressing the needs of the
multigenerational workforce, here are some
additional ways that corporate America is
making the case for e-learning as a valuable
tool.
CAREER DEVELOPMENT
A December 2006 survey by BlessingWhite
focusing on career development reported that
41 percent of respondents felt their
employer’s approach to career development
failed to meet their personal needs. What
employees really want to know is where to
go, what to take, and how to squeeze it in
without taking too much time away from the
office.
As a result, job-role-specific learning
roadmaps are now a hot topic in talent
management. What you’ll find within many of
these corporate roadmaps is e-learning.
Including custom e-courses developed
internally and off-the-shelf products,
e-learning is being used to conduct employee
orientations, develop soft skills and
provide the foundation for technical
certification programs. From a management
perspective, e-learning demonstrates a
commitment to employee growth, is a critical
component in an overall human capital
strategy, and is a solid retention tool to
keep employees loyal in an economy where the
Society for Human Resource Management
estimates turnover costs at $5 trillion
annually.
CHANGE MANAGEMENT
In many cases, the development of new
training offerings is the result of a change
impacting an organization or its employees.
Whether it’s training designed to increase
awareness about a new initiative, a new
technology or a new process, ensuring that
the company’s target audience receives the
information needed to support the change is
critical.
The goal of any change management initiative
is adoption. Many organizations are using
e-learning as a change management vehicle to
minimize resistance and ensure employees
have the information needed to rapidly move
up the adoption curve.
COMPLIANCE
According to a 2005 National Business Ethics
survey, the most common types of unethical
employee misconduct range from abuse or
intimidating behavior to discrimination and
sexual harassment. The Conference
Board/Ethics and Compliance Officers (ECOA)
survey shows company anti-corruption
practices becoming significantly widespread.
So many corporations are under pressure to
demonstrate compliance with legal statutes
and minimize risk.
E-learning can be used to set expectations
from the beginning as a means to communicate
the parameters, requirements and
consequences of not following company
policy. Whether it’s a Sarbanes-Oxley course
or an interactive simulation on sexual
harassment, e-learning has become an
effective means for many corporations to
demonstrate compliance through mandated
training.
MULTIGENERATIONAL
WORKFORCE
With 75 million “Millennials” entering the
U.S. workforce in the years to come, the
pressure to understand, retain and cater to
their needs is immense. Millennials — with
an extreme amount of technical savvy and
comfort with Web 2.0 tools — have shorter
attention spans than their generational
peers and prefer bite-sized education they
can access whenever they want it.
Whether e-courses are delivered on a PC or
an iPod, the modular nature of e-learning
not only helps to fulfill the needs of these
knowledge-hungry workers with an appetite
for technology, it also helps their “Gen-X”
and “Baby-Boomer” peers as well. Whether
taken in an office or commuting on a bullet
train, elearning can be an educational
compromise to meet the conflicting needs of
all members of the multigenerational
workforce.
With the expansion of e-learning into so
many facets of their businesses, it’s
clearly evident why more C-level executives
and higher are leveraging e-learning to
enable their strategic initiatives.
—Tom Floyd is the CEO of management
consulting firm IEC (www.ieconsulting.biz)
in San Jose, Calif. Reach him at
tfloyd@ieconsulting.biz.
|
e
