Over the past 18 months or so, the American Management
Association (AMA) has been actively expanding its
ability to measure the effectiveness of its learning
solutions. The AMA used the widely known and accepted
Kirkpatrick model as the foundation of its approach. As
a result, it can now offer clients the option to measure
the effectiveness of the learning solution on the
individual and organization. This was somewhat tricky,
because the AMA provides both public seminar solutions
and corporate, customized solutions — the former
presenting more of a challenge than the latter.
The AMA has traditionally measured Level I successfully.
Level I evaluation measures the learner’s reactions of
the learning event. AMA gathers data from learners on
their perceptions of the facilitator, learning design,
materials, facility, hotel accommodations and
registration process. We want data from customers on the
whole learning experience so we can analyze customer
perceptions and take specific actions. Over the many
years, the AMA has been collecting Level I data so we
can identify trends and see where customers’ needs are
strongest.
ON TO HIGHER LEVELS
Newer to the AMA measurement push is a solution for
Level II evaluation, which is focused on determining
learning transfer. Tackling Level II measurement meant
that we needed new systems, content and processes,
including:
>>valid
and reliable assessment tools
>>Web-based
content
>>a
delivery mechanism (LMS and LCMS technology)
>>reporting
capabilities
>>a
refined marketing and customer communication process
Our level II evaluation approach did offer one break: we
did not need to develop separate solutions for our
public seminar and corporate businesses. Learning is
learning — it does not really matter if that learning
was delivered in a public seminar room or a corporate
training facility.
Level II is about creating a baseline measurement before
any learning takes place and then assessing what was
learned after the learning experience. So that’s what we
did: we created assessments, continuously measured
assessment reliability and validity, delivered the
assessments through LMS technology, distributed post
seminar assessments, and reported data to communicate
learning transfer. We also took the next logical step:
identifying learning gaps and then delivering targeted
Web-based content to close those gaps.
AMA was determined not to simply report on learning
transfer, but to also report on learning gaps identified
by assessment responses, communicate learning gaps to
learners and deliver targeted “tune-up” content designed
to close identified gaps. The immediate and targeted
distribution of tune-up content clearly has a positive
effect on learning transfer data.
Level III evaluation, focused on measuring behavior
change back at work, required a slightly different
solution for our public seminar and corporate
businesses.
In our public seminar business, we normally do not have
direct access to the organization to effectively measure
Level III. We can ask learners — through surveys, for
example — how they think they are bringing the learning
back to the office. Our goal is to strengthen Level III
evaluation in a public seminar environment by connecting
with the learner’s manager and asking questions to
determine the level of behavior change.
Level II evaluation for our corporate solutions is much
easier to measure. We have access to the organization,
so we can determine with our clients how we’ll approach
Level II evaluation. Once the methodology is finalized,
we can deliver content in the modality that meets the
client’s expectations.
RETURN ON EXPECTATIONS
The AMA identifies Level IV evaluation using the most
recent definition, which is focused on measuring return
on expectations. You might be concerned if you were
asked to provide return on investment data (the formerly
accepted definition of Level IV) after a training
intervention. Rightfully so, many outside variables can
effect how a team performs. Was it the training? Was it
a recent bonus? Was it acknowledgement from the CEO? It
is a monumental task to “prove” the change resulted
solely from training.
Return on expectations is a much more manageable and
effective definition. For corporate clients, we can
certainly identify key expectations before the
intervention begins. We can agree on how we’ll measure
success and how we’ll identify success. Level IV is not
used for our public seminar clients. In a public seminar
setting, the learning experience is focused on the
individuals’ development. We certainly identify
expectations for the learner at many points in the
learning experience but the collective, organizational
data is not available.
The biggest challenge in creating and delivering a solid
measurement approach is not the creation of assessments,
validating assessments, building and tuning up content,
implementing a LCMS and LMS or designing reports; it is
getting learners to complete the assessments and
content. Measurement requires that the learners,
managers and leaders take ownership and responsibility
for each level of evaluation. Learners and managers must
complete pre- and post-learning surveys and assessments.
Leaders must clearly identify expectations, success and
evaluate the data to determine what worked and what
needs more focus and attention.
—Nicholas Igneri is Director of Learning Technologies
for the American Management Association. Contact him via
e-mail at nigneri@amanet.org.
e
