ELearning! Magazine
GETTING FEEDBACK IS JUST THE FIRST STEP. ACCURACY IS NEXT.
BY JERRY ROCHE
Assessment — an important, well-established and popular slice of the e-learning universe — includes runtime assessment systems plus planning, quality, analysis, grading and feedback tools.
Accurate assessments have a world of uses, perhaps the most important being their ability to motivate employees by providing a sense of achievement. That said, the sad fact is that most businesses are not assessment- savvy — yet.
“Almost all of us are just not getting valid feedback on our instructional-development efforts,” notes Dr.Will Thalheimer on theWebsite of his company,Work-Learning Research, Inc.
When Thalheimer asked group of about 100 e-learning professionals the highest level of assessment they did, 11 percent said they conducted no evaluation and none said they measured return-on-investment. “Even after all the hot air expelled, ink spilled and electrons excited in the last 10 years regarding how we ought to be measuring business results, nobody is doing it,” Thalheimer says.
“We’re in a dark fog about how we’re doing, so we have massively impoverished information to use to make improvements.”
A LONG HISTORY?
“Assessments have been around as long as e-learning — going on 20 years, back to floppy disks in late ‘80s,” says Marketing Manager Brian McNamara of Question- Mark. “Most authoring systems and LMS’s have some kind of component to create questions or quizzes. The level of sophistication varies.”
In other words, the e-learning industry still has room for improvement. “The industry has come a very long way in the last 10 years,” says Hank Riehl, Director of Competency Management Solutions at SkillSoft. “There’s a wide range of competency libraries and batteries of criteria that form the foundation of these assessments and the basis of these models. A dozen have pretty widespread traction and market recognition.”
More sophistication is definitely needed because companies are, more and more, making assessments for regulatory compliance. “An ability to document training and education, along with an audit trail, is important,” notes McNamara. “Also, more companies are establishing internal certification systems. So with the stakes so high, it’s important that businesses take all the necessary steps to make sure they are properly assessing their e-learning efforts.”
But organizations don’t need to reinvent the wheel. “They should look at assessment models based on the nature of the nature of the population being catalogued,” suggests Riehl. “Some assessment models are stronger than others in sales and marketing; others are stronger in information technology domains. Some competency models focus on particular disciplines; others are more broadly based, dealing with interpersonal and behavioral competencies but stay out of the harder domainspecific skills.”
ASSESSMENT BEST PRACTICES
Among other characteristics, effective assessment models must be flexible, formal, reusable, complete and reproducible. They must also be “medium neutral,” so that they can be used in different publication formats, like theWeb, or paper-andpencil tests. And they must fit in available standards and specifications.
According to Kathleen Scalise of the University of Oregon and Bernard Gifford of the University of California at Berkeley, a potential limitation for realizing the benefits of computer-based assessment is in designing questions and tasks with which computers can effectively interact while still gathering meaningful measurement evidence.
Multiple-choice questions, Scalise and Gifford say, “limit the computer platform’s potential for rich and embedded assessment.” Fully constructed responses like a traditional essay “can be a challenge for computers to meaningfully analyze, even with today’s sophisticated tools,” they add. According to Mark Nichols of the Bible College of New Zealand, an effective assessment system can:
>> encourage deep approaches to learning;
>> correct student misconceptions;
>> pace student progress;
>> reflect on and improve course materials in terms of their effectiveness;
>> motivate students to explore the subject matter further;
>> help improve their analytical and critical thinking skills; and more. Nichols lists three kinds of assessments:
1 Self-marking tools include true/false, multiple choice and ordering questions that can be answered in pre-determined ways, used by most online assessment tools.
2 Simulation-based tools measure student performance in a simulated environment. They can be very effective, but they can be very expensive to purchase and create.
3 Collaborative and feedback-oriented tools such as bulletin boards and document revision permit increased levels of feedback at point of relevance.
“With the exception of the simulationbased assessments, online assessment tools are relatively easy to make use of,” says Nichols. “Self-marking tools increase efficiency and convenience for the instructor and also ensure swift marking.
Collaborative and feedback tools increase the validity of assessment (because they measure understanding) and the effectiveness of assessment as a learning tool (providing customized feedback is provided by the instructor).”
A HIGH-VALUE ALTERNATIVE
Assessment tools surely measure knowledge in a particular subject or domain. But some proponents insist that measuring ability, potential ability or actual performance is equally important, especially in a business management context.
“To many, an assessment means a hard test,” says Riehl. “Something that has right and wrong answers, that is scored and will ultimately reflect proficiency in some topic. In my world, the term‘assessment’ has to do with a self-appraisal that might occur in a performance management context. An option is a multi-rater or 360- degree approach that includes self-appraisal, one or two or three peers, managers and subordinates. All sorts of slants allow the organization greater insights. It’s a more accurate approach, but it’s also much more work.”
It’s also more expensive.
“Many organizations balk,” Riehl admits. “In actuality, many aren’t ready for this kind of process, a learning growth model. It speaks to the maturity of the organization in terms of the processes it might be willing to take.
“The payback comes in two areas: explicit, in the data itself to make more informed personnel decisions. The implicit is more powerful but fuzzier, and that’s the very process of doing assessments. It creates a lot of inward thinking and self-reflection. The renewed individual commitment to developing one’s career can be very powerful. Collectively, the workforce gets a better grasp individually of where they are and where they need to go developmentally.”
CONCLUSION
The Oregon and California researchers noted earlier in this article reach this conclusion:
“Developers must understand the range of decision-making approaches available in assessment and must plan their e-learning products to optimize some good measurement and assessment strategies. This would help fulfill the promise of the emerging field of personalized e-learning, and might bring to fruition some new tools that could substantially help instructors and students in the teaching and learning process.”
--
Subscribe Now—to secure your own personal editions
|
e
