Designing the Evaluation: What Evaluation Questions Are Addressed in an Outcomes-based Approach5 In-session evidence of: Assessment Tasks Process Skills Concepts and Issues Event Agenda (productivity, participation) Event Environment (logistics, organization) What will learners do in the training to demonstrate evidence of what they can do? For example, in-session evidence of performance – assessment of tasks. What skills must the learner master to demonstrate what they can do? For example, specific acts, ability to do tasks – process skills. What must the learner understand to demonstrate what they can do? For example, specific patterns, values, motivations – concepts and issues. How did the activities and facilitator contribute to participants’ learning? For example, provided activities that appealed to diverse ways of learning. How did the facilities, technology and equipment contribute to participants’ learning in the session? For example, offered web-based options to paper-based exercises. Did participants effectively demonstrate how they applied their new skills? Tools: Questionnaires or structured interviews with participants Participant reflections Simulations Direct observations Video or audio tapes Can participants demonstrate mastery of the intended skills? Can they list the skills? Can they communicate those skills to others? Tools: Paper or electronic questionnaires, including and tests Focus groups, interviews Participant reflections Personal learning logs Case analysis Can participants demonstrate their understanding of the concepts and issues? Can they list them? Can they communicate those concepts and issues to others? Paper or electronic questionnaires, and tests Focus groups, interviews Participant reflections Personal learning logs Case analysis At our next educational event, how could participants support each other’s learning more effectively? What did you do to contribute to the success of the educational event, other than participate? What are you learning from other participants? Tools: Questionnaires – paper or electronic Participant reflections Structured interviews group What aspects of the facility supported (did not support) your participation? What two words would you use to describe the facilities? If we conduct the next education event at this facility, what could be improved? Questionnaires – paper or electronic Participant reflections Structured interviews Focus group
Clearly label all graphs and diagrams so they are visible to the audience. Slides should reinforce highlights, not repeat your words. Keep the design and uncluttered. Leave empty space around the text and graphics. Be sure the entire slide can be seen easily from all parts of the room. Font size and colour matter. Aim for a 30 point font. Use colors that contrast and compliment so and images are easy to recognize. Make text colours dark contrasted with a background, or light contrasted with a dark background. One picture is many words. Keep the background consistent and subtle. Use quality graphics that enhance the topic of the slide. Standardize position, styles. Limit the number of slides to maintain audience interest. Tips: Know your audience: those who may read the Evaluation Report and/or listen to your presentation about the report. Provide them with answers to: “So what? What’s in it for me, our organization and our clients?” Then breathe a sigh of relief! References Canadian Society for Training and Development (CSTD) Competencies for Learning Professionals. Quick Reference. 2013. . Last accessed 30 October 2017. Steihl, R. & Lewchuk, L. (2008). The Outcomes Primer. Corvallis, Oregon: The Learning Organization. Strachan, D. & Pitters, M. (2009). Managing Facilitated Processes. San Francisco: Jossey-Bass/Wiley. Plain Language Guidelines, Government of Canada . Last accessed 30 October 2017. Steihl, R. & Lewchuk, L. (2008). The Assessment Primer: creating a flow of learning evidence. Corvallis, Oregon: The Learning Organization. Lofland, John & Lofland, Lyn H. (1995). Analyzing Social Settings: A to Qualitative Observation and Analysis. Belmont, CA: Wadsworth. Resources
The following resources were also consulted in the creation of this toolkit: Analyze This!!! Learning to analyze data. (2008). The Association for Learning Development in Higher Education. . Last accessed 30 October 2017. Guskey, Thomas R. (2005/2006). A Conversation with Thomas R. Guskey. The Evaluation Exchange, A periodical on emerging strategies in evaluation exchange. Harvard Family Research Project, Harvard Graduate School of Education. Volume XI, Number 4, Winter. Evaluation Toolkit (2012). Northwest Center for Public Health Practice, School of Public Health, University of Washington. Practical tool to help in effectively evaluating your trainings. Presentation Toolkit (2012). Northwest Center for Public Health Practice, School of Public Health, University of Washington. Strachan, D. (2007). Making Questions Work. San Francisco: Jossey-Bass.
Appendix A – Templates Step 1: Determine the Evaluation’s Purpose Designing the Evaluation: What You and Others Want to Know Interest in Evaluation In-session evidence of: Post-session evidence of: Stakeholders What learner must do when at work Skills and tasks the learner must master Concepts, issues the learner must understand Event process, productivity, participation Event environment, logistics, management Performance of learning at work Factors that affect their job performance Participants Participant’s Supervisor Participant’s Client Session leader/ educator organizer Public Planning Committee members Funders Other?
Step 2: Estimate Evaluation Resources Evaluation Resources Budget Some questions to consider regarding evaluation resources: What resources/funds do you have at your disposal? What expenses should you anticipate? What resources do you have access to? Expense Item Estimate Actual Paper evaluation – printing $ $ Telephone for interviews, group discussion $ $ Venue rental – for focus group discussion, report presentation $ $ Refreshments, breaks $ $ Technical equipment $ $ Admin costs for inputting $ $ Travel and/or parking costs (staff/participants) $ $ Accommodation costs (staff/participants) $ $ Data analysis – human resources $ $ Report writing, editing $ $ Backfill expenses $ $ Other $ $ TOTAL:
Step 3: Design the Evaluation Tool(s) Designing the Session2 – Template from Step 2: Use an Outcomes-Based Design from the Toolkit for Planning Education Events (discussed in more detail in the Toolkit for Planning Educational Events – accessed online at ; type toolkit name into search box) Intended Outcomes Assessment Tasks Process Skills Concepts and Issues What do learners need to be able to DO “in work” that we’re responsible for addressing “in the training session”? For example, post-session performance. What will learners do in the training to demonstrate evidence of what they can do? For example, in-session evidence of performance – assessment of What skills must the learner master to demonstrate what they can do? For example, specific acts, ability to do tasks – process skills. What must the learner understand to demonstrate what they can do? For example, specific facts, patterns, values, motivations – concepts and issues.
Questions Regarding the Educational Event Environment Quantitative: Please circle the appropriate numbers to describe your opinion of the education event setting: Poor 1 Fair 2 Average 3 Very Good 4 Excellent 5 Qualitative: What else could the facility have provided to enhance your participation in the PWS educational event? (one thing) Questions About the Post-session Experience Questions Regarding Performance of Learning at Work Quantitative: Circle the number that best describes how successful you were in realizing the intended PWS learning outcome. 1 Not at all 2 Very little 3 Somewhat 4 a bit 5 A great deal Qualitative: What are you differently at work with individuals with PWS as a result of attending this education event? Questions Regarding Employment Matters that Affect At-work Performance Quantitative: When “transfer of learning” problems were identified within the organization, how you rate your organization in addressing those problems quickly and efficiently, so training implementation could continue? 1 Poor 2 Fair 3 Good 4 Very Good 5 Outstanding Qualitative: Describe two on-the-job training implementation successes that were recognized and shared in your organization. Step 4: Implement the Evaluation Tool(s) Critical Path for Evaluation Process Create your own list or use the templates provided, adjusting as needed. Elements to consider in your critical path include: √ Date Task (if required) Lead