Describe the (ROI) methodology and its purpose. Table 1-2 on p. 14 adds this to Kirkpatricks info. Converts the info in Level 4 to monetary value. Comparison of costs to monetary benefits. ROI provides the best argument to decision-makers of the value of training. It is also the biggest challenge in our field (Star opinion) Review and illustrate the components of the balanced scorecard approach to measurement and evaluation. The Balanced Scorecard lets organizations evaluate their effectiveness on 4 key areas. Consumer’s perspective Innovation and Learning perspective The internal business perspective The financial perspective. Note these four levels sort of correspond to Kirkpatrick’s—just on an organizational level. Define meta-evaluation and describe its use in the project management of evaluation. Meta-evaluation is an evaluation of evaluations. It assures quality of evaluations used and if of multiple evaluations may indicate trends. Evaluating the evaluation is part of the project management process of a PM of evaluation. (p. 16). 2- STATISTICAL THEORY AND METHODS Objective Information Define and illustrate the measures of central tendency. Mean – the average (sum of numbers ÷ number of numbers) Median – middle point with half numbers above and half below Mode – the most frequently occurring number Define and compare the various types of frequency distributions. Frequency distribution – shows how actual numbers group. Relative frequency distribution – uses percentage of observations in what’s shown. Implication is frequency distributions are picture-based (graphic), like the bell curve (normal distribution).
Identify the main sources of measurement error, including the various forms of bias. Bias exists, so take actions to minimize it. Sampling Bias – choosing individual or types of people that influence the results Insufficient Sample size (see figure 3-1) – if not large enough sample, then skews effectiveness of results. Observation Bias – obtrusion; awareness of being watched; and subjective bias of people doing observation Bias in Interviews and Focus Groups – need good sampling, participants need to feel willing and safe to answer, interviewer subjectivity or leading questions can cause bias Central Tendency Bias – tendency to avoid extremes and choose most neutral on Likert scale Facilitator Bias – if facilitator has bias towards test subjects, doesn’t believe in value of treatment or data Emotional Bias – participants allow feelings for instructor influence judgment on training materials. Restriction of or Range Error – predisposed attitude causing rater to stay within same general area on scale Describe the basic rights of human subjects (participants). Anonymity of raters Neutral person collects feedback Explain purpose of evaluation Name and describe three tools for identifying problems within an organization. Lewin’s Forcefield analysis (driving forces vs. resisting forces) Cause-and-Effect Analysis (Fishbone/Ishikawa) Hypothesis Testing (test a null hypothesis) Outline the steps in research design development. Determine the purpose Determine stakeholders Determine the level Identify program objectives Plan data collection Plan data analysis Plan communication Develop a project pl;lpan List and discuss the advantages and disadvantages of various data collection methods. Simulations – creates a safe, real-world-like tools/situations Surveys – questionnaire see table 3-1 for advantages /disadvantages of electronic surveys Examinations, Assessments, and Tests (see table 3-2) Self-Evaluations (disadvantage—people are not always about self) Archival or Existing Data (see table 3-3)
Identify sources of estimating and reporting error and best practices for avoiding these errors. Examples or error types in evaluation: On surveys – some answers may not be completed. On surveys – respondent may not understand directions Data entry errors If identify out-of-range conditions (results), may eliminate. Paraphrasing can sometimes change the meaning. Evaluator bias – being too emotionally involved in the evaluation, so inclined to shape the results for desired results. Best practices – bullet points pp. 79 – 80. Recognize bias / influences Be open to findings Generate options for recommendations and next steps Review data from multiple perspectives Include demographics when people focus of analysis Organize the data by process Group issues and compare to other data findings Analyze data from the perspective of a case study View data with out of your comfort zone approaches Make linkages – evaluation is all about cause and effect Discuss the issues involved with the synthesis of data. Three major tasks are involved with analyzing data: sorting, tabulating, and comparing with summarized data. List the methods of communicating data results to and match each style with an intended purpose. Communication should be timely and targeted to specific audiences. It should communicate so as to be unbiased, modest, and consistent. All communication should reflect a tone of using evaluation data and communication to be supportive, helpful, sincere, and timely. Matching styles: Reflection – thinking data until pattern or like is seen Two-axis Matrix – x-y axes chart. Three-axis Matrix – cube-shaped object with x-y-z axes Flowchart – shows process flow, decision points, input-output Events network – visual of what should be Dichotomy – This or That categorization Argumentation – similar to debate, aimed at resolving two or more theses/positions. Ends with best possible hypothesis & its supporting logic Graphic models – organizational charts, diagrams, maps… Makes information visual and easy to understand.