What were the costs? What were the cost-benefits? Adapted from: Stufflebeam, D. (2001). Evaluation models. New directions for evaluation, 2001(89), 7-98. Evaluation Plan Example #1: UTSA DHS Evaluation Plan Logic Model Component Indicator How Collected When Collected Activities (Management Focus) Developed Long-Range Strategic Plan (P2 g) Communication Goals (P2 g) Outreach Goals (P2 g) Compliance; Stakeholder Perceived Quality Review of Project Documents; Interviews with Early Career Awardees and Mentors Dec 2014; Aug 2015; Aug 2016; Aug 2018 Established Evaluation Metrics (P2 h) Stakeholder Perceived Quality Interviews with Early Career Awardees and Mentors Sept 2014; Annually in August Thereafter Establish an Enduring and Comprehensive Program of Study (P2 a) Compliance with Project Plan; Perceived Quality Review of Project Documents; Interviews with Early Career Awardees, Mentors and DHS Scholars Fall 2014 for launch in Spring 2015; Annually in August Thereafter Advisory Group Meetings Compliance with Project Plan; Impact of Meetings Review of Project Documents; Interviews with Early Career Awardees and Mentors Biannually (August and February – Tentative) PI Travel Compliance Interviews with Early Career Awardees and Mentors Aug 2015; Annually in August Thereafter Consulting for Curriculum Development Compliance; Stakeholder Perceived Quality Interviews with Dr. Nikki Austin (Curriculum Consultant), Early Career Awardees and Mentors Fall 2014 and as needed with Advisory Board and Mentor Consultation Establish Meaningful Partnerships with COEs (P2 e) Compliance; Issues to Be Resolved Interviews with COE PI’s, Early Career Awardees and Mentors Dec 2014; Aug 2015; Annually in August Thereafter Implemented Long-Range Strategic Plan (P2 g) Communication Goals (P2 g) Outreach Goals (P2 g) Compliance with Plan Created by the UTSA Grant Faculty/Staff Review of Project Documents; Interviews with Early Career Awardees and Mentors Spring 2015; Annually in August Thereafter
Courses Perceived Quality Survey of Early Career Awardees and Mentors and Administrators Aug 2015; Aug 2015; Annually In August thereafter Internships Quality of Internships Survey of UTSA Faculty and Students; Phone Interviews with Host Companies Aug 2015; Annually In August thereafter Research Plan Perceived Quality Survey of Early Career Awardees and Mentors and Aug 2015; Aug 2015; Annually In August thereafter Research papers and publications (P2 i) Progress Indicator (e.g., number planned, number in draft) Review of Project Documents; Interviews With Early Career Awardees and Mentors (possibly DHS Scholars) Aug 2015; Aug 2015; Annually In August thereafter Outcomes & Impacts Students Benefit from Program Activities (P2 d and P2 l) Student Certifications Undergraduate Research Undergraduate Experiences Compliance; Participation Levels; Perceived Quality Review of Project Documents; Interviews With of Survey of Early Career Awardees and Mentors and Aug 2015; Aug 2015; Annually In August thereafter Eligible Homeland Security Workforce (P2 b) Perceived Quality Survey of Early Career Awardees and Mentors and Aug 2016, Aug 2018, Aug 2020 Outlets and Career Paths (P2 n) Employment: DHS, Fed Lab, Graduate School Perceived Quality Survey of Early Career Awardees and Mentors and Aug 2016, Aug 2018, Aug 2020 Evaluation Plan Example #2: UTSA NSF Noyce Activity, Output /Product, Impact Indicator How Collected When Collected Activities Workshop - CEMM PD for 1st Cohort Master Teaching Fellows (MTFs) Compliance with Project Plan; Perceived Quality; Self-Reported Learning (1st MTFs) Evaluator Review; Interviews With MTFs and UTSA Faculty/Staff; Survey Fall 2014 Workshop - Action Research PD for 1st Cohort Mentors Compliance with Project Plan; Perceived Quality; Self-Reported Learning (1st MTFs) Evaluator Review; Interviews With MTFs and UTSA Faculty/Staff; Survey Summer 2015 Workshop - CEMM PD for 2nd Cohort Mentors Compliance; Perceived Quality; Interviews With MTFs and UTSA Fall 2015
Self-Reported Learning (2nd MTFs) Faculty/Staff; Survey Workshop - Action Research PD for 2nd Cohort Mentors Compliance; Perceived Quality; Self-Reported Learning (2nd MTFs) Interviews With MTFs and UTSA Faculty/Staff; Survey Summer 2016 Workshop - MS&T Teacher Presentations Compliance; Perceived Quality; Interviews UTSA Faculty/Staff Spring 2017 Workshop - CEMM PD by Mentors for Mentees Compliance; Perceived Quality; Self-Reported Learning (Mentees) Interviews With MTFs and UTSA Faculty/Staff; Survey of Mentees Fall 2017 Workshop - Action Research PD by Mentors for Mentees Compliance; Perceived Quality; Self-Reported Learning (Mentees) Interviews With MTFs and UTSA Faculty/Staff; Survey of Mentees Spring 2018 Workshop - MS&T Teacher Presentations Compliance; Perceived Quality; Interviews UTSA Faculty/Staff; Summer 2019 Workshop - MS&T Teacher Presentations 2nd Cohort Compliance; Perceived Quality; Interviews UTSA Faculty/Staff; Spring 2020 CEOP Use by Mentors Compliance; Perceived Quality; Interviews With MTFs and UTSA Faculty/Staff Spring 2015 - Spring 2019 Systemic Intentional Study by Mentors Compliance; Perceived Quality; Interviews With MTFs and UTSA Faculty/Staff Fall 2015; Fall 2016 Campus Systemic Intentional Study by Mentors Compliance; Perceived Quality; Interviews With MTFs and UTSA Faculty/Staff Summer 2018; Summer 2019 Content Specific Training (CST) Compliance with Project Plan; Perceived Quality; Self-Reported Learning Interviews With MTFs and UTSA Faculty/Staff; Survey End of Each Semester SAMSEC Compliance with Project Plan; Perceived Quality; Self-Reported Learning Post-Session Survey Summer 2015 Summer 2016 Summer 2017 Summer 2018 Summer 2019 Summer 2020 Observations of Mentors Compliance; Perceived Quality Interviews UTSA Faculty/Staff End of Each Semester Participation in CofP Compliance; Perceived Quality Evaluator Review Spring 2015 and periodic follow up Recruitment & Selection of 1st cohort Compliance; Perceived Quality Interview with UTSA faculty/staff Summer/Fall 2014 Recruitment & Selection of 2nd cohort Compliance; Perceived Quality Interview with UTSA faculty/staff Spring/Summer 2015 Iterative Development Phase I Compliance; Perceived Quality Interview with UTSA faculty/staff End of Year 1 Iterative Development Phase II Compliance; Perceived Quality Interview with UTSA faculty/staff End of Year 2