M1 Productivity & Satisfaction Survey (Pre-test) TW1TW2TAI1 M2 Reaction Survey TW3TW4TAI2 M3 Reaction Survey TW5TW6TAI3 M4 Reaction Survey M5 Outcome Survey M6 Productivity & Satisfaction Survey (Post-test) M7 Focus Group T = Component Treatment; W = Workshop; AI = Application Intensive; M = Measurement Data Collection Plan and Instruments The Data Collection Plan provided in Table 3 provides a framework to guide the data collecting portion of the evaluation. Instruments are provided and may be modified to fit the exact situational factors that may present in the client’s environment. All instruments include specifics about the purpose and the “who, what, where, when” of how we suggest the data be collected. Data Collection Plan Table 3: Data Collection Plan Evaluation Level Who (Data Source) Objective Evaluation Instrument Key Question(s) Addressed Timing Responsible Party Context Cohort 1 Mgt. participants What is missing, what is needed to support FD’s leadership development goals? Is the FLP comprehensive enough to meet the of FD’s leadership goals? Focus Group Skills Big Picture Thinking Leadership Pipeline 4 months following Application Intensive 3 (Post) Focus Group facilitator to be designated by Financial Training Director, VUMC Input Cohort 1 Mgt. participants What leadership resources are available as a result of the program effort? What supports are necessary to reinforce the changes after the program is completed? Focus Group Skills Big Picture Thinking Leadership Pipeline 4 months following Application Intensive 3 (Post) Focus Group facilitator to be designated by Financial Training Director, VUMC Reaction All workshop participants What do participants perceive regarding the FLP’s relevance and timeliness? What do participant perceive regarding their ability to apply skills learned back on the job? Reaction Survey Skills Immediately following each Application Intensive Director, VUMC Financial Training or their designee Outcome All workshop participants How well did the FLP enhance the leadership skills and big picture thinking capabilities of the joint finance department managers? Outcome Survey Skills Big Picture Thinking 3 months following Application Intensive 3 Director, VUMC Financial Training or their designee
survey to a comparable group of Finance Department employees whose managers are not participating in the FLP (i.e. control group). The control group results would then be analyzed as described herein and compared to the results of the employees whose managers are participating in the program. The expectation of results would be that the overall productivity and satisfaction rates would be higher in the FLP group than in the control group. If results trend together then it is indicative that productivity and satisfaction levels of employees whose managers participate in the FLP are not being significantly influenced by their managers’ participation, regardless of the levels reported. Instrument 2 - Reaction Survey Purpose: The results of this survey may be used to evaluate for the levels and trends for initial reactions of FLP participants. The FLP Logic Model assumes that 75% of participants are applying skills in the work setting at a Strongly Agree or Agree level; therefore, the reaction survey results will provide the anticipated use from participants that should align with the 75% that will use the skills. Decision makers may use the results of this survey in conjunction with the results of the Focus Group, Outcome Survey, and Productivity & Satisfaction Survey results to gain a comprehensive view of the outcomes resulting from the FLP. Coding & Data Analysis: The following describes how to code and analyze the data captured by the Reaction Survey. Questions from this survey are bucketed into two primary categories: Relevancy & Timeliness Indicator (1-3) The content of these two workshops are relevant to leadership skills in my position. The FLP content is relevant for all finance operations at Vanderbilt. The instructor gave the content in a timely manner. Anticipated Ability to use Tools Indicator (4-10) I can see the value that applying the FLP will have on other departments The skills learned in the FLP increase my ability to understand the impact that my work has on others. I now have more understanding of the purposes and structure of other departments The FLP is going to have positive impact on my ability to partake in culture change I am confident that I will use these skills in the workplace The application intensive of this part of the FLP is relevant to my position I think that consistent application of the skills taught will have a positive impact on: My direct reports’ relationship with me My ability to work collaboratively with other leaders in the FD My relationship with the department that depend on my work (my customers) My work productivity My own self confidence as a leader
Only in the short term / temporary Improved customer satisfaction rating/ customers noticing a positive difference Improved employee satisfaction rating/ employees more engaged, motivated Improved individual performance review ratings/ better work performance, products Employees more confident to apply for higher leadership positions/ employees feel more empowered Other My final question to each of you is this: If you were king or queen of Vanderbilt for a day and you could make any changes you wish to the FLP or any aspect of leadership development in Finance, what would you do? (Feel free to think big here…money and time are no object). Embedding with customers for bigger picture thinking/short-term assignments at customer location Have one finance department instead of two/ combine our teams/cross training Ice cream Fridays/ social events/ social media contents The Advisory Board facilitator become a full-time trainer/ add staff person dedicated to leadership learning Open up the FLP to all Finance employees/ other departments FLP should be all action learning with little or no classroom learning/ real world work project cases to work on together as Cohort Virtual online leadership courses available 24/7 Focus Group Coding During the interview, Focus Group participants may comment on multiple ideas or themes in their responses. Figure 10 below provides a sample of how evaluators should be diligent to watch for this and to code multiple ideas appropriately. Figure 10: Focus Group Coding Example Presentation Plan We believe our client is best served by providing the plan in Table 10 for communicating results of the evaluation plan to the appropriate parties. For our purposes, the two stakeholder groups identified as communication targets for the evaluation results will be Cohort 1 participants and Brett Sweet & the VU Finance Executive team. Presentations with Cohort 1 will focus on communicating survey results, initial implications in terms of programmatic decision-making, and soliciting verification of recommendations to be brought to Brett Sweet and the executive team. Presentations to the Executive