Acknowledgments WestEd wishes to recognize those individuals who supported our work in the field with Commonwealth educators and administrators and who contributed substantively to the development of this report: Tim Broadrick, Technical Director, Shawsheen Valley Regional Vocational Technical School District Tom Hickey, Superintendent-Director, South Shore Vocational Technical High School Ron Noble, Educator Evaluation Project Lead, Massachusetts Department of Elementary and Secondary Education (ESE) Maura Russell, Career/Vocational Technical Education Unit, ESE We would like to thank Tom Belland and Andrew Morrison for permission to use the image from Blackstone Valley Regional Vocational Technical High School for the cover of this report. We also would like to thank all of the career/vocational technical education (CVTE) educators and administrators from across the state who participated in meetings held in January and February 2014 and submitted sample assessments that are featured in this report: Career Cluster Educators and Administrators District/Organization Agricultural and Natural Resources William Florek Aaron Polansky Janis Gentile Sarah Ard Dhimitri Skende Berkshire Hills Bristol County Agricultural Lower Pioneer Valley Education Collaborative Minuteman RVT Worcester Technical High School Arts and Communication Michele Watson Maxwell Barbara Jean Chauvin Michael Azevedo John Brochu Leslie Weckesser Cambridge Rindge & Latin School Norfolk County Agricultural Shawsheen Valley RVT South Middlesex RVT Southeastern RVT Business and Consumer Services Diane Potter Deborah Lagasse Maureen Griffin Bristol-Plymouth RVT Greater Lowell RVT Greater Lowell RVT
Cheryl English Rosemary Slattery Elizabeth Alicea Beverly DeSalvo Leanne Lyons John Foisy Andrea Deane Peabody High School Peabody High School Southern Worcester County RVT Whittier RVT Whittier RVT Worcester Technical High School Worcester Technical High School Construction Eileen Homen Thomas Aubin John Hardacre Oscar Amaral Thaddeus Haggerty Russell Esau John Manetta Peter Maxfield Edward Carll Jeremy Guay Matthew Gomes James Smith Attleboro High School Greater Fall River RVT Greater Lawrence RVT Greater New Bedford RVT Greater New Bedford RVT Leominster Center for Technical Education Medford Vocational HS Montachusett RVT South Shore RVT Southern Worcester County RVT Westfield High School Worcester Technical High School Education Susan Edmonds Karla Boudreau Donald Jarvis Michelle Roche Julie Bowen Attleboro High School Greater Lowell RVT Lower Pioneer Valley Education Collaborative Minuteman RVT Weymouth High School Health Services Sheri Membreno Emily Lebo Diane Norkiewicz Linda Cote Rosemary Pisani Erin Marcotte Susan Lahtinen-Phelps Kathleen Holman Deborah Collins Attleboro High School Boston Public Schools Greater Lawrence RVT Greater Lowell RVT Greater Lowell RVT Greater New Bedford RVT Leominster Center for Technical Education North Shore RVT South Shore RVT Hospitality and Tourism Mike Ananis Caroline Freitas Cambridge Public Schools Cape Cod Regional
If using comparable versions of a test, need to ensure that measures are not so different that changes in performance may be due to variations in the measures rather than to actual student growth. Time interval between tests must be explicit and intentional. It may be based on time (e.g., after ten weeks of instruction) or curriculum (e.g., after 25 lessons). If linked assessments are scored on a common scale, scores yielded should represent the full range of performance at the beginning and the end of school year. Common Approaches to Measuring Growth: Repeated Measures Assessment Description: With a repeated measures assessment, one or more assessments are administered multiple times to get information about how students performed at different points in time. This approach generalizes gain scores from multiple time points through the calculation of an average gain or slope (learning trajectory). Districts interested in this option may use traditional or measures as repeated measures assessments. Table 5. Strengths and Limitations of Repeated Measures Assessment Strengths Limitations Allows educators to use current assessments that are already administered multiple times during the school year. Generally relatively short and deeply integrated into the curriculum. Provides information to educators about improvements at various times across the year. Educators receive continuous feedback about students, and can modify their practices accordingly. Growth score can be generated using methods ranging from simple to sophisticated. Strengths of pre-test/post-test are relevant to this approach. However, precision of growth estimate generally increases using multiple time points. Requires authentic tasks so that improvement over time reflects growth rather than familiarity with the assessment. Requires educator capacity to record and organize multiple data points for each student. Intervals between testing should be consistent across students and over time. Limitations of pre-test/post-test are relevant to this approach e.g., assumes a common scale that allows scores to be compared across different points in time and assumes score differences to be interpretable as representing increased mastery of specific knowledge or set of skills. traditional
In fact, we have even started trying our hand at using , or curriculum-embedded performance assessments, that we have borrowed and modified from the (MCUs) for Culinary Arts on Food Safety and Sanitation, provided by ESE. During future professional development days, we hope to guide educators toward refining instruments that will share common protocols for administering, scoring, interpreting, and reporting across the district for all Chapter 74 programs. But for our maiden voyage with DDMs, we have chosen two measures that address previously identified achievement gaps across our student body in the domains of career readiness, specifically employability skills, and literacy, more generally. DDM #1: A Measure of Communications Skills DDM #1 is a holistic evaluation that makes use of a writing rubric developed by to evaluate growth in writing and argumentation abilities among all of Tri-County’s grade 10 students (approximately 210 students). Ms. George’s preferred approach to moving DDM efforts forward is to engage faculty in deep thinking during professional learning opportunities and to leverage the meetings that she currently has with CVTE administrators and educators across the Commonwealth as opportunities to discuss DDM development and implementation options. Her recent efforts have been focused on research-based strategies for teaching and assessing writing, with the goal of preparing educators for DDMs that measure students’ growth in literacy. The approach has been a boon to shop educators who are strong technical educators but often lack the educational background to feel fully comfortable and confident in teaching embedded writing skills as part of technical skills attainment. The standardized approach to writing instruction and evaluation provided by the Collins Writing Program, Ms. George discovered, has offered CVTE educators the scaffolding, tools, and supports that they need to foster greater communication fluency among their students. Educators in the academic content areas initially expressed reservations about the prescriptive, “process-focused” techniques that Collins recommends, but after working with the new rubric, they have come to appreciate the shared vocabulary and holistic approach to evaluating writing. In addition, now that dialogue among academic and CVTE educators is more focused and directed toward improving literacy skills as a shared responsibility, these educators have moved toward exploring a collaboration in the rating and scoring process. Now, in many instances, both an academic educator and a CVTE educator rate writing assignments using a set of common holistic rubrics that establish performance levels at every grade and competency level.