Table 1 Sources of Validity Evidence Summarized in this Supplemental Report for PASA Reading, Math, and Science Description Evidence Validity Concern Findings Uses/Changes Proficiency Study Item frequencies of proficient A students taking B and proficient B students taking C items Construct were some items from higher level test that students could do. Students performed less well on items from higher test level than on items at their assigned test level Study provided preliminary information about student performance on higher level items, but only proficient students were assessed. Prompted a large scale study as a follow up. Proficiency 2011-12 Appendix B Item frequencies of proficient A students taking B items and proficient B students taking C items Construct were several items from each higher level test form that students could do, including some items for which a large percentage of students (e.g., above 60%) assigned to a lower test level could earn a score of 5. Results will be used in conjunction with results from the 2012-13 proficiency study to examine any needed restructuring of the PASA Additional item level analyses are currently being explored to further examine learning progressions. Proficiency Study 2012-13 Item frequencies of students at each grade level on items within the same content strand across grades Construct Planning Stage, to be conducted in Spring 2013 Results will be used in conjunction with results from the 2011-12 proficiency study to examine any needed restructuring of the PASA LCI Appendix C LCI data from teachers who administered PASA in 2010-11. Utilized subset of LCI responses Consequential Construct PASA students have similar learning characteristics to students taking AA-AAS across states. Some Methods for identifying students for the PASA will remain. Results suggest some differences in A, B and
assessment anchors were phased out of the PASA Science. External Alignment Study Appendix G Panelist ratings Content Majority of assessment content linked to content standards. Some content at certain grade levels not linked. Revision to items and item writing procedures Table 3. cont’d. Description Evidence Validity Concern Findings Uses/Changes Impact of PASA Survey Appendix H Consequential validity survey Consequential Three-hundred twenty one teachers completed a consequential validity survey, 131 in year 1, 51 in year 2, and 139 in year 3. Results indicated that the AA-AAS does impact instructional practices, but not comprehensively. Lead to development of Leadership Institute and recommendations for professional development and training. A follow up enacted curriculum study is being planned for 2012-13. Classroom Artifact Appendix H Artifact collection, coding, and analysis Consequential Fifty-eight teachers provided 152 artifacts. Majority were science, linked to standards, reached DOK of Levels 1and 2 Lead to development of Leadership Institute and recommendations for professional development and training. A follow up enacted curriculum is being planned for 2012-13.
Focus Group of Teachers of SWSCD Study Focus groups data from teachers of science to students with significant cognitive disabilities Consequential Twenty seven participants provided information about collaboration, content, instruction and challenges in teaching science to SWSCD Lead to development of Leadership Institute and recommendations for professional development and training
Comparisons of Proficiency Status of Students as Determined by Rather 1-2008, Rater 2-2008, and Rater 2009 who Used the Revised Rubric Reading Math Reading 1st Score Reading Reliability Reading Rescore 2009 Math 1st Score Math Reliability Math Rescore 2009 34A Emerging 4 3 6 7 7 10 Novice 10 13 13 11 11 13 Proficient 11 9 6 21 21 16 Advanced 0 0 0 0 0 0 34B Emerging 2 3 6 1 1 1 Novice 21 20 20 4 4 5 Proficient 3 3 0 3 3 2 Advanced 0 0 0 0 0 0 34C Emerging 2 1 2 2 1 3 Novice 2 3 3 6 6 6 Proficient 2 2 1 2 3 1 Advanced 0 0 0 0 0 0 56A Emerging 3 2 4 3 2 4 Novice 13 11 6 6 8 Proficient 8 11 8 15 16 12 Advanced 0 0 0 0 0 0 Table 4. Cont’d. Reading Math Reading 1st Score Reading Reliability Reading Rescore 2009 Math 1st Math Reliability Math Rescore 2009 56B Emerging 2 2 4 3 2 4 Novice 16 17 16 16 17 16 Proficient 9 8 7 8 8 7 Advanced 0 0 0 0 0 0 56C Emerging 0 0 0 1 1 1 Novice 3 3 7 2 3 5
Conclusions: Results from the scoring of 3s and 4s study indicate that the scores are being assigned for a variety of reasons according to the rubric. One concern is that students are receiving a score of four on an item when the teacher has to re-prompt the student but does not provide additional information. Students who consistently receive scores of three on items will not meet the numeric cut score to be called proficient or higher on the assessment. The rubric was not modified after this study. Section 2: Additional Validity Evidence for the PASA Reading Criterion Related Evidence, PASA Reading Purpose: The primary purpose of this was to extend previous work that used teacher estimates to describe the reading skills of students with significant cognitive disabilities who participate in the AA-AAS by gathering data on actual reading performance for a group of participating students in one northeastern state. Additionally, we aimed to examine relationships among measures of early reading, performance on the AA-AAS, demographics, and disability. To meet these aims, we administered two types of reading CBM (e.g., grade K-3 word reading fluency and grade 1-5 passage reading fluency). The research questions that guided the project were: How do students with significant cognitive disabilities (i.e., students assigned by their IEP team to take the AA-AAS) perform on early grade word and passage reading CBM?