HANDOUT #4.1 Module 4.1: Establish a Customized Standard-Setting Approach Standard-setting is the process of developing “cut scores” and performance level descriptors that identify performance at established points along the score. The modified contrasting group (MCG) approach is an examinee-centered method in which panelist classify the performance of students based upon both their knowledge of the students’ skills, knowledge, and abilities in conjunction with those expectations articulated in the content standards 4.1 Workflow: Establish a Customized Standard-Setting Approach 4.1 Procedural Steps: Modified Contrasting Groups (MCG) Approach 4.1.1 Design 4.1.1.a Preparation Identify targeted content standards and associated skills that the selected assessment evaluates. Gather a group of student samples to be reviewed by a qualified panel of scorers. The student samples should represent a wide range of performance/achievement. Determine criteria characteristic of a “proficient” performance or a display representative of content mastery, and the minimum requirements for “passing” the assessment or task. Record these criteria in the Design portion of the Establish a Customized Standard-Setting Approach Framework. Given the “proficient” guidelines, determine criteria characteristic of “advanced” performance and “below proficient” performance. “Advanced” performance indicates content mastery that exceeds the desired value, while “below proficient” performance indicates a significant and unsatisfactory deficit in demonstrating content mastery. Record these criteria in the Design portion of the Framework. Assign a performance level to and state a rationale for each selected student sample. Record these in the Framework. 4.1.1.b Approach Set a date for and notify potential panelists of a preliminary meeting that will evaluate the student work samples and train panelists in scorer calibration. (Note: procedures for scorer calibration are covered in depth in Module 3.3) Since this process relies almost solely on human judgement, a diverse group is essential. Panelists should have experience with the subject matter/grade level being evaluated, as well as with students participating in this subject and grade level. Subject matter experts,
completely and thoroughly. responses are more indicative of a 6th or 7th grader than a 5th grader. student’s performance exceeds the expectations of placed upon a typical 5th grade student. Deviation 0 0 -1 Identified Issues N/A N/A This panelist was not aware of and did not review the rubrics for the constructed response section. Calibration Training-Student Sample #2 Anchored Performance Level Proficient Panelist Panelist 1 Panelist 2 Panelist 3 Performance Level Assigned Proficient Proficient Proficient Performance Level Rationale This sample is also impressive, but much less so than the “Advanced” sample. The multiple choice section is nearly perfect, but the writing section bogs down this student’s sample. This sample seems to be in the “middle of the pack.” Though this student misses some questions, especially in the constructed response section, they are questions that I would not expect most 5th graders to answer correctly, while most of the correct answers are indicative of typical 5th grade ELA knowledge. Deviation 0 0 0 Identified Issues N/A N/A N/A Calibration Training-Student Sample #3 Anchored Performance Level Below Proficient Panelist Panelist 1 Panelist 2 Panelist 3 Performance Level Assigned Proficient Below Proficient Below Proficient Performance Level Rationale This sample contains more errors than the last, but the knowledge displayed is still satisfactory for a 5th grader. Inconsistent thought process in evidence-based SR question clusters; this is indicative of reading comprehension. Nearly 50% multiple choice questions and about 75% of constructed response questions are answered incorrectly. Deviation +1 0 0 Identified Issues This panelist did not examine sequences of incorrect answers; incorrectly answered N/A N/A
Calibration Training-Student Sample #2 Anchored Performance Level Proficient Panelist Panelist 1 Panelist 2 Panelist 3 Performance Level Assigned Proficient Advanced Advanced Performance Level Rationale This sample contains mostly correct answers, but the overall performance is “average” for a 7th grade student. All questions are answered completely and all answers stay on-topic. Student displays knowledge of passage content through writing. Answers are not perfect but remain on-topic. Deviation 0 +1 +1 Identified Issues N/A Panelist 2’s rationale is sound, but this sample is not indicative of an above-average 7th grade student. Panelist 3’s rationale is sound, but this sample response is not indicative of an above-average 7th grade student. Calibration Training-Student Sample #3 Anchored Performance Level Advanced Panelist Panelist 1 Panelist 2 Panelist 3 Performance Level Assigned Advanced Advanced Advanced Performance Level Rationale Almost no errors. Answers are on-topic and do not stray from the given prompts. The student’s annotation of the reading passages is evidence of higher-order thinking and organizational skills. Answers are thorough and mostly error-free. Answers stay and all parts of all prompts are addressed and expanded upon. Deviation 0 0 0 Identified Issues N/A N/A N/A Build Phase-Meeting 2 Meeting Date: August 14, 2014 Panelists/Qualifications: L. Sanchez (Middle School Assistant Principal), A. Cooper (Middle School Instructional Coach), C. Graham (Middle School Instructional Coach) Subject/Grade Level and Assessment/Task: Placement Exam for Pilot Enrichment Grade 7 Social Studies Course Performance Levels and Descriptors-Draft Set (from Meeting 1) Advanced Proficient Below Proficient Students at this level demonstrate a highly developed ability for their grade level to interpret, evaluate, analyze, and synthesize Students at this level demonstrate a satisfactory ability for their grade level to interpret, evaluate, analyze, and synthesize information from a Students at this level demonstrate a partially developed ability for their grade level to interpret, evaluate, analyze, and synthesize information
information from a variety of primary and secondary sources in order to demonstrate an understanding of key social studies content and concepts. Students at this level demonstrate a highly developed ability for their grade level to apply social studies content and concepts to real-life situations by transferring knowledge from the abstract to the concrete level. variety of primary and secondary sources in order to demonstrate an understanding of key social studies content and concepts. Students at this level demonstrate a satisfactory ability for their grade level to apply social studies content and concepts to real-life situations by transferring knowledge from the abstract to the concrete level. from a variety of primary and secondary sources in order to demonstrate an understanding of key social studies content and concepts. Students at this level demonstrate a partially developed ability for their grade level to apply social studies content and concepts to real-life situations by transferring knowledge from the abstract to the concrete level. Panelist Anchor Set Assembly-Round 1 (Proficient Level) Student Samples (#) 2, 7, 9 Panelist Panelist 1 Panelist 2 Panelist 3 Student Sample Number 9 9 2 Performance Level Rationale This sample’s answers are adequate, but do not exceed expectations. Answers are on-topic but ideas are not developed thoroughly. This response’s answers are relevant to the questions but not in a manner that can be considered above average. Deviation from Anchored Performance Level 0 0 0 Identified Issues N/A N/A N/A Panelist Anchor Set Assembly-Round 2 (Advanced Level) Student Samples (#) 3, 6, 8 Panelist Panelist 1 Panelist 2 Panelist 3 Student Sample Number 6 3 8 Performance Level Rationale This sample is mainly error-free; all parts of all prompts are answered thoroughly and correctly. The answers in this response display content knowledge and writing skills that exceed those of a typical 7th grader. The student’s answers are correct and written in a precise manner that reflects deep understanding of the content in the passages. Deviation from Anchored Performance Level 0 0 0 Identified Issues N/A N/A N/A Panelist Anchor Set Assembly-Round 3 (Below Proficient Level) Student Samples (#) 1, 4, 5 Panelist Panelist 1 Panelist 2 Panelist 3 Student Sample Number 4 4 5 Performance Level Rationale The test is mostly blank. Student hardly attempts to answer any questions. The student’s answers are lengthy, but are irrelevant to the questions asked on the test. Deviation from Anchored Performance Level 0 0 0
Increase the depth and breadth of knowledge required to achieve each performance level across time spans (e.g., a higher level of knowledge needed at the end of a unit as opposed to at its beginning) and across grade levels. 4.2.1 Reading READING – GRADES 3-5 Performance Level Descriptors Advanced Proficient Below Proficient Students at this level, when using grade-appropriate text, will consistently, effectively, and/or skillfully use the knowledge and skills articulated in the state content standards for reading. Students at this level, when using grade-appropriate text, will frequently, accurately, and/or satisfactorily use the knowledge and skills articulated in the state content standards for reading. Students at this level, when using grade-appropriate text, will inconsistently, inadequately, and/or partially use the knowledge and skills articulated in the state content standards for reading. Content Summary Expectations: Grades 3-5 Apply a wide range of strategies to read independently Identify author’s purpose based on text organization and content Use phonics and word recognition to decode and understand new words when reading Use meaning and knowledge of words to develop and increase reading vocabulary Demonstrate comprehension before, during, and after reading by retelling, restating, and summarizing, as well as by supporting and expanding ideas and concepts Demonstrate oral and silent reading fluency of grade-level texts Compare ideas and differentiate fact from opinion within and among texts Connect information with relevant prior knowledge to make predictions Make inferences and draw conclusions about content, ideas, and author’s decisions Distinguish between essential and nonessential information; read and respond to essential information Read and understand, identify and analyze, and compare the use of various literary genres Identify and interpret basic figurative language and literary devices; differentiate between literal and meanings Listen critically to others and respond with grade-level appropriate questions or ideas Use appropriate clarity and volume when speaking formally Identify differences in formal and informal language in speech, writing, and literature Explain and conduct inquiry-based research on topics using a variety of teacher-guided media sources and strategies Produce a product that presents findings, gives reasonable conclusions, and cites sources Understand and evaluate media for informational and entertainment purposes; use media and technology to extend learning