Edwin Analytics, school and district leaders encouraging collaboration with other staff members on the use of Edwin Analytics, and school and district leaders encouraging staff to use Edwin Analytics to help in decreasing achievement gaps. Slightly fewer respondents (24 percent) reported being moderately or extensively supported by school and district leaders providing ongoing training for use of Edwin Analytics, possibly suggesting ongoing training was either not effective or not valuable to Analytics users. Figure 27. Factors Supporting Use of Edwin Analytics (N = 135) Source: Edwin Analytics Survey Interviews with district administrators, however, suggest that districts have not been quick to give teachers access to the system, which is now an option. One reason for this is that they must designate each user’s access type individually, a long process in large or even moderately large districts. Furthermore, three of the district administrators expressed concern that the system would be too complicated to expect teachers to use.
Directions: After the observation, use your descriptions to answer the following questions. 1. The trainer articulates the state’s goals for the rollout of Edwin T&L. Yes No If yes, note: goals that the trainer articulates any reactions/discussion/pushback from participants 2. The trainer discusses expectations for how districts will implement Edwin T&L. Yes No If yes, note: expectations that the trainer articulates any reactions/discussion/pushback from participants 3. Participants discuss districts’ goals for the adoption of Edwin T&L. Yes No If yes, note: goals that participants discuss any discussion of how the district goals for Edwin relate to state goals for Edwin 4. The trainer leads a discussion about participants’ expectations for how educators in their districts will use Edwin T&L. Yes No
Directions: After the observation, use your descriptions to answer the following questions. 1. Participants discuss district’s goals for the adoption of Edwin T&L. Yes No If yes, note: goals that participants discuss any discussion of how the district goals for Edwin relate to State goals for Edwin 2. The trainer leads a discussion about participants’ expectations for how educators in their districts will use Edwin T&L. Yes No If yes, note: expectations that participants discuss any discussion of how the district expectations for Edwin relate to State expectations for Edwin 3. Participants ask about how Edwin Analytics can be used in conjunction with Edwin T&L. Yes No If yes, note: participants’ questions the trainer’s response(s) 4. The trainer discusses how Edwin T&L can be used to address achievement gaps in student achievement (e.g., tools for differentiating instruction, disaggregated Yes No
I do not yet understand how to use the Curriculum Tools. I do not know how to modify lessons in the Model Curriculum Units. The Model Curriculum Units do not pertain to my grade level or subject area. The district’s priorities for ET&L do not emphasize Curriculum Tools. Digital Resources This section is about your use of ET&L’s Digital Resources (PBS Learning Media) this year. How often do you use the Digital Resources (PBS Learning Media) to do the following? Never Once or Twice This Year A Few Times per Semester A Few Times per Month A Few Times per Week Review digital resources that are available Link digital resources while creating an assessment or instructional resource in ET&L Include digital resources in a lesson plan How difficult or easy was it to do the following in the Digital Resources (PBS Learning Media)? Very Easy Moderately Easy Neither Easy Nor Difficult Moderately Difficult Very Difficult Not Applicable/ Never Used Review digital resources that are available Link digital resources while creating an assessment or instructional resource in ET&L Include digital resources in a lesson plan To what extent does the following statement represent a barrier to your school’s use of ET&L Digital Resources (PBS Learning Media)? Not a Barrier A Slight Barrier A Moderate Barrier An Extensive Barrier I have difficulty finding Digital Resources to support my work. The Digital Resources available are not relevant to the subject matter I am teaching. The videos in the Digital Resources don’t
Appendix F. ET&L and Analytics Survey Frequency Tables I. ET&L Teacher Survey Respondent Background Table F1. Grades Taught by Survey Respondents, N = 153 Which grade(s) do you teach? (Check all that apply.) Percentage Prekindergarten 2.6% Kindergarten 17.6% First 20.9% Second 22.2% Third 27.5% Fourth 28.1% Fifth 31.4% Sixth 17.6% Seventh 13.7% Eighth 11.8% Ninth 11.1% Tenth 12.4% Eleventh 14.4% Twelfth 14.4% Ages 18–22 students 0.7% Note: Percentages may exceed 100 because respondents could choose more than one response. Table F2. Survey Respondents’ Primary Job Role, N = 158 How would you define your primary job role? (Choose one.) Percentage General education teacher 58.2% Academic coach 8.9% Special education teacher 6.3% Specialist 6.3% Instructional leader 4.4% English language learner (ELL) teacher 3.8% Department chair 3.2% Technology coach 3.2%
District administrator 1.3% Data coach 1.3% Principal 0.6% Assistant principal 0.6% Dual language learner (DLL) teacher 0.0% English as a second language (ESL) teacher 0.0% Career, vocational, and technical education teacher (CVTE) 0.0% Other role* 1.9% *Guidance counselor (1.3%), special education coordinator (0.6%) Definition and Communication of ET&L Goals, Timelines, and Excepted Uses Table F3. Awareness of the State’s Purpose for Developing ET&L, N = 157 To what extent are you aware of the State of Massachusetts’s purpose for adopting ET&L? Percentage Not at all aware 7.6% Minimally aware 33.1% Moderately aware 50.3% Extremely aware 8.9% Table F4. Beliefs of the State’s Purpose for Development of ET&L, N = 156 Which of the following do you believe are the State of Massachusetts’s major purposes for developing ET&L? (Check all that apply.) Percentage Provide curriculum and assessment tools in a single system. 76.9% Provide tools that help teachers assess student mastery of standards. 69.9% Providing tools and resources to support teaching and learning 61.5% Provide timely and actionable information on students. 60.3% Facilitate the transition to the Common Core State Standards. 59.0% Provide access to use and customize the Model Curriculum Units. 46.2% I am not aware of the State’s purposes for developing ET&L. 6.4% Other* 3.8% Note: Percentages may exceed 100 because respondents could choose more than one response. *For example: To support curriculum development with resources, to evaluate teachers, to have a universal database for assessment across the state.
Align curriculum with state standards 132 69.7% 20.5% 4.5% 5.3% 0.0% Align instruction with state standards 130 69.2% 20.0% 5.4% 5.4% 0.0% Table F24. Use of Assessment Tool Data/Reports With Others, N = 135 With which of the following people did you review/discuss data or reports from ET&L Student Assessment Tools this school year? (Check all that apply.) Percentage Other teachers 37.0% Team leaders/coaches 32.6% I have not used Student Assessment Tools this school year. 32.6% Not applicable 23.0% School administrators 22.2% Students 8.9% Parents 2.2% Note: Percentages may exceed 100 because respondents could choose more than one response. Table F25. Barriers to Use of Assessment Tools To what extent does each of the following represent a barrier to your use of ET&L Student Assessment Tools? N An Extensive Barrier A Moderate Barrier A Slight Barrier Not a Barrier I already use another system to create assessments and review student data (e.g., Galileo or TestWiz). 128 14.8% 17.2% 15.6% 52.3% I don’t yet understand how to create assessments in ET&L. 127 15.7% 18.9% 16.5% 48.8% I do not know how to access to test items outside my grade level or my subject area. 127 16.5% 16.5% 22.0% 44.9% I cannot find test items appropriate for my subject area. 130 16.9% 16.2% 22.3% 44.6% I struggle to understand the information in the Student Assessment reports. 125 12.0% 10.4% 21.6% 56.0% The Student Assessment Tools are not yet fully functional. 125 24.8% 16.0% 20.0% 39.2%
Table F26. Frequency of Use of Curriculum Tools How often do you use the Curriculum Tools to do the following? N Never Once or Twice This Year A Few Times per Semester A Few Times per Month A Few Times per Week Utilize Model Curriculum Units available from ESE 135 78.5% 16.3% 3.0% 1.5% 0.7% Search for instructional materials by standard 133 74.4% 16.5% 4.5% 3.8% 0.8% Plan/schedule curriculum delivery 134 82.8% 9.7% 3.7% 3.0% 0.7% Develop or deliver lesson plans 133 84.2% 8.3% 3.8% 1.5% 2.3% Table F27. Ease of Use of Curriculum Tools How difficult or straightforward was it to do the following in the Student Assessment Tools? N Very Difficult Moderately Difficult Neither Nor Difficult Moderately Very NA/ Never Used Utilize Model Curriculum available from ESE 133 11.1% 14.8% 27.8% 16.7% 29.6% 59.4% Search for instructional materials by standard 132 11.7% 13.3% 30.0% 21.7% 23.3% 54.5% Plan/schedule curriculum delivery 132 19.1% 10.6% 27.7% 23.4% 19.1% 64.4% Develop or deliver lesson plans 129 18.2% 11.4% 34.1% 18.2% 18.2% 65.9% Note: As a result of rounding, percentages may not total to 100. Easy to difficult options are calculated out of the sample who indicated use; N excludes those who responded NA. Factors and Conditions That Support or Hinder Use of ET&L
Summer 2013 (July–August, 2013) 0.0% Fall 2013 (September–December, 2013) 28.3% Spring 2014 (January–June, 2014) 26.4% Summer 2014 (July–August, 2014) 0.0% Fall 2014 (September–December, 2014) 13.2% I have not received any training in ET&L yet. 0.0% Table F44. Respondents Type of Training Received, N = 52 Please describe the type of training you received on ET&L. (Check all that apply.) Percentage Webinar and online training provided by Thinkgate and ESE 61.5% Face-to-face spring 2013 training for pilot districts provided by Thinkgate and ESE 26.9% Face-to-face training provided by my district 25.0% Self-paced training using Thinkgate University 21.2% Face-to-face training for other districts provided by Thinkgate and ESE 15.4% Note: Percentages may exceed 100 because respondents could choose more than one response. Table F45. Survey Responses on Aspects of ET&L Trainings How much do you agree or disagree with the following statements about the ET&L training? N Strongly Disagree Disagree Agree Strongly Agree My ET&L trainer was well prepared. 52 1.9% 11.5% 59.6% 26.9% The training had technical difficulties that hindered the presentation. 51 3.9% 49.0% 33.3% 13.7% The training answered questions I had. 51 2.0% 23.5% 64.7% 9.8% The training included information about the online training resources available in Thinkgate University. 53 0.0% 20.8% 62.3% 17.0% The training prepared me to access the system (i.e., how to log in). 53 5.7% 15.1% 62.3% 17.0% The training prepared me to use the system. 53 7.5% 26.4% 54.7% 11.3% Note: As a result of rounding, percentages may not total to 100.
Table F55. Assessments Shared With Other Educators, N = 49 Estimate the number of assessments you have share with other educators in your district. Percentage None 71.4% Fewer than 2 4.1% 3–5 6.1% 6–9 6.1% More than 10 12.2% Note: As a result of rounding, percentages may not total to 100. Table F56. Ease of Use of Assessment Tools How difficult or straightforward is it to do the following in the Student Assessment Tools? N Very Difficult Moderately Difficult Neither Nor Difficult Moderately Very NA/ Never Used Create individual assessment items 44 4.8% 23.8% 23.8% 28.6% 19.0% 52.3% Use MCAS items to create an assessment 44 5.3% 15.8% 21.1% 31.6% 26.3% 56.8% Use NWEA items to create an assessment 44 6.3% 12.5% 18.8% 43.8% 18.8% 63.6% Use Key Data Systems items to create an assessment 44 9.1% 18.2% 27.3% 18.2% 27.3% 75.0% Use other (third items to create an assessment 43 7.1% 28.6% 21.4% 14.3% 28.6% 67.4% Generate standard data reports (e.g., distractor analysis, standards by student) 44 4.3% 21.7% 17.4% 34.8% 21.7% 47.7% Track vocational/technical competencies 40 0.0% 50.0% 0.0% 50.0% 0.0% 95.0% Note: As a result of rounding, percentages may not total to 100. Easy to difficult options are calculated out of the sample who indicated use; N excludes those who responded NA. Table F57. Frequency of Use of Data From Assessment Tools
How often have you used the data from the Student Assessment Tools to do the following this school year? N Never Once or Twice This Year A Few Times per Semester A Few Times per Month A Few Times per Week Identify individual students who need remedial assistance 45 64.4% 20.0% 8.9% 4.4% 2.2% Identify individual students who need extensions or challenges 45 73.3% 11.1% 8.9% 4.4% 2.2% Identify student groups for instruction 44 68.2% 13.6% 11.4% 4.5% 2.3% Identify and correct gaps in the curriculum for all students 45 73.3% 17.8% 2.2% 4.4% 2.2% Create of modify benchmark assessments 45 71.1% 15.6% 6.7% 4.4% 2.2% Align assessments with State standards 45 73.3% 15.6% 4.4% 4.4% 2.2% Adjust instructional strategy or learning format 44 81.8% 4.5% 6.8% 4.5% 2.3% Align curriculum with State standards 45 75.6% 17.8% 2.2% 2.2% 2.2% Note: As a result of rounding, percentages may not total to 100. Table F58. Use of Assessment Tool Data/Reports With Others, N = 44 With which of the following people did you review or discuss data/reports from ET&L Student Assessment Tools this school year? (Check all that apply.) Percentage School administrators 56.8% Team leaders/coaches 54.5% Teachers 45.5% Not applicable 31.8% Students 6.8% Parents 2.3% Other (no one) 2.3% Note: Percentages may exceed 100 because respondents could choose more than one response. Table F59. Barriers to Use of Assessment Tools To what extent does each of the following represent a barrier to your use of ET&L Student Assessment Tools? N An Extensive Barrier A Moderate Barrier A Slight Barrier Not a Barrier I already use another system to create assessments and review student data (e.g., Galileo or TestWiz). 43 4.7% 23.3% 14.0% 58.1%
I don’t yet understand how to create assessments in ET&L. 43 14.0% 18.6% 23.3% 44.2% I do not know how to access to test items outside the target grade level or subject area. 42 11.9% 16.7% 23.8% 47.6% I cannot find test items appropriate for the target subject area. 41 14.6% 14.6% 22.0% 48.8% I struggle to understand the information in the Student Assessment reports. 40 5.0% 10.0% 12.5% 72.5% The Student Assessment Tools are not yet fully functional. 39 10.3% 30.8% 15.4% 43.6% Note: As a result of rounding, percentages may not total to 100. Table F60. Frequency of Use of Curriculum Tools How often do you use the Curriculum Tools to do the following? N Never Once or Twice This Year A Few Times per Semester A Few Times per Month A Few Times per Week Utilize Model Curriculum Units available from ESE 45 75.6% 15.6% 4.4% 2.2% 2.2% Search for instructional materials by standard 44 79.5% 9.1% 6.8% 2.3% 2.3% Plan/schedule curriculum delivery 43 86.0% 4.7% 4.7% 2.3% 2.3% Table F61. Ease of Use of Curriculum Tools How difficult or straightforward was it to do the following in the Curriculum Tools? N Very Difficult Moderately Difficult Neither Nor Difficult Moderately forward Very forward NA/ Never Used Utilize the Model Curriculum available from ESE 45 5.3% 10.5% 5.3% 47.4% 31.6% 57.8%
*Other district administrator (4.8%), guidance counselor (3.4%), instructional leader (8.8%), other school administrator (6.1%) Definition and Communication of Edwin Analytics Goals, Timelines, and Expected Uses Table F70. District and State Communication of the Purpose of Edwin Analytics To what extent do you agree or disagree with the following statements? N Strongly Disagree Disagree Neutral Agree Strongly Agree The State has clearly communicated the purpose of Edwin Analytics. 142 1.4% 15.5% 28.2% 43.7% 11.3% The district has clearly communicated the purpose of Edwin Analytics. 141 0.7% 14.9% 27.0% 49.6% 7.8% Table F71. Major Purposes of Edwin Analytics as Expressed by the State, N = 138 Which of the following are the major purposes of the Edwin Analytics as expressed by the State? (Check all that apply.) Percentage Giving educators a more comprehensive view of students, teachers, schools, districts and the State 76.8% Providing tools and resources that will help educators improve teaching and learning 66.7% Providing tools and resources that will help educators improve education outcomes 59.4% Providing timely and actionable information 50.0% Decreasing achievement gaps 42.8% Providing tools and resources that will help educators improve school readiness 32.6% Other (Please specify.)* 3.6% Note: Percentages may exceed 100 because respondents could choose more than one response. * Align curriculum to the frameworks (0.7%), don’t know (0.7%), improve MCAS performance (0.7%), providing administration with tools to help show patterns (0.7%), PLCs within schools and throughout the district (0.7%) Table F72. Major Purposes of Edwin Analytics as Expressed by the District, N = 137 Which of the following are the major purposes of Edwin Analytics as expressed by your district? (Check all that apply.) Percentage Tracking assessment results 85.4% Identifying achievement gaps 78.1% Identifying gaps in curriculum 60.6% Providing early intervention for students who are struggling 43.1% Understanding if students are ready for college 13.1% Understanding how past students have done in postsecondary education 8.8%
Understanding if students are ready for school (in Grades K–3) 6.6% Understanding district finances 4.4% Other (Please specify.)* 3.6% Note: Percentages may exceed 100 because respondents could choose more than one response. * None (2.9%), teacher evaluation (0.7%) Table F73. District Expected Use of Edwin Analytics Reports, N = 138 Which of the following types of reports from Edwin Analytics does your district expect you to use? (Check all that apply.) Percentage Student growth analysis 85.5% Item analysis 81.9% Achievement analysis 75.4% Early warning indicators (EWIS) 42.8% Curriculum analysis 38.4% Classroom analysis 38.4% Enrollment indicators (attendance, demographics, educational environment, retention, suspension) 30.4% Postsecondary analysis 12.3% Staffing indicators 10.1% Financial indicators 6.5% Other (Please specify.)* 2.9% Note: Percentages may exceed 100 because respondents could choose more than one response. * None (1.4%), reports about MCAS and students that have yet to take or pass it (0.7%), teacher transitions and demographics (0.7%) Table F74. Types of Edwin Communications Please indicate Yes or No for the following statements. N Yes No I received information about Edwin Analytics training opportunities from ESE and/or my district. 138 71.0% 29.0% I received information about Edwin Analytics webinars. 138 55.8% 44.2% I received information about Edwin Analytics EWIS video tutorials. 133 42.1% 57.9% I have visited ESE’s Edwin website. 138 86.2% 13.8%
Who received the training, and how were they selected? Was the training mandatory? Paid? Was it part of the district’s overall professional development plan? Who provided the training? In your opinion, was the training adequate to allow teachers and staff to successfully use Analytics? What is the district/school’s plan for people who haven’t received training? Current Use and Monitoring of Edwin Systems ET&L Please describe the ways in which ET&L is currently used among staff in your school. Probe: Who is using it? Is ET&L used (or planned for use) to develop curriculum (lesson plans, curriculum units, curriculum maps)? Is ET&L used (or planned for use) to develop or administer assessments? Are data from ET&L assessments used to inform instruction? Interventions/ supports for students? Are data from ET&L used to inform strategies for closing the achievement gap for low-income and minority students, and second language learners? Which features of ET&L do teachers use the most? Why? Administrators? Probe: Does this usage follow the district’s plan/goals? If not, does usage reflect teacher initiative, or the tools that are most needed by teachers, or those that are easiest to use, or other factor? Are staff in your district using Edwin Teaching and Learning as planned (considering where you are in your implementation)? Probe: How do you know? (e.g., are data being utilized as expected, are planned instructional responses effective?) Is there a structure in place to review use of the system? How do you identify barriers to planned use? Analytics Please describe the ways in which Analytics is currently used among staff in your school. Probe: Who is using it? Is it used to inform how classes are formed? Groupings within classes? Priorities for teacher professional development? Decisions about interventions?