metadata: authors: 0: author: Angel-Urdinola,Diego
1: author: Burgos Davila,Sebastian Francisco
count: Ecuador
docna: 0: docna: Using Student and Teacher Assessments to
            Design More Pertinent In-Service Teacher Training : The Case
            of Ecuador
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Effective Schools and Teachers,Skills Development and Labor Force Training
teratopic: Education,Social Protections and Labor
abstracts: cdata!: The development of pertinent and
            effective in-service teacher training remains a policy
            challenge for many countries in Latin America and the
            Caribbean. Ecuador stands out as a country in the region
            that has made significant investments in teacher training in
            the past decade. However, most in-service training provision
            has been designed without enough elements to properly
            address teachers' skills gaps. This paper proposes a
            roadmap for improving the design of in-service teacher
            training in Ecuador using available data from student and
            teacher assessments. Although countries in the region have
            made important efforts to carry out periodic evaluations of
            student and teacher performance, the data resulting from
            these evaluations are rarely used to guide teacher
            development programs. The analysis presented in this paper
            suggests that doing so has the potential to raise program
            pertinence while allowing the prioritization of investments
            in teachers and students with the greatest needs.
geo_reg_key: 119228,517191
repnme: repnme: Using Student and Teacher Assessments to
            Design More Pertinent In-Service Teacher Training : The Case
            of Ecuador
display_title: Using Student and Teacher
            Assessments to Design More Pertinent In-Service Teacher
            Training : The Case of Ecuador
listing_relative_url: /research/2019/08/31323810/using-student-teacher-assessments-design-more-pertinent-in-service-teacher-training-case-ecuador
topicv3: Teachers Management m1326119 1523,Education, Skills Development and Labor
            Market m1326135 1484,Teacher Training m1326786 1532,Education m1327048 470,Training Programs m1504140 2817
publishtoextweb_dt: 2019-08-08T00:00:00Z
historic_topic: Education,Social Protections and Labor
no_of_pages: 24
author_keys: 0: author_key: 438716
1: author_key: 1689815
model_extractions: Policy Research Working Paper 8970 Using Student and Teacher Assessments to Design More Pertinent In-Service Teacher Training The Case of Ecuador _Diego F. Angel-Urdinola Sebastian Burgos-Dávila_ Education Global Practice August 2019
Policy Research Working Paper 8970

# **Abstract**

The development of pertinent and effective in-service teacher training remains a policy challenge for many countries in Latin America and the Caribbean. Ecuador stands out as a country in the region that has made significant investments in teacher training in the past decade. However, most in-service training provision has been designed without enough elements to properly address teachers’ skills gaps. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Although countries in the region have made important efforts to carry out periodic evaluations of student and teacher performance, the data resulting from these evaluations are rarely used to guide teacher development programs. The analysis presented in this paper suggests that doing so has the potential to raise program pertinence while allowing the prioritization of investments in teachers and students with the greatest needs.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dangelurdinola@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Using Student and Teacher Assessments to Design More Pertinent InService Teacher Training: The Case of Ecuador

# Diego F. Angel-Urdinola and Sebastian Burgos-Dávila

The World Bank<sup>1</sup> JEL Classification: I20, I24

# **Keywords:**

Education quality, teacher training, professional development, Ecuador.

> 1 This paper was prepared to guide the technical assistance to support the implementation of Ecuador’s Comprehensive <mark>System for the Development of Education Professionals (</mark> SIPROFE) program within the context of the World Bank’s _Supporting Education Reform in Targeted Circuits_ Project. The authors acknowledge the technical support and strategic guidance of Rafael de Hoyos, World Bank; Juan Ponce, _Facultad  Latinoamericana de  Ciencias Sociales_ (FLACSO, Ecuador); and Jorge Yepez, Hortencia Bustos, and Lucia Pinto; Ministry of Education of Ecuador.

1
# **I. Introduction**

In-service teacher training is an essential component of Ecuador’s teachers’ professional development strategy. However, available training interventions offered by the state do not necessarily respond to the teachers’ skills gaps. This occurs because available training programs do not systematically rely on diagnostics of the teachers’ needs. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Current teacher assessments in Ecuador have an active cognitive component capable of assessing the level of knowledge of teachers in the field they teach. They provide valuable information for determining the specific areas of knowledge that teachers have problems with. Coupling such information with available data on student academic achievement, the state could develop not only more pertinent in-service training programs, but also target interventions to teachers that serve the student populations with the greatest needs.

Many countries in Latin America and the Caribbean (LAC) face shortages of good teachers, which undermines the capacity of their systems to improve student achievement in the short, medium, and long term (Rivkin, Hanushek, and Kain 2005; Hanushek and Woessmann 2006; Figlio and Kenny 2007; Chetty, Friedman, and Rockoff 2014). Individuals who become teachers in LAC, most of whom are female, tend to be socioeconomically vulnerable and to underperform in university entrance exams compared to the average university student (Bruns and Luque 2014). Many structural factors explain this phenomenon, such as the lack of incentives to choose teaching as a profession and high turnover of teachers (Ingersoll 2004; Woessmann 2011; Muralidharan and Sundararaman 2011). Moreover, salaries for teachers in the LAC region tend to be substantially lower than in the average profession (Moura-Castro and Ioschpe 2007), which limits the capacity of the sector to attract the best professionals.

Despite remarkable progress in teachers’ educational attainment (at present, most teachers in the LAC region have completed a tertiary degree), graduates from the teaching profession enter classrooms with significant cognitive and pedagogical gaps (Bruns and Luque 2014).<sup>2</sup> These deficiencies weaken education systems in the region, as teachers’ cognitive skills influence the achievement of their students (Hanushek, Piopiunik, and Wiederhold forthcoming).

> 2 According to data from Ecuador’s Ministry of Education (MINEDUC), the public system of primary education in Ecuador employs 163,999 teachers. Of this total, 81 percent have a bachelor’s degree in education, 14 percent have a postgraduate degree (mainly in education), and the remaining 5 percent are not college graduates.

2
In response to the cognitive gaps of teachers, countries in LAC have relied on in-service teacher training as part of their teachers’ professional development strategies (Levy and Schady 2013; Avalos 2007). In-service teacher training is a significant element of non-salary spending in most LAC education systems (Bruns and Luque 2014). In some countries in the region, teachers participate in training between 21 and 35 days per year (OECD 2009).

A primary weakness of in-service teacher training programs is that they seldom respond to the actual skills gaps of teachers (Bruns and Luque 2014; Popova et al. 2018). Ávalos (2007) conducted a review of several in-service teacher training programs in LAC. The study concludes that in-service teacher training in the region should focus more on strengthening teachers’ cognitive deficiencies, given the fragility of their initial training. Although teacher effectiveness involves several factors (such as adequate mastery of the curricula, high motivation, and proper use of pedagogy), the author highlights the importance of prioritizing training efforts in the region as a corrective measure to help teachers reach an adequate mastery of the curricula.

Ecuador has made substantial efforts to improve teacher quality in recent years. In the past decade, the Government of Ecuador has embraced important reforms in teacher development practices including (a) periodic assessment of teachers through standardized tests; (b) setting standards for the admission of teachers into the elite group of nationally recognized teachers, known as the _Magisterio Nacional_ ; (c) better teacher remuneration, partially linked to performance evaluations; (d) the development of a structured teaching career path; and (e) creation of the National Education University ( _Universidad Nacional de Educación_ ) for new teachers (AngelUrdinola and Vera-Jibaja 2018).<sup>3</sup> Moreover, Ecuador has invested significant resources in teacher training.

Nevertheless, in-service training programs in Ecuador have not necessarily responded to teachers’ development needs, and their effectiveness remains unassessed. Teachers who underperform in their knowledge assessments do not always participate in available training programs, take-up rates for some available programs are low, and there is no hard evidence that training programs have contributed to improving teaching effectiveness or student learning.

> 3 Between 2006 and 2015, teacher salaries increased on average from US$396 to US$817 per month (surpassing inflation). Moreover, 17,000 new teaching positions were created from 2011 to 2013, contributing to an 11 percent expansion in the number of teachers employed by the MINEDUC (Harvey 2016). In 2016, 90.6 percent of the teachers were regularized with open-ended appointments, and 29.4 percent (or about 479,000 teachers) benefited from a salary increase due to the _Recategorización_ Program.

3
This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Even though Ecuador has made substantial efforts to periodically evaluate student and teacher performance, the data resulting in these evaluations have not been used systematically to guide teacher development programs. The results of this study indicate that systematically using the data has the potential to help design more relevant programs that respond to observed skills gaps and allow prioritizing investments in teachers and students with the most substantial needs.

The study also intends to raise awareness about the potential gains of capitalizing on available information on student and teacher assessments. Many top education systems invest heavily in student and teacher evaluations. Finland, Japan, the Republic of Korea, China, the United States, and Singapore, among others, all have robust systems that rely on comprehensive student and teacher assessments that serve as inputs to develop more pertinent in-service teacher training programs. However, to date, the use of data resulting from teacher evaluations in Latin America for formative purposes is much more limited (Bruns and Luque 2014). Chile and Mexico are the education systems in Latin America with the most consolidated systems of teacher evaluation. Among other information, both systems collect information about teachers’ cognitive skills. Data resulting from these assessments are often used to make decisions about a teacher's career progression, but not necessarily to develop tailor-made in-service training programs. Thus, the framework presented by the study applies to other countries that, like Ecuador, dispose of detailed information about cognitive skills gaps of teachers, but largely underuse it.

The paper is structured as follows. Section II briefly describes the design and implementation of, and investments made in, in-service teacher training in Ecuador by the Ministry of Education (MINEDUC) under its main national teacher development program, the Comprehensive <mark>System for the Development of Education Professionals</mark> (SIPROFE) program. Section III provides a brief overview of the information collected by the national teacher’s assessment ( _Ser Maestro_ ) and discusses how the information included in this assessment could guide the design of in-service training courses. Section IV presents different policy scenarios that use data from student and teacher evaluations to better design in-service teacher training. A brief conclusion follows.

4
# **II. In-service teacher training in Ecuador**

In 2011, Ecuador enacted a new education law ( _Ley Orgánica de Educación Intercultural_ ) that provided the education sector with a renewed vision of the country’s educational processes and actors. The law recognizes and gives high value to the work of teachers and introduces important elements to promote and develop the teaching career. For instance, Chapter 4 of the law specifies that teachers in Ecuador shall have free access to professional development opportunities, such as in-service training, to continuously update their knowledge and pedagogical practices. Chapter 10 establishes the need for the creation of a National University of Education, whose objective would be to support and develop the teaching profession and to better train new teachers. The law also establishes a more transparent teaching career path, whereby promotions are based on obtaining satisfactory results in teaching evaluations and on active participation in continuing education programs.

In 2008, the MINEDUC began making important efforts to promote in-service teacher training through its Comprehensive <mark>System for the Development of Education Professionals</mark> (SIPROFE), overseen by the MINEDUC’s Undersecretariat of Professional Development. SIPROFE’s main objective is to improve the country’s education quality through the provision of training to teachers and school administrators. To deliver in-service teacher and administrator training, the SIPROFE offers a variety of courses, including remedial and developmental courses, orientation to new teachers, and school management. The duration of courses varies between 8 and 60 hours, although most courses have a duration of either 20 or 40 hours (MINEDUC 2016). Some courses are sequential (that is, are a prerequisite for others) to promote and incentivize the sustained participation of teachers. Training programs under the SIPROFE have traditionally been delivered through agreements with universities and pedagogical institutes ( _Institutos Superiores Pedagógicos_ , ISPED) that satisfy predetermined quality and service delivery requirements.<sup>4</sup> Data from the MINEDUC suggest that during 2010–18, the SIPROFE program delivered in-service teacher training to about 113,000 teachers (representing about 75 percent of the total teacher population). According to the MINEDUC, between 2015 and 2018, the program benefited

> 4 ISPEDs do not operate anymore. Also, universities that can participate as providers of training under the SIPROFE program have to satisfy the maximum level of accreditation (Level A). This requirement limits the territorial supply of training courses, as not many regional universities satisfy this requirement.

5
an average of over 10,000 teachers a year, with an average yearly investment of US$2.7 million.<sup>5</sup> The cost of in-service teacher training is between US$28.5 and US$38.5 per hour of training (MINEDUC 2016).

In-service training under the SIPROFE program is mandatory for teachers who underperform (that is, teachers with a score lower than 700 points in the test) on the teacher assessment _Ser Maestro_ , a test implemented periodically by the National Institute of Evaluation (INEVAL). However, this requirement has not been systematically enforced. Participation in training is optional for all other teachers and school administrators in the public education system. Teachers’ participation in these training programs is among one of many criteria considered for career promotion.

In terms of design features, most training programs take place on school premises and outside the teachers’ working time (that is, holidays, evenings, and weekends). However, in recent years, training providers have introduced some flexibility in the curricula by providing some content (and thus training hours) online. Also, teachers can take training courses during working hours if they obtain permission from school district authorities. To complete a course, participating teachers need to attend all the required course hours and pass a course evaluation. Also, to promote attendance and completion, the MINEDUC requires that teachers who sign up for a training program and fail the course (due to lack of attendance or if they fail the course exam) pay for the training cost out-of-pocket. Training opportunities are normally communicated to teachers by e- mail when new courses open. The content and curricula of the courses are generally designed between MINEDUC administrators and training providers based on the ministry’s identified priorities and the provider’s capacity to deliver courses (that is, courses are largely supply-driven). Neither the MINEDUC nor training providers conduct a systematic diagnostic to identify teacher learning needs.

To date, the effectiveness of the SIPROFE program remains largely unassessed. The MINEDUC has not conducted a systematic evaluation of the program and its effects on teacher and student outcomes. There have been some attempts to collect qualitative information to assess the program’s effectiveness through interviews and focus groups with teachers and school

> 5 Moreover, between 2014 and 2017, funds from the SIPROFE were used to support 3,924 teachers in obtaining a master’s degree in education.

6
principals. Nevertheless, most of the information available relies on small samples and cannot be assumed to be representative for statistical inference.

Results from qualitative interviews conducted by Portaluppi Castro (2012) and FabaraGarzon (2013) show, for instance, that teachers identify several problems with the training courses offered by the SIPROFE program. Focus group participants indicated that the training programs offered under SIPROFE lacked relevance and that courses were theoretical and provided little space for dialogue, analysis, and contextualization. Participants also indicated that there was little follow-up (for example, mentoring, coaching, feedback mechanisms) after training courses were completed. Some participants complained about the quality and pedagogical approaches of the trainers.

On the positive side, participants agreed that the efforts made through the SIPROFE program are a clear signal of the interest of the state in teachers’ professional development. Participants also indicated that some teachers participated in the program to be promoted. Finally, participants suggested that the program could be improved by (a) having a better diagnostic of teachers’ training needs and offering progress to address them; (b) complementing training courses with more follow-up support, such as coaching and mentoring; (c) making it easier to obtain permission to attend training during working hours; and (d) making courses more interactive, flexible, and relevant to local contexts.

# **III. Teacher assessments in Ecuador**

The _Ser Maestro_ assessment, the government’s main instrument to quantify the level of knowledge of Ecuadoran teachers, is implemented by Ecuador’s National Institute of Evaluation and is an important pillar of the country’s teacher evaluation model. The assessment measures the level of knowledge of teachers in the field they teach, including specific topics they must master to promote an adequate learning process in their subjects. The assessment is applied to all teachers in active service (that is, it is census-based). The last assessment was conducted in 2016.

The assessment includes 19 knowledge areas ( _Saberes_ ) that include subjects spanning Ecuador’s three main educational cycles: Early Education (pre-K), Basic General Education (BGE) (grades 1 to 10), and Unified General Baccalaureate (UGB), equivalent to high-school (grades 1 to 3). Each subject includes several “curricular areas,” which assess the level of knowledge of the teacher in specific aspects of the national curriculum within the subject matter.

7
<!-- Start of picture text --> =<br>o t<br>'<br>soc}oO tI<br>=<br>(oy= '{<br>fom <e) IIt<br>oO 1<br>NO, q<br>t<br>2<br>—<br>Ccoes) i{<br>Vie* t<br>a {<br>oO I<br>2° '<br>1<br>t<br>°o |<br>400 600 800 1000<br>Score - Ser Maestro Test<br><!-- End of picture text -->
distribution mean and one standard deviation; (c) high-standard-performance teachers, which are those with scores between the mean and one standard deviation above the mean; and (d) highperformance teachers, which are those with scores above one standard deviation above the mean. Based on these definitions, 6.28 percent of all teachers in Ecuador are classified as being in the low-performance group, 44.45 percent are classified as being in the low-standard-performance group, 38.26 percent are classified as being in the high-standard-performance group, and 11.01 percent are classified as being in the high-performance group.

Assessment results by knowledge topic indicate that a large majority of teachers (50 to 70 percent) underperform in mathematics, Basic General Education, and natural sciences (BGE level), and in mathematics, physics, and biology (UGB level). Overall teacher performance is generally better in subjects that pertain to social sciences and literature (table 1).

**<u>Table 1. Knowledge Areas Included in the</u>** **_<u>Ser Maestro</u>_** **<u>Assessment, by Education Cycle</u>**

|||P<br>Standard|erformance<br>Standard|||% Teachers|
|---|---|---|---|---|---|---|
||Low<br>(%)|<br>Low<br>(%)|<br>High<br>(%)|High<br>(%)|Total<br>Sample|Evaluated<br>in the<br>Subject|
|**General Subjects (all levels)**|||||||
|Management|6.67|39.28|33.78|20.27|2.235|2.38|
|Computers|4.84|38.99|42.44|13.73|3.655|3.89|
|Philosophical thinking|5.74|44.93|29.39|19.93|296|0.32|
|Artistic education|6.02|38.44|38.35|17.20|1.163|1.24|
|Special education|6.8|37.62|45.78|9.82|723|0.77|
|Physical education|6.48|33.72|44.25|15.55|4.102|4.37|
|Early education|4.44|45.61|41.66|8.30|14.281|15.20|
|Citizenship education|6.17|42.08|34.95|16.81|827|0.88|
|**Basic General Education (BGE)**|||||||
|Basic general education (grades 2o–7o)|4.71|47.33|39.02|8.94|41.993|44.69|
|Natural sciences|8.64|47.94|30.38|13.03|5.056|5.38|
|Social studies|6.80|34.62|42.69|15.88|4.659|4.96|
|Language and literature (grades 8o–10o)|5.15|27.38|41.89|25.58|2.991|3.18|
|Mathematics (grades 8o–10o)|13.97|55.52|23.88|6.64|3.208|3.41|
|**Unified General Baccalaureate (UGB)**|||||||
|Physics|21.67|51.71|23.38|3.23|526|0.56|
|History and social sciences|18.10|40.10|31.36|10.43|1.591|1.69|
|Language and literature|9.01|36.50|37.66|16.83|2.775|2.95|
|Biology|6.66|47.10|39.33|6.91|811|0.86|
|Mathematics|16.91|56.49|22.28|4.32|2.271|2.42|
|Chemistry|7.81|33.25|38.79|20.15|794|0.85|
|Total<br>_Source:_Author using_Ser Maestro_2016 data.|6.28|44.45|38.26|11.01|93.957|100.00|

_<mark>Source:</mark>_ <mark>Author using</mark> _<mark>Ser Maestro</mark>_ <mark>2016 data.</mark>
A most interesting feature of the _Ser Maestro_ assessment, as mentioned, is that teachers’ knowledge is tested against a series of “curricular areas” within each subject. Such a level of granularity of information enables the identification of very specific knowledge areas where teachers display cognitive gaps. Data from the _Ser Maestro_ identify the share of correct and incorrect teacher responses in the assessments. The specificity of this information enables personalized teacher development programs to be designed based on actual needs and knowledge gaps (table 2).

10
**Table 2. Curricular Areas (by Subject) with Low Levels of “Correct” Response Rates**

|**SUBJECT AREA**|% of correct<br>answers|Curricular areas where teachers display the lowest performance (by<br>teaching**s**ubject)|
|---|---|---|
|Management|30.0|1. Identification of the main elements of labor contracts<br>2. Knowledge of the main elements required for strategicplanning|
|Computers|25.0|1. Characteristics of “data mining”|
|Philosophical thinking|30.0|1. Identification of aesthetic postulates of emblematic exponents<br>2. Identification of main characteristics of the Andean cosmovision|
|Artistic education|35.0|1. Identification of characteristics of technical elements of the arts<br>2. Identification of characteristics of elements of the arts<br>3. Identification of genres or plastic expressions|
|Special education|35.0|1. Identification of characteristics of the components of a curriculum<br>2. Description of weakened psychological processes|
|Physical education|40.0|1. Recognition of adequate postures<br>2. Determination of parameters to assess physical capabilities|
|Early education|30.0|1. Identification of phonological awareness features<br>2. Identification of rights and responsibilities of children|
|Citizenship education<br>|25.0|1. Identification of practices that promote a culture of peace<br>2. Recognition of the characteristics of citizenship|
|Basic general education|25.0|1. Set theory<br>2. Identification of ordered pairs<br>3. Identification of metabolic processes|
|Natural sciences BGE|25.0|1. Identification of characteristics of autotrophic organisms<br>2. Identification of properties of chemical compounds|
|Social studies BGE|20.0|1. Relationship between humanism and cultural changes<br>2. Recognition of laws or actors that regulate markets|
|Language and literature BGE|40.0|1. Recognition of authors’ shortcomings after reading a text|
|Biology UGB|25.0|1. Identification of the main functions of nucleic acids<br>2. Recognition of hormone functions<br>3. Identification of different types of aquatic biomes|
|Mathematics BGE|25.0|1. Geometry<br>2. Set theory|
|Physics UGB|15.0|1. Notions of thermodynamics<br>2. Calculation of physical magnitudes<br>3. Characterizations of energy|
|History and social sciences UGB|20.0|1. Recognition of authors of contemporary philosophy<br>2. Identification of social characteristics from reading texts|
|Language and literature UGB|30.0|1. Recognition of logical connectors from the reading of a text|
|Mathematics UGB|20.0|1. Set theory<br>2. Geometry<br>3. Notions of cardinality|
|ChemistryUGB<br>_Source:_Author using_Ser Maestro_ <br>_Note:_BGE = Basic General Educa<br>|20.0<br>2016 data.<br>tion; UGB = U|1. Identification of types ofphotovoltaic cells<br>nified General Baccalaureate.|

11
# **IV. Use of assessments to design in-service training programs**

# _Identification of More Pertinent Training Content_

Based on available funding and preestablished yearly training targets, administrators of the SIPROFE program reach out to universities (or other training providers) to have them propose training content. As mentioned, the training curricula for these courses are not based on a systematic assessment of needs, but rather on the university’s expertise, the supply of courses, the availability of trainers, and general delivery capacity (that is, courses are supply-driven).

According to data provided by the National Directorate of Continuous Education (a division under the MINEDUC’s Undersecretariat of Professional Development), since its introduction in 2008, the SIPROFE program has offered, on average, 15 in-service training courses per year. These training programs are categorized into two main groups: (a) complementary programs, which are largely focused on building the capacity of teachers to revise and update the curricula; and (b) remedial programs, which aim to address teachers’ knowledge gaps. Table 3 lists the remedial courses most recurrently offered by the SIPROFE program. Results indicate that courses offered are highly focused on pedagogy and general education, with a limited focus on improving cognitive skills.

**Table 3. Remedial Courses Offered under the SIPROFE Program, 2012–16**

|**Course Name**|**Training Hours**|
|---|---|
|Pedagogy for Natural Sciences (BGE)|40|
|Pedagogy for Social Sciences (BGE)|40|
|Pedagogy for Mathematics (BGE)|40|
|Pedagogy for promoting critical thinking in the classroom|40|
|Pre-K education|50|
|Strategies to evaluate student learning|60|
|Inclusive education|10|
|Introduction of information technology and communications|50|
|Critical reading|60|
|General pedagogy|60|
|Prevention of student risks (sexual abuse, violence, etc.)|30|

_Source_ : MINEDUC 2016.

Program administrators and participating training providers could use the information of the _Ser Maestro_ assessments to make courses more demand-driven and responsive to teachers’ observed cognitive skills gaps. As mentioned, the assessment identifies both the subjects and curricular areas where teachers display cognitive deficits. Administrators, for instance, could develop reports using the _Ser Maestro_ results that show the observed performance of teachers by 12
subject and curricular areas at the district level. For example, figure 2 displays (for a selected school district in Ecuador) the average performance of teachers in the district in curricular areas related to the teaching subject literature and language for UGB. A similar exercise could be replicated in other subject areas, such as mathematics for UGB, physics for UGB, and so forth. The figure plots the average percentage of correct answers in a given curricular area for all teachers in the district who currently teach this subject. The red, yellow, and green bars display the subjects on which teachers, on average, correctly answered less than 30 percent of the subject-related questions, between 30 and 35 percent of the subject-related questions, and between 40 to 60 percent of the subject-related questions, respectively. This information could be used by program managers and universities to develop a more pertinent training curriculum that responds to actual knowledge gaps of teachers in a given school district.

A similar exercise using the results of the _Ser Maestro_ assessments could be conducted at the school level. The MINEDUC could provide school directors with individualized teacher reports that contain similar information for all schoolteachers. Based on such information, teachers and directors could develop individualized training plans. Since at the district level the supply of services (in partnership with local universities and pedagogical institutes) would also consider the training needs of teachers in the district, teachers and directors could choose from the available courses offered at the district level the best ones that suit the teacher’s individual training needs. Of course, adapting the supply of services to the actual needs of teachers would take time. Also, for the process to work, the MINEDUC probably would need to provide technical assistance to school directors to be able to interpret teacher reports and to develop individualized training plans. The MINEDUC could start by testing this model in some school districts (and learn from the experience) before scaling up the model nationally.

13
# **Figure 2. Average Percentage of Teacher’s Correct Answers by Curricular Area in a Selected School District in Ecuador for Teachers of Literature and Language, UGB**

<!-- Start of picture text --> Identify language level or levels.<br>From a text, identify synonyms or antonyms according to the context.<br>From a fragment or list of characteristics, identify non‐literary texts<br>From a text or list of sentences, recognize the correct use of capital letters<br>MODERATE<br>Recognize language functions.<br>From a list or table, recognize words that are formed with affixes. COGNITIVE GAPS<br>Identify the morphology of the verb.<br>From a list of text characteristics, identify the literary subgenres.<br>From a list of words or in tables, recognize the meaning of the affixed<br>From a list of sentences, identify the correct use of punctuation marks<br>From a list of words or sentences, recognize foreign words in the text<br>From a text, recognize implicit ideas.<br>From a text, recognize explicit ideas.<br>From a list of Ecuadoran literary works, relate them to their Latin...<br>From a fragment of a literary work, identify the literary movement / epoch...<br>From a fragment of a literary work, identify the literary gender where it...<br>From a sentence, recognize its variable or invariable parts.<br>From a sentence, identify its syntactic elements.<br>Recognize the definitions of syntactic elements in a sentence.<br>From a list of verses (independent) or a stanza, identify the literary work<br>From a text, table, or list of sentences; recognize the author's vices<br>From a stanza, identify the kind of rhyme or rhythm of the verse.<br>From a text, list, or table; recognizes the characteristics of literary schools<br>From a list, identify the use of homonymous words.<br>From a text, table, or list of sentences; recognize the author's vices LARGE<br>From a list of literary works, identify its author. COGNITIVE GAPS<br>From a list / table of authors, identify the literary genre to which it belongs<br>From a case, list, or table; identify the main communication elements<br>From a text, list, or table; recognize famous authors or works<br>Recognize the subgenre to which the fragment of a literary work belongs.<br>From a list of verses (independent) or a verse, recognize the metrics.<br>From a text or list, recognize the logical connectors.<br>0.0 20.0 40.0 60.0 80.0 100.0<br>% of correct answers<br><!-- End of picture text -->

## _Source:_ Author using _Ser Maestro_ 2016 data.

_Note:_ The red, yellow, and green bars display the subjects on which teachers, on average, correctly answered less than 30 percent of the subject-related questions, between 30 and 35 percent of the subject-related questions, and between 40 to 60 percent of the subject-related questions, respectively.

_Can the results of the Ser Maestro assessments be trusted?_ Using the _Ser Maestro_ assessments to guide the content of in-service teacher training could be problematic if the knowledge areas measured by the exam are not necessarily related to the curricula imparted in the classroom. If this were the case, investing in training programs to solve the cognitive skills gaps of teachers may not necessarily translate into better student learning.

Ecuador uses the _Ser Bachiller_ assessment, a census-based instrument, to assess student learning. The _Ser Bachiller_ assessment is a compulsory exam students take to graduate from the 14
<!-- Start of picture text --> o<br>° ba : .<br>fe)Oo@ ee,2. ARR ote,s “shaatee .<br>° aes0Riaoe 240os otLiane4LOSGiesea yeeSnBeeee°°<br>Ke) tose eee ome .<br>“ee% 2 0s cree 6 © ~<br>° ° 5oe?© 8wee. 8al.cee | o® . ° .<br>.<br>°<br>OoN<br>6 7 8 9 10<br>School's average Language score Ser Bachiller<br>* School's average Language score Ser Maestro --------- Fitted values<br><!-- End of picture text -->

<!-- Start of picture text --> So : . .<br>\ e °<br>Ssco) Se)1 hn1 Fy stea ‘3,<br>2 awanna nnn nnn enna nagae,igMle5% Oo fo.* Geeh, 8-0+ anerofy<br>ee eit Ss v. a<br>oeee iar a<br>i=]8+t °e oPemWS. lattegeOE8iete2 SOs#18oheeeee«|ts<br>ele H<br>2o.|1 3' 1<br>N { H H<br>e ' 1 1<br>5 6 7 8 9 10<br>School's average Mathematics score Ser Bachiller<br>* School's average Mathematics score Ser Maestro --------- Fitted values<br><!-- End of picture text -->
**Table 4. Correlations between Scores on** **_Ser Bachiller_ and** **_Ser Maestro_ Assessments at the School** **<u>Level – Language and Mathematics (UGB)</u>**

||**Lang**<br>|**uage**<br>|**Mathe**<br>|**matics**<br>|
|---|---|---|---|---|
|Lo of score on Ser Maestro|Log score<br>_Ser Bachiller_<br>0.186***|Log score<br>_Ser Bachiller_<br>0.176***|Log score<br>_Ser Bachiller_<br>0.156***|Log score<br>_Ser Bachiller_<br>0.152***|
|g|[0.017]|[0.018]|[0.014]|[0.014]|
|Shl’ ttl tdt||0.000||-0.000**|
|coos oa suens<br>School’s total mathematics teachers||[0.000]||[0.000]<br>0.007***|
|||||[0.002]|
|School’s total language teachers||0.003*<br>[0.002]|||
|Ctt|0.824***|0.877***|0.995***|1.007***|
|onsan|[0.113]|[0.116]|[0.089]|[0.089]|
|Observations|1.271|1.271|1.164|1.164|
|Adjusted R-squared|0.081|0.086|0.097|0.107|

_Source:_ Author using _Ser Maestro_ 2016 and _Ser Bachiller_ 2017/2018 datasets. _Note:_ *** = P ≤ 0.001. Standard errors in brackets.

# _Improving Program Targeting_

In-service training opportunities in Ecuador under the SIPROFE program are made available to teachers on a first-come, first-served basis. Once these training opportunities become available, program administrators contact teachers by e-mail or through local campaigns organized by the MINEDUC in coordination with school district authorities. Teachers who want to participate in the available programs sign up until all slots are filled.

The combined information from the _Ser Bachiller_ and the _Ser Maestro_ assessments constitutes a powerful tool to target interventions in schools (and school districts) with the largest needs. Figure 4 plots school-level overall results of the _Ser Maestro_ (y-axis) and _Ser Bachiller_ (xaxis) assessments. The dotted lines in the figure represent test scores that are one standard deviation above and below the average scores. Each dot in the figure represents a school. The color of the dots represents the poverty rates of the parishes (the smallest administrative territorial disaggregation) where schools are located. The blue dots represent the schools located in areas with the highest poverty rates, and the black dots represent the schools located in areas with the lowest poverty rates.<sup>7</sup> Figure 4 illustrates that schools with the worst-performing teachers also have the worstperforming students (bottom left panel of the chart). While this relationship does not necessarily imply a causality, it contributes to identifying the worst-performing institutions, presumably those

> 7 Poverty is measured by the (census-based) Unsatisfied Basic Needs Index produced in 2010 by Ecuador’s National Office of Census and Statistics.

16
<!-- Start of picture text --> (2) I 1 (|<br>® °° fa ER eee, °°<br>- ° -----------e-- .\ « - + ae . ay ate’ oenr nnn nnn Benen nnn--<br>8 be ry £5 a ee 9 eo te%e” =<br>© ae 3 tot 3 OS! x sa al ae<br>s2 ° Vy ot Oe)vod s! 32 oe °<br>ry ° ehee if ae@ oe: ° i @ .<br>oO oe ee” i— 1 I<br>o \ |<br>gS . | | !<br>=eb1|<br>e<br>v=) ,<br>o O 1] 1 1<br>67 8 9 10<br>Ser Bachiller 2017/2018 test average results<br>e Less than 25% ° Between 25% and 50%<br>e Between 50% and 90% ¢ More than 90%<br><!-- End of picture text -->
**Table 5. Education Districts with Average Scores in the** **_Ser Maestro_ and** **_Ser Bachiller_ Assessments** **<u>that are Below One Standard Deviation from the Mean</u>**

|District<br>ID|Number<br>of<br>Schools|%<br>Schools|Province|Region|_Ser Bachiller_<br>Average Score|_Ser Maestro_<br>Average Score|
|---|---|---|---|---|---|---|
|21D02|2|2.33|Sucumbíos|Amazonía|6.70|484.38|
|14D05|8|9.30|Morona Santiago|Amazonía|6.23|497.28|
|16D02|4|4.65|Pastaza|Amazonía|6.53|497.31|
|13D01|1|1.16|Manabí|Costa|6.75|499.50|
|21D04|5|5.81|Sucumbíos<br>|Amazonía|6.65|500.96|
|19D01|1|1.16|Zamora<br>Chinchipe|Amazonía|6.52|509.60|
|16D01|9|10.47|Pastaza|Amazonía|6.53|512.91|
|15D01|7|8.14|Napo|Amazonía|6.74|520.99|
|06D01|1|1.16|Chimborazo|Sierra|6.57|521.00|
|14D01|1|1.16|Morona Santiago|Amazonía|6.35|527.70|
|09D20|1|1.16|Guayas|Costa|6.94|528.00|
|08D02|4|4.65|Esmeraldas|Costa|6.60|528.67|
|08D04|3|3.49|Esmeraldas|Costa|6.75|530.48|
|14D02|4|4.65|Morona Santiago|Amazonía|6.51|532.68|
|08D03|2|2.33|Esmeraldas|Costa|6.60|536.65|
|14D03|4|4.65|Morona Santiago|Amazonía|6.60|537.36|
|08D01|7|8.14|Esmeraldas|Costa|6.76|541.35|
|22D01|2|2.33|Orellana|Amazonía|6.80|543.06|
|09D02|1|1.16|Guayas|Costa|6.83|545.67|
|22D02|7|8.14|Orellana|Amazonía|6.76|547.01|
|21D03|1|1.16|Sucumbíos|Amazonía|6.56|552.50|
|13D07|1|1.16|Manabí|Costa|6.60|552.67|
|08D06|2|2.33|Esmeraldas|Costa|6.76|555.42|
|05D04|1|1.16|Cotopaxi|Sierra|6.80|555.60|
|08D05|1|1.16|Esmeraldas|Costa|6.93|556.25|
|09D11|1|1.16|Guayas|Costa|6.61|561.60|
|12D06|1|1.16|Los Ríos|Costa|6.76|562.17|
|21D01|1|1.16|Sucumbíos|Amazonía|6.45|564.67|
|12D01|1|1.16|Los Ríos|Costa|6.95|568.00|
|19D02|1|1.16|Zamora<br>Chinchipe|Amazonía|6.84|572.80|
|09D06|1|1.16|Guayas|Costa|6.79|575.94|
|**Total**|86|99.97|||||

_Source:_ Author using _Ser Maestro_ 2016 and _Ser Bachiller_ 2017/2018 datasets.

18
# **V. Conclusion**

Teacher performance constitutes a critical determinant of student learning. As such, continuous contact with poor-performing teachers over several years could result in significant learning gaps. In Ecuador, evidence indicates that children in poor localities face significant risks of being taught by teachers who do not master the curricula of the class they teach. Results from teacher assessments indicate that a large majority of teachers underperform in mathematics, Basic General Education, and the natural sciences (BGE level), and in mathematics, physics, and biology (UGB level). Given the fact that many teachers in Ecuador enter the education system with important cognitive gaps, in-service teacher training could be an important tool to strengthen teacher performance.

Ecuador in recent years has made important investments in improving teacher development and quality. Since 2008, the MINEDUC has been promoting in-service teacher training through its Comprehensive <mark>System for the Development of Education Professionals</mark> (SIPROFE) program. Since 2010, the SIPROFE program has delivered in-service teacher training to about 75 percent of Ecuador’s total teacher population. Nevertheless, training programs delivered under the SIPROFE program have been supply-driven, focused on improving pedagogy, and not necessarily designed to respond to the observed teacher’s cognitive skills gaps. Moreover, their effectiveness has been largely unassessed, and program targeting has not prioritized school districts where teacher development programs are needed the most.

Ecuador currently uses teacher and student assessments that can identify in detail the cognitive deficiencies of students and teachers. These data, however, have not been used to guide the design of in-service teacher training programs. Ecuador’s teacher assessment ( _Ser Maestro_ ) evaluates teachers on the subjects they teach in the classroom and identifies curricular areas within a teaching subject where teachers display cognitive gaps. The specificity of this information enables personalized development programs to be designed based on teachers’ actual needs. Program administrators and participating training providers could use this information to develop a more pertinent training curriculum that addresses the knowledge gaps of teachers across different school districts.

Furthermore, combining available information on student and teacher assessments would allow policy makers in Ecuador to identify the worst-performing schools. Given that resources to provide in-service training opportunities are limited, the government could use this information to 19
target interventions to schools with the greatest needs. SIPROFE administrators could use these assessment results plus other information; such as proximity to training providers, local poverty rates, and school size; in their yearly programming, which targets selected territories nationwide.

The results of the _Ser Maestro_ assessments could also serve to develop diagnostics for schools and teachers. The MINEDUC could provide school directors, teachers, and parents with individualized reports that contain information about observed cognitive gaps. Based on such information, teachers and school principals could develop individualized training plans for schools and teachers. Using available information from student and teacher assessments, thus, could become a key input for policy makers in countries that use such information, such as Ecuador, to develop more pertinent and effective teacher development programs. Since more countries are periodically conducting student and teacher assessments, the recommendations presented here for Ecuador could well be taken into consideration regionally and globally.

Of course, the process of adapting the supply of training services to the actual needs of teachers (and to local needs) takes time. However, Ecuador, as well as many other countries in the region, could consider using available evaluations to design and implement in-service training programs of higher relevance and impact while allowing the prioritization of investments in teachers and students with the greatest needs.

20