ENQA
AGENCY
REVIEW

2017

ENQA AGENCY REVIEW:
ACCREDITATION ORGANISATION OF THE
NETHERLANDS AND FLANDERS (NVAO)


ANDY GIBBS, OBE DE VRIES, LAURA BECCARI, ALEXANDRA RAIJMAKERS
13 SEPTEMBER 2017

.......................................................................................................................... 3
.................................................................................................................................... 5
BACKGROUND OF THE REVIEW AND OUTLINE OF THE REVIEW PROCESS ............................................ 5
BACKGROUND OF THE REVIEW .............................................................................................................. 5
MAIN FINDINGS OF THE 2012 REVIEW ................................................................................................... 5
REVIEW PROCESS ................................................................................................................................ 7
HIGHER EDUCATION AND QUALITY ASSURANCE SYSTEM OF THE AGENCY ........................................ 9
HIGHER EDUCATION SYSTEM ................................................................................................................. 9
QUALITY ASSURANCE ......................................................................................................................... 11
NVAO............................................................................................................................................... 14
NVAO’S ORGANISATION/STRUCTURE .................................................................................................. 14
NVAO’S FUNCTIONS, ACTIVITIES, PROCEDURES ..................................................................................... 15
NVAO’S FUNDING ............................................................................................................................ 17
............................................... 18
ESG PART 3: QUALITY ASSURANCE AGENCIES ................................................................................ 18
ESG 3.1 ACTIVITIES, POLICY, AND PROCESSES FOR QUALITY ASSURANCE ................................................... 18
ESG 3.2 OFFICIAL STATUS .................................................................................................................. 24
ESG 3.3 INDEPENDENCE .................................................................................................................... 25
ESG 3.4 THEMATIC ANALYSIS ............................................................................................................. 26
ESG 3.6 INTERNAL QUALITY ASSURANCE AND PROFESSIONAL CONDUCT ................................................... 28
ESG 3.7 CYCLICAL EXTERNAL REVIEW OF AGENCIES ................................................................................ 29
ESG PART 2: EXTERNAL QUALITY ASSURANCE ................................................................................ 31
ESG 2.1 CONSIDERATION OF INTERNAL QUALITY ASSURANCE .................................................................. 31
ESG 2.2 DESIGNING METHODOLOGIES FIT FOR PURPOSE ........................................................................ 33
ESG 2.3 IMPLEMENTING PROCESSES.................................................................................................... 34
ESG 2.4 PEER-REVIEW EXPERTS .......................................................................................................... 36
ESG 2.5 CRITERIA FOR OUTCOMES ...................................................................................................... 38
ESG 2.6 REPORTING ......................................................................................................................... 39
ESG 2.7 COMPLAINTS AND APPEALS.................................................................................................... 42
................................................................................................................................... 44
SUMMARY OF COMMENDATIONS ................................................................................................... 44
OVERVIEW OF JUDGEMENTS AND RECOMMENDATIONS ............................................................... 44
............................................................................................................................................. 46
1/60

ANNEX 1: 2012 AND 2017 EXTERNAL REVIEWS: A COMPARATIVE OVERVIEW OF THE FINDINGS
.......................................................................................................................................................... 46
ANNEX 2: PROGRAMME OF THE SITE VISIT ..................................................................................... 49
ANNEX 3: TERMS OF REFERENCE OF THE REVIEW .......................................................................... 53
ANNEX 4: GLOSSARY ...................................................................................................................... 59
ANNEX 5. DOCUMENTS TO SUPPORT THE REVIEW ........................................................................ 60
DOCUMENTS PROVIDED BY NVAO ...................................................................................................... 60
OTHER SOURCES USED BY THE REVIEW PANEL ........................................................................................ 60

2/60

This report analyses the compliance of the Accreditation Organisation for The Netherlands and
Flanders (Nederlands Vlaamse Accreditatie Organisatie, NVAO) with the Standards and Guidelines
for Quality Assurance in the European Higher Education Area (ESG). The purpose of the review is to
verify that NVAO acts in substantial compliance with the ESG as adopted at the Yerevan ministerial
conference of the Bologna Process in 2015. ENQA requires all member agencies to undergo an
external cyclical review, at least once every five years. Substantial compliance with the ESG is a
condition for membership.
NVAO was established by Treaty on 1 February 2005 by the Dutch and Flemish governments as a binational organisation. It has operated as the formal accreditation body in these countries since then.
NVAO passed two previous ENQA reviews, in 2007 and in 2012, and is registered in EQAR since
November 2008.
In light of the documentary and oral evidence, considered by the review panel, the review panel has
the following judgements and recommendations on separate standards.
F = Fully compliant; S = Substantially compliant, P = Partially compliant, N = Non-compliant
STANDARD

JUDGEME
NT

ESG 2.1 CONSIDERATION OF

F

RECOMMENDATION

INTERNAL QUALITY ASSURANCE

ESG

2.2

DESIGNING

METHODOLOGIES

FIT

F

FOR

PURPOSE

S
ESG

2.3

IMPLEMENTING

PROCESSES

ESG

2.4

PEER-REVIEW

NVAO remains sensitive regarding issues of implementation as
experienced by stakeholders and adopts a clearer terminology to
differentiate between substantively different approaches to follow
up of decisions.

F

EXPERTS

ESG

2.5

CRITERIA

FOR

S

OUTCOMES

S
ESG 2.6 REPORTING
ESG 2.7 COMPLAINTS AND

P

APPEALS

S
3.1 ACTIVITIES, POLICY AND
PROCESSES

FOR

QUALITY

ASSURANCE

3.2 OFFICIAL STATUS
3.3 INDEPENDENCE
3.4 THEMATIC ANALYSIS

F
F
S

NVAO reflects on the grading system as presently applied and
considers making it as straightforward as necessary for the purpose
of yes/no/conditional accreditation.
NVAO analyses the actual and potential readership of its reports and
their needs, and develops new means to reach a readership among
students and employers.
NVAO develops a complaint procedure and opens a section
‘Complaints and appeals’ at its website with appropriate formats for
complaints and appeals.
NVAO prepares coherent development plans on the future shortterm and long-term development of accreditation processes in
Flanders and The Netherlands, on the basis of comprehensive
evaluations of the NL- and FL-pilots and taking into account
expectations about the quality assurance in higher education by all
relevant stakeholders.

NVAO reviews the approach to thematic analysis and in consultation
with stakeholders, identifies a common purpose and a means to
regularize processes and priorities in this area.

3/60

3.5 RESOURCES
3.6
INTERNAL

QUALITY

ASSURANCE

F
S

AND

NVAO remains alert on loose loops in the plan-do-check-act-cycle of
the quality assurance system.

PROFESSIONAL CONDUCT

3.7

CYCLICAL

EXTERNAL

F

REVIEW OF AGENCIES

On the basis of these judgements the review panel is satisfied that, in the performance of its
functions, NVAO is in substantial compliance with the ESG.

4/60

This report analyses the compliance of the Accreditation Organisation for The Netherlands and
Flanders (Nederlands Vlaamse Accreditatie Organisatie, NVAO) with the Standards and Guidelines
for Quality Assurance in the European Higher Education Area (ESG). It is based on an external review
conducted from November 2016 until August 2017 (from self-analysis by the agency until the
finalisation of the review report).
BACKGROUND OF THE REVIEW AND OUTLINE OF THE REVIEW PROCESS
BACKGROUND OF THE REVIEW
ENQA’s regulations require all member agencies to undergo an external cyclical review, at least once
every five years, in order to verify that they act in substantial compliance with the ESG as adopted at
the Yerevan ministerial conference of the Bologna Process in 2015.
As this is NVAO’s third review, the panel is expected to provide clear evidence of results in all areas
and to acknowledge progress from the previous review. The panel has adopted a developmental
approach, as the Guidelines for ENQA Agency Reviews aim at constant enhancement of the agencies.
MAIN FINDINGS OF THE 2012 REVIEW
Summary of the 2012 review
The panel is convinced that NVAO acts in compliance with the ENQA membership regulations and is
in substantial compliance with the Standards and Guidelines for Quality Assurance in the European
Higher Education Area. Details of the measure of compliance are given below. The Panel therefore
recommends to the Board of ENQA that NVAO’s Full Membership of ENQA be confirmed for a
further period of five years.
During the 2012 site visit, interviewees mentioned several times that NVAO is in a ‘transition state’
as models and procedures for quality assurance were being changed in relation to Flanders or had
recently been changed in The Netherlands. It was the view of the review panel that a situation of
change and development is to be expected for most quality assurance agencies. The panel
recognises that NVAO and its partners have work ahead in order to establish the institutional
assessment approach in Flanders as well as to consolidate the new practice in The Netherlands. The
panel encourages all the parties to learn from the experience gathered thus far.
Compliance
The Panel concludes that NVAO fully complies with the following ESG 2005 Standards
 3.2 ‘Official status’, (now ESG 2015, Standard 3.2, ‘Official status’);
 3.3 ‘Activities’(now partly ESG 2015, Standard 3.1 ‘Activities, policy, and processes for quality
assurance’);
 3.4 ‘Resources’( now ESG 2015 2015, Standard, 3.4 ‘Resources’);
 3.5 ‘Mission statement’ (now partly ESG 2015, Standard 3.1 ‘Activities, policy, and processes
for quality assurance’);
 3.6 ‘Independence’( now ESG 2015, Standard 3.3 ‘Independence’);
 3.7 ‘External quality assurance criteria and processes used by members’ (now partly ESG
2015 Standard 2.3 ‘Implementing processes’, Standard 2.4 ‘Peer-review experts’, and
Standard 2.7 ‘ Complaints and appeals’);
 3.8 ‘Accountability procedures (now partly ESG 2015 Standard 3.6 ‘Internal quality assurance
and professional conduct’, and Standard 3.7 ‘Cyclical external review of agencies’;
 2.1 ‘Use of internal quality assurance procedures’ (now ESG 2015 Standard 2.1
‘Consideration of internal quality assurance’);
 2.6 ‘Follow-up procedures’ (now partly ESG 2015 Standard 2.3 ‘Implementing processes’);
5/60



2.7 ‘Periodic reviews’ (now ESG 2015, Standard 1.10, Cyclical external quality assurance’).

On the following ESG 2005 Standards the review panel’s conclusion was ‘substantial compliance’:
 3.1 ‘Use of external quality assurance procedures for higher education’ (now partly ESG
2015 Standard 3.1 ‘Activities, policy, and processes for quality assurance’);
 2.2 ‘Development of external quality assurance processes’ (now partly ESG 2015 Standard
2.2 ‘Designing methodologies fit for purpose’);
 2.4 ‘Processes fit for purpose’ (now partly ESG 2015 Standard 2.2 ‘Designing methodologies
fit for purpose’);
 2.5 ‘Reporting’ (now ESG 2015 Standard 2.6 ‘Reporting’);
 2.8 ‘System-wide analyses’(now ESG 2015 Standard 3.4 ‘Thematic analysis’).
On the following ESG 2005 Standard the review panel’s conclusion was ‘partial compliance’:
 2.3 ‘Criteria for decisions’ (now ESG 2015 Standard 2.5 Criteria for outcomes’).
Recommendations
The main areas for further improvements relate to public information in various senses and to the
formulation of explicit and public criteria about how NVAO reaches its accreditation decisions in
relation to the programme assessments in all cases. On specific standards the review panel
expressed the following recommendations:
ESG 2005 Standard 2.1 ‘Use of internal quality assurance procedures’
[NVAO] makes the link between the NVAO standards and ESG Part 1 more explicit in the assessment
frameworks.
ESG 2005 Standard 2.2 ‘Development of external quality assurance processes’
[NVAO] further strengthens the focus on quality improvement and enhancement of the HEI’s.
ESG 2005 Standard 2.3 ‘Criteria for decisions’
 [NVAO] refines the descriptions of the aims and objectives, ensures that they are prominent
in the frameworks and show how the various elements of the frameworks contribute to the
aims and objectives;
 [NVAO] formulates explicit and public criteria about how it reaches its accreditation
decisions in relation to the programme assessments in all cases;
 [NVAO] establishes a clear procedure on how to handle cases where the conclusions in the
assessment report are not accepted by NVAO;
 [NVAO] strengthens the predictability of the timeframe and efficiency of its decision-making
process.
ESG 2005 Standard 2.5 ‘Reporting’
[NVAO] clarifies
a. the purpose of every kind of report;
b. the readership; and
c. the needs of the various kinds of readers in order to enhance the readability;
ESG 2005 Standard 2.8 ‘System-wide analysis’
[NVAO] gives a high priority to
a. the identification of the interested parties for system-wide analyses and of their needs;
b. the definition of a realistic schedule of system-wide analyses; and
c. the production of system-wide analyses corresponding to the needs which were identified.

6/60

Further to the panel’s recommendations the Board of ENQA reconfirmed NVAO’s full membership of
ENQA in a letter, dated 29 January 2013. In this letter the Board suggested NVAO to pay attention to
the following issue:
ESG 2005 Standard 3.7 ‘External QA criteria and processes used by the members’
To improve the consistency of accreditation judgments reached. Clustering of assessments may be
beneficial to improve consistency in assessments and decision making, as well as to answer the
stakeholders’ need for more system-wide analysis.
REVIEW PROCESS
The 2017 external review of NVAO was conducted in line with the process described in the
Guidelines for ENQA Agency Reviews and in accordance with the timeline set out in the Terms of
Reference. The panel for the external review of NVAO was appointed by ENQA and composed of the
following members:
 Andy Gibbs (Chair), Senior Lecturer, formerly Glasgow Caledonian University, the UK (EUA
nominee)
 Obe de Vries (Secretary), Independent consultant and QA professional, The Netherlands
(ENQA nominee)
 Laura Beccari, Scientific collaborator, AAQ , Switzerland (ENQA nominee)
 Alexandra Raijmakers, Student at University of Oslo, Norway (Dutch national) (ESU nominee)
Maria Kelo, director of ENQA, acted as process coordinator for the review.
From September 2016 onwards panel members had regular e-mail exchanges about the organisation
of the review. In December the panel received the self-assessment report (SAR) and from then on
work on the review started. Following a proposal by the chair, specific standards were allocated to
the panel members as specific focus points for their attention. In February 2017 the panel had a
telephone conversation in which both practical details of the review, the proposed time schedule for
the site visit, preliminary impressions and wishes for additional documents were discussed. At that
time also special points for further investigation were brought together in a “mapping grid for panel
members”. Meanwhile the panel secretary had regular contacts with the ENQA process coordinator
and the NVAO review coordinator. Further e-mails on SAR and additional documents were
exchanged in February and March. On 28th March, a day before the site visit the chair and secretary
met to establish a comprehensive document with proposed “topics per session”.
On 29th March, the panel met in The Hague for two hours, shortly before the start of the site visit, to
discuss the topics. It was decided that, while an individual panel member would have a first
responsibility for bringing up topics related to ‘his or her standard’ in specific sessions, other panel
members could join in in the discussions whenever suitable. Thus, there would be no specific leadperson per session, apart from the chair.
Self-assessment report
The self-assessment report is a 76 page document with some 300 pages Annexes. According to the
SAR it was produced in a collaborative effort of NVAO staff and Board. In May 2016 the first draft
was accepted by the General Board of the NVAO for discussion with external stakeholders and
NVAO’s Advisory Council. The final version was distributed to the panel in December 2016, after
scrutiny on formal aspects by the ENQA-secretariat.
The SAR contains chapters on higher education and quality assurance in the context of the NVAO;
history, profile and activities of NVAO; a SWOT analysis; HE-QA-activities of NVAO; NVAO’s processes
and methodologies; NVAO’s internal quality assurance; its international activities; and compliance
with ESG Part 3 and Part 2. Also information and opinions of stakeholders are summarized, as well as
7/60

recommendations and main findings from previous review(s) and NVAO’s resulting follow-up. The
SAR ends with current challenges and areas for future development.
The Annexes consist of CV’s of NVAO Board Members; key figures on higher education in The
Netherlands and Flanders; assessment frameworks for the higher education accreditation system of
The Netherlands 2014; assessment framework for the higher education accreditation system of The
Netherlands 2016 and overview of standards and assessment rules frameworks 2014-2016;
Frameworks for higher education Flanders and Quality Code 2015; details of NVAO activities and
procedures including communication activities; differentiation in judgements in The Netherlands and
Flanders; internal quality assurance, feedback analysis and samples of surveys; profiles panel chair
and process coordinator; thematic analyses: short description and references.
In the course of February and March additional documents were supplied, partly on request of the
panel (see Annex 5 of this report).
Site visit
The site visit took place from 29 to 31 March according to the time schedule as given in Annex 2.
Interviewees were the SAR team, NVAO Executive and General Board, NVAO staff, representatives of
the Dutch and Flemish ministries of higher education, Dutch and Flemish institutions of higher
education, assessment agencies, panel members, students, teachers and employers.
The panel had allowed an ‘open hour’ for any stakeholder, who wasn’t interviewed and still had a
wish to speak with the panel about the functioning of NVAO. This open hour was announced at the
NVAO-website well before the site visit, but nobody showed interest to make use of this
opportunity.
During the interview sessions with the SAR team, and with the students, teachers and employers
and during final de-briefing a representative of The Netherlands’ Inspectorate of Education was
present as an observer. The purpose of this was for her to get acquainted with the present
accreditation system, in preparation of an evaluation survey among Netherlands HE-institutions, to
be held in September 2017. The observation followed the ENQA-protocol for observers, and it was
ensured that none of the interviewees had objections.
Before the start of the interviews and after the second day of the site visit the panel had clarification
meetings with NVAO process coordinator on specific topics. In the course of the site visit additional
documents were supplied on request of the panel (see Annex 5 of this report).
During the site visit panel members had various internal, consensus-forming discussions: on each
day before the beginning of the interviews; in between and after interview sessions; and in the
scheduled concluding meeting of the panel at the last day of the site visit. As regards the scope of
the site visit the panel decided that activities referring to advisory work and decision making based
on specific evaluations (see NVAO’s FUNCTIONS, ACTIVITIES, PROCEDURES, p15) were as yet not
within the scope of the ESG. Therefore they will not be covered in this report.
After the site visit panel members sent individual notes on specific standards to the secretary, who
then compiled a first draft of the review report. After scrutiny by the chair and other panel members
a final version of this draft was sent to the ENQA process coordinator by the end of May. After
further refinements the report was sent to NVAO for a check on factual details in the first half of
June. In the first half of July the panel received an extensive reaction from NVAO. In this reaction a
number of minor textual changes were proposed, as well as corrections of factual mistakes, partly
due to inaccuracies in the SAR. Furthermore, the panel was asked for some clarification on specific
8/60

considerations. Apart from that the NVAO Board raised some issues in which the Board had a
different opinion than the panel. In the second half of July and the first half of August the panel
considered all these elements and where relevant, amended the text. The judgements on specific
standards were not affected. The final external review report was presented to the ENQA Board on
13th September 2017.
HIGHER EDUCATION AND QUALITY ASSURANCE SYSTEM OF THE AGENCY
HIGHER EDUCATION SYSTEM
In the SAR an extensive description of The Netherlands and Flemish higher education systems is
provided, together with key figures on higher education in The Netherlands and Flanders. In Table 1
a summary of these data is shown.
Table 1: Key figures on higher education in The Netherlands (2015/16) and Flanders (2014/15)
The Netherlands Flanders
16.900.726
6.471.996*

Inhabitants
Institutions

Programmes

Students

Staff

Universities
14
Universities of Applied Sciences (NL) and 37
University Colleges (FL)
Recognised private institutions (NL) and 70
registered institutions (FL)
Other statutory registered institutions

6
21

Academic bachelor’s programmes
Higher prof. ed. Bachelor’s programmes
Priv. Inst. Bachelor’s programmes
Advanced higher prof. ed. Bachelor’s
programmes
Associate degree programmes
Priv. Inst. Associate degree programmes
Academic Master’s programmes
Priv. Inst. Master’s programmes
Higher prof. ed. Master’s programmes
Advanced academic master’s programmes

233
290

402
1048
164

5
6

68
128
44
797
75
240

589

227

Fulltime students academic programmes
258.100
Fulltime students higher prof. ed. 442.600
programmes
Full time students Arts education (FL)

108.098
109.448

Staff (persons) academic programmes

15.085

Staff (persons) higher prof.ed. programmes

51.246
(=44.243 fte)
45.838
(=33.840,8 fte)

6.709

10.637

* not including Brussels (1.180.531 inhabitants)
Dutch academic education
Academic bachelor’s programme make up the first cycle and comprise 180 ECTS (3 years) and can be
oriented towards a single discipline or have a broad, multidisciplinary profile. Most bachelor
9/60

graduates continue their studies in a master’s programme. They are often required to go through a
pre-master’s programme before applying for admission into a master’s programme in another
discipline. The bachelor’s programme is positioned at EQF-level 6 and leads to the degree Bachelor
of Arts/Science, or Bachelor of Laws for law programmes. Master’s programmes regularly comprise
60 ECTS (for most specialisations), 120 ECTS (for engineering, science and teacher training, or
research master’s programmes) or 180 ECTS (medicine, veterinary science, pharmaceutical science
and dentistry). The master’s degree is positioned at EQF-level 7. The degree associated with
academic master’s programmes is Master of Arts/Science, depending on the specialisation. Students
who graduate from a law programme receive the title Master of Laws (LLM).
Dutch higher professional education
Universities of applied sciences and private institutions offer professional bachelor’s, associate
degree, and master’s programmes. Some institutions also offer academic master’s programmes,
mostly in Business Administration. The bachelor’s programme comprises 240 ECTS (4 years).
Graduates can apply for admission to an academic master’s programme, which also often requires
finishing a pre-master’s programme. Short cycle programmes of 120 ECTS are connected to the
curriculum of professional bachelor’s programmes. They lead to an Associate Degree (AD).
Graduates can continue their studies in a professional bachelor’s programme within the same
domain. Professional master’s programmes comprise 60 ECTS (1 year) or 120 ECTS (2 years) for
some specialisations, such as Music or Architecture. These programmes are mostly intended for
practising professionals. The master’s programmes are positioned at EQF-level 7. Degrees associated
with professional bachelors and master’s programmes are Bachelor and Master, with the addition of
Arts/Science or the professional domain (e.g. Social Work).
Flemish academic education
The first cycle consists of bachelor’s programmes positioned at EQF-level 6, which comprises 180
ECTS and nominally takes three years, leading to the degree Bachelor, with the addition of the field
of study. The second cycle consists of master's programmes of at least 60 ECTS for Humanities, Fine
Arts, Industrial Engineering, and Social Sciences, and 120 ECTS for Sciences, Engineering, Music,
Medicine, and Biomedical Sciences. Specifically, research-oriented master’s programmes also
comprise 120 ECTS. Master's and Advanced master’s programmes are positioned at EQF-level 7 and
have an academic orientation but can in addition also have a professional orientation. They are
concluded with a final project, referred to as master’s dissertation, of 15-30 ECTS. The project should
comprise at least one fifth of the number of credits of the entire programme. Advanced master's
programmes of 60 ECTS are also positioned at EQF level 7, and aim at further deepening the
knowledge and competences in a certain field of study. A student must already hold a master's
degree to enrol. Study programmes in Arts have a professional or an academic orientation.
Bachelor’s programmes comprise 180 ECTS (3 years), master’s programme 60 ECTS (Fine Arts,
Theatre) or 120 ECTS (Music).
Flemish professionally-oriented higher education
Professionally oriented bachelor’s programmes at EQF-level 6 make up the first cycle. They comprise
at least 180 ECTS and take three years. Short cycle programmes of 90/120 ECTS, positioned at EQFlevel 5, are positioned in between secondary education and professionally oriented bachelor’s
programmes. They lead to the Associate Degree. Advanced bachelor's programmes of at least 60
ECTS aim at deepening the knowledge and/or competences acquired in a professional bachelor's
programme. A student must therefore already hold a bachelor's degree to enrol.
Mutual recognition and international branches of Dutch and Flemish programmes
Since 2009 Flanders and The Netherlands recognized each other’s diplomas provided they had been
accredited by NVAO. On 18 May 2015 the Flemish Government, the Government of the Federation
10/60

Wallonia-Brussels, the Government of The Netherlands and the Government of Luxembourg agreed
to mutual recognition of bachelor’s and master’s degrees in higher education.
In Flanders, the quality assurance of higher education provided abroad has always been part of the
regular accreditation remit. As a result, the education provided at several international branch
campuses has been accredited as part of a regular accreditation procedure. Until 2016, foreign
students in Dutch programmes were required to spend at least 25 percent of the duration of their
study at the Dutch location to receive a Dutch degree. The Ministry of Education is preparing a
procedure for allowing Dutch institutions to provide an entire programme at a foreign campus
(transnational education), on the condition that the institution already offers the programme at its
Dutch location, that no public money is used to finance the foreign campus, and that the aim of the
transnational programme is not entirely commercial. NVAO will assess the quality of the
transnational education based on a specific framework and advise the Minister of Education. After
approval, the transnational programme will be included in all regular accreditation procedures. The
Ministry plans to have this procedure in place in 2017.
QUALITY ASSURANCE
The Netherlands and Flanders have separate systems of quality assurance, which are laid down in
legislation. In The Netherlands: ‘Wet op het Hoger Onderwijs en Wetenschappelijk onderzoek
(WHW), art 5a Accreditatie in het hoger onderwijs. In Flanders: Codex Hoger Onderwijs, Deel 2, Titel
II, Hoofdstuk 2, Accreditatieorganisatie. NVAO develops and maintains different frameworks for the
two countries, which need approval by the Ministers of Education in The Netherlands and by
Parliament in Flanders.
External quality assurance in The Netherlands and Flanders applies both to new and existing
programmes (accreditation) and institutions (NL: institutional audit, FL: institutional review).
Recently the Dutch and Flemish systems of quality assurance entered a second phase in their
development. In the first phase (2004-2010) the system focused primarily on programme
assessment. In the second phase (2010-2016) The Netherlands introduced institutional audits as
complementary to limited programme assessments. Because quality assurance, strategic policy and
human resource management are assessed at the institutional level, the limited assessments focuses
on the content of the curriculum, assessment and the intended and achieved learning outcomes,
leaving more room for genuine peer review and dialogue between colleagues.
In Flanders, frameworks for limited programme assessments were introduced in 2012. In 2014, a
pilot round of institutional reviews (without binding legal consequences) was added to the system.
Programme accreditation in The Netherlands and Flanders (in general)
Programme accreditation is mandatory for all new and existing programmes in higher education in
Flanders and The Netherlands. It entitles the programmes to be registered in the national registers
of higher education and, when offered by publicly funded institutions, to receive public funding and
to grant legally recognised degrees and diplomas. The duration of an accreditation is six years in The
Netherlands, eight years in Flanders, which is set in the law to change to six years in 2021. The
frameworks used in both countries for the assessment of new and existing programmes contain
standards relating to the following four quality aspects:
1. the objectives and intended learning outcomes of the programme;
2. the curriculum, the learning environment and teaching staff, and the quality assurance
system;
3. the assessment procedures and examinations;

11/60

4. the achieved learning outcomes of existing programmes, or of new programmes when they
are already running.
Programmes are assessed in clusters within the same domain. Assessment panels are composed of
independent experts (peers) and are subject to approval or advice by NVAO. All assessment reports
and all decisions of NVAO on the basis of these reports are made public by NVAO. In the assessment
frameworks of The Netherlands and Flanders implementation varies in some details per procedure
and per country.
Institutional audits in The Netherlands (current situation)
The institutional audit in The Netherlands introduced in 2010 is optional. It assesses the educational
vision of an institution and the polices for putting this vision into practice, including human
resources and internal quality assurance. A positive result entitles institutions to use the limited
frameworks for (initial) accreditation. The audit does not entail institutional accreditation with selfaccrediting powers at programme level.
Programme review in The Netherlands (current situation)
In 2013, the Dutch system of quality assurance was evaluated by the Inspectorate of Education and
the Ministry of Education. The Dutch and Flemish Courts of Audit also performed an evaluation,
which was mainly directed towards the functioning of NVAO. NVAO evaluated both systems and its
own role too. The 2013 evaluations of the Dutch system were followed by intensive consultation
with the field by the Minister of Education, in preparation for a strategic agenda which called for an
optimisation of the accreditation system, resulting in a report with the title: ‘Tailor-Made
Accreditation’ (Accreditatie op Maat). A steering group of stakeholders developed a proposal for the
new policy. The 2016 framework is the result of this initiative, together with a proposal for a change
of legislation.
The new 2016 framework has come into force in 2017 and optimises existing practices and
procedures in order to reduce the administrative burden, to increase the flexibility of the system,
and give actors more ownership over assessment procedures. The framework is radically simplified:
the previous seven individual frameworks for limited and full (initial) accreditation and for
institutional audit are reduced to two, and also the number of standards is reduced. Overlap is
further reduced between the standards for institutional audit and (initial) limited programme
accreditation, and the standards refer more explicitly to the latest version of ESG.
According to the document ‘Update on developments in Dutch higher education and in the structure
of NVAO’ (17 March 2017) one element in the changes to the system introduced with the new
framework in 2016 concerns a pilot with a lighter form of programme assessment that is planned to
start in September 2017. For programmes in this pilot only intended and achieved learning
outcomes (Standards 1 and 4) will have to be assessed by an external panel, while the quality of the
learning environment and the system of assessment (Standards 2 and 3) will be assessed using peer
review under the umbrella of the internal quality assurance of the institution. During the pilot
programmes can opt to join the regular programme assessment clusters for the assessments of the
intended and achieved learning outcomes, or have these assessed in a separate process. The
institutions in the pilot will still undergo the regular institutional audits.
The pilot will run for six years from 2017 to 2023 and is to be evaluated in the fifth year. NVAO will
be involved in the monitoring of the pilot. Institutions eligible for the pilot need to have a good track
record in recent assessments. A very small number of institutions have expressed interest in taking
part in the pilot. The Ministry of Education submitted a proposal for the implementation of the pilot
to NVAO for consultation.
12/60

Programme accreditation and institutional review in Flanders (current situation)
In 2014, within Flemish institutions it was felt that the double burden of a pilot round of institutional
reviews and programme assessments to be introduced that year was too demanding. This coincided
with a growing awareness among institutions about the importance of quality improvement and
reflection on the most effective programme assessment method, be it internal or external,
particularly in programmes that already passed one or more accreditations.
As a consequence, a pilot was proposed in which the institutional review was to be extended to
cover all aspects of the ESG, including guaranteeing the quality of study programmmes. Institutions
willing to take part in this “extensive” institutional review would be subject to transitional
arrangements for programme accreditation. During the pilot, the period of accreditation was to be
extended for programmes of participating universities and university colleges, except for: the initial
and first accreditation of new programmes, the programmes that had a reduced period of
accreditation (“in an improvement period”) and international joint programmes that are considered
to be accredited in accordance with the stipulations of a European funding programme, e.g. Erasmus
Mundus.
In 2015, the Flemish Minister of Education asked NVAO to design a new framework for this pilot
phase. In 2016 and 2017 the pilot round of extensive institutional reviews is being held. The
outcomes of the reviews will be published, but have no legal consequence. An evaluation of the pilot
at the end of 2017 will decide on the outline of the new accreditation system in 2020. Assessments
of all programmes are still required for all other institutions not undergoing the comprehensive
institutional review and for all programmes falling under the exception criteria mentioned above.
The extensive institutional review assesses an institution’s educational policy as well as its conduct in
assuring the quality at the level of programmes, in line with a Quality Code developed by NVAO. This
code intends to make the comprehensive institutional review in line with the ESG and requires the
inclusion of external experts, peers and stakeholders and the publication of information on the
quality of individual programmes.
In 2015, new frameworks for (initial) programme accreditation were developed and put in place,
which comprise three standards:
1. objectives and qualifications;
2. learning content and learning environment;
3. assessment procedures, examination and achieved learning outcomes.
For programmes not offered by institutions undergoing the institutional review, a fourth standard on
the quality assurance system is added to the assessment.
Institutions and assessment agencies
A specific feature of the external quality assurance system in The Netherlands and Flanders is that
institutions have a leading role in proposing panels. They are assisted by assessment agencies, which
are typically agencies commissioned by institutions, not by NVAO. This is considered to enhance the
ownership by institutions and create room for more diversity related to the specific nature of
programmes.
In The Netherlands the main agencies are QANU, Hobéon, NQA, AeQui, and Certiked. QANU is EQARregistered. Some of the other agencies are in the process of acquiring registration. In small, specific
areas, international agencies such as Musique, AACSB and EQUIS are active. QANU focuses on
academic programmes, while the others provide assistance to higher professional programmes
mainly.

13/60

Until 2015, VLUHR-KZ was formally appointed as an assessment agency in Flanders. Presently,
Flemish institutions are free to choose any EQAR-registered agency for the quality assessment of
existing programmes, or an agency recognised by the accreditation agency. NVAO has a procedure
for such recognition. NL-institutions are allowed to organise assessments by themselves, but they
only do this in a very limited number of cases. VLUHR-KZ is registered in EQAR for its assessments of
formally registered programmes in Flanders. In the accreditation frameworks, NVAO has developed
guidelines for the quality and independence of the assessment panels, the support by trained
secretaries and instructions for the assessment reports.
In both NL and FL, the agencies select panel members, plan and organise site visits and write the
reports of assessments. They provide expertise and assistance to institutions, for instance on how to
write self-evaluation reports, and practical guidelines for the programme staff who prepare the
assessments. The agencies must follow NVAO protocols and guidelines for assessments as laid down
in the accreditation framework. In this manner, NVAO ensures that all reviews and the decisions
based on these are in line with ESG. The institutions commission the assessment agencies to
coordinate assessments. Therefore, the agencies have a formal relationship with the institutions, but
not with NVAO. NVAO has no direct influence on the activities of the agencies. In regular
consultations with the agencies (at least three times per year), NVAO discusses the quality of
assessments, the reports, and other issues that may come up in the handling of applications. In
recent consultations, the quality of assessment reports and the administrative burden have been a
recurrent topic in the consultations.
NVAO
NVAO was established by Treaty on 1 February 2005 by the Dutch and Flemish governments as a binational organisation. It has operated as the formal accreditation body in these countries since then.
NVAO passed two previous ENQA reviews, in 2007 and in 2012, and is registered in EQAR since
November 2008.
NVAO’S ORGANISATION/STRUCTURE
NVAO is governed by a General Board consisting of 13 members, four of whom form the Executive
Board. In the Executive Board two members are from The Netherlands and two from Flanders. All
members of the General Board are appointed by the Flemish and Netherlands Ministers of
Education. An Advisory Council meets irregularly, and is gradually replaced by intensive resonance
groups. By treaty, the ‘Committee of Ministers’ (Comité van Ministers), comprising the Ministers of
Education of The Netherlands and Flanders, supervises NVAO’s activities.
The organizational structure within NVAO consists of two departments, a Flemish one and a
Netherlands one, each headed by a general manager. The Flemish team comprises of 8.8 fte, the
Dutch team of 20.8 fte. A team of support services, serving the Board and the two departments
comprises of 11 fte and consists of IT services, financial services, HRM services, Legal support, and
secretariats for the Board and Management Team, including a position of HRM Manager of 0.6 fte.
This organisational structure is very recent (March 2017), as before the organisation had no split in a
Flemish and Dutch department. According to the SAR the division in two departments has strategic
as well as practical reasons. NVAO’s strategy recognizes that the system of quality assurance in
Flanders and The Netherlands develop each in their own directions, which calls for some adaptations
in the organisation of NVAO to optimally support both systems. Both the nature and the scope of the
workload will differ for the two countries, which would make it more efficient to set up two separate
teams. There is an efficiency benefit in specializing on the specific new procedures in the two
systems. The training of NVAO staff specifically for these new approaches in both countries is more

14/60

efficient than maintaining a large capacity with a more general level of specialisation. The new
organisational structure is supported by the budget arrangements.
NVAO’S FUNCTIONS, ACTIVITIES, PROCEDURES
Mission
In the Strategy document 2017-2020, dated March 2017, NVAO’s mission is formulated as follows:
The Accreditation Organisation of The Netherlands and Flanders (NVAO) is a quality
assurance agency that, in an expert and independent manner, provides insight into
the quality of higher education in The Netherlands and Flanders, and that promotes
the quality culture within the higher education institutions in The Netherlands and
Flanders. It decides on the accreditation of existing and new programmes, and
assesses the quality assurance in place in higher education institutions. Thus, NVAO
monitors quality. The NVAO decisions result in recognition of diplomas and degrees,
and where applicable in funding of programmes. NVAO bases its assessment on
recommendations by recognised experts.
ESG and core values of NVAO
Quality assurance in both countries is based on the principles expressed in the European Standards
and Guidelines. Core values to NVAO are: independence, clarity and transparency, respect and trust.
The following basic methodologies are applied:
 Peer review and the use of generic frameworks and standards, and an emphasis on
ownership by those who create quality;
 Accountability and improvement are integrated in quality assurance, and the development
of a quality culture is considered equally important as accountability;
 A single framework applies to both academic and professionally oriented programmes.
Programme accreditation is complemented by an institutional audit or review.
Activities
The SAR mentions a number of activities that NVAO carries out in accordance with its legal tasks and
activities linked to that.
Activities based on legal tasks
Activities under this heading refer to (initial) accreditation of programmes and institutions in higher
education, e.g..
 Deciding on applications for accreditation for programmes in higher education including
Associate Degrees/HBO5, in Flanders and The Netherlands, including the so-called Caribbean
Netherlands: the Caribbean islands Bonaire, St. Eustatius and Saba.
 Coordinating assessments and deciding on initial accreditation of programmes in Flanders
and The Netherlands, on application by institutions.
 Accreditation of joint programmes preferably based on mutual recognition agreements and
stimulating the use of these agreements; accreditation of joint degrees on the basis of the
joint degree protocols for (initial) accreditation.
 Coordinating the assessments of and deciding on institutional audits in The Netherlands and
institutional reviews in Flanders, on application by institutions.
 Coordinating assessments of higher education programmes on the Caribbean islands of
Curaçao, Aruba and St. Maarten (so-called Caribbean part of the Kingdom of The
Netherlands), on request of the Ministry of Education of The Netherlands.
 Coordinating assessments and deciding on the granting of distinctive (quality) features for
programmes and institutions in The Netherlands and Flanders.

15/60

NVAO’s assessments in the Caribbean region are carried out on the same basis as the regular
assessments in The Netherlands, whereby NVAO upholds the same goals and objectives. They are to
be distinguished between activities in Bonaire, Saba and St. Eustatius, which are regular
municipalities of The Netherlands on the one hand, and Curaçao, Aruba and St. Maarten, which are
independent countries within the Kingdom of The Netherlands, on the other hand. In the latter case
NVAO carries out assessments on request. NVAO has clear protocols for all activities related to
accreditation (distinctive features, approval of panels, drafting frameworks, assessments of features
of programmes). As most of the procedures coincide with accreditation assessments, NVAO upholds
the same principles in these. Protocols are discussed with stakeholders.
Activities linked to legal tasks
The SAR also mentions activities linked to legal tasks, in particular:
 Drafting and maintaining frameworks for accreditation for programmes and institutions in
higher education, both in The Netherlands and in Flanders (ESG 2.2 and 2.3);
 Approving of/advising on panels for programme assessments proposed by institutions in
Flanders and The Netherlands (ESG 2.4);
 Producing thematic analyses or evaluations, on the basis of assessments (ESG 3.4).
General activities connected to assessments
The above mentioned activities are supported by more general activities such as publishing
assessment reports and decisions on accreditation of new and existing programmes, institutional
audits or reviews and all other NVAO procedures; informing the public on the quality of higher
education in Flanders and The Netherlands and implementing the training of panel chairs and
secretaries in assessments organised by NVAO or by assessment agencies and activities related to
international cooperation in the field of external quality assurance.
Advisory work and decision making
A separate field of activities is concerned with advisory work and decision making based on specific
evaluations, such as:
 Advising the Minister of Education on applications for combining two or more existing
programmes into broader programmes (NL);
 Advising the Minster of Education on applications for extending (and reducing in Flanders)
the formal duration of programmes, including research master’s programmes of 120 EC
(advice in Flanders, initial accreditation in The Netherlands);
 Deciding on the changes of names of programmes and degrees (NL);
 Advising the Minister of Education on allowing additional admission criteria for programmes
related to specific educational concepts (NL).
As mentioned before, the panel considered whether these types of activities fall within the scope of
ESG. Having heard the ENQA Secretariat, considering the activities weren’t directly related to ESGlinked activities up to now, and taking into account that only since 2016 the NL framework integrates
these activities in procedures related to (initial) accredition and institutional audit, the panel
concluded that these types of activities will not be referred to in this report.
International cooperation
According to the SAR NVAO’s active role in European networks on quality assurance brings
international expertise to its activities. NVAO makes use of expertise from international experts in
assessment panels, as do the assessment agencies. International experience and knowledge is a
standard requirement for all panels appointed or approved by NVAO.
NVAO has also actively promoted the mutual recognition of accreditation of joint programmes, and
signed the MULTRA agreement initiated by ECA, which facilitates a single accreditation procedure in
16/60

one country instead of multiple procedures in all countries of the joint programme consortium
where accreditation is required. Under the MULTRA agreement, NVAO takes an independent
decision based on a foreign accreditation. The same philosophy of a single accreditation procedure
and mutual recognition is applied to the European Approach for Quality Assurance of Joint
Programmes which was accepted by the EHEA Ministers in Yerevan in May 2015. NVAO signed a
bilateral mutual recognition agreement with the German Accreditation Council in July 2015. The
number of joint programmes that will be accredited on the basis of the European Approach for QA
of Joint Programmes is expected to rise.
NVAO’S FUNDING
The SAR describes that the NVAO office has a number of staff comparable to 43 fte. An overview of
financial data handed over during the site visit showed that the annual budget (2016) is 7,7 m€.
NVAO is financed by the Dutch and Flemish governments with a ratio of 60/40 to a total amount of
5.5 m€. Fees in The Netherlands and Flanders amount to 2.2 m€. The fees for accreditation are in
The Netherlands and Flanders 800 € and 500 € respectively, for initial accreditation in The
Netherlands 17,1 k€ and in Flanders 5 k€.

17/60

ESG PART 3: QUALITY ASSURANCE AGENCIES
ESG 3.1 ACTIVITIES, POLICY, AND PROCESSES FOR QUALITY ASSURANCE
Standard:
Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a
regular basis. They should have clear and explicit goals and objectives that are part of their publicly
available mission statement. These should translate into the daily work of the agency. Agencies
should ensure the involvement of stakeholders in their governance and work.
ENQA review 2012
The present ESG 2015 Standard 3.1 contains elements of the former ESG 2005 Standards 3.1 (‘Use of
external quality assurance procedures for higher education’), 3.3 (‘Activities’) and 3.5. (‘Mission
Statement’).
Compliance with ESG 2005 Standard 3.1, 3.3 and 3.5
On ESG 2005 Standard 3.1 the 2012 review panel concluded ‘substantial compliance’, on ESG 2005
Standards 3.3 (‘Activities’) and 3.5 (‘Mission statement’) ‘full compliance’.
The panel’s conclusion of ‘substantial compliance’ with Standard 3.1 was partially based on the
assessment of ESG 2005 Part 2 Standards 2.1 - 2.8, which were described and analysed under
Standard 3.1. The judgements on these standards were as follows:
On the ESG 2005 Part 2 Standards 2.1, 2.6 and 2.7 the review panel concluded ‘full compliance’:
 2.1 ‘Use of internal quality assurance procedures’ (now ESG 2015 Standard 2.1
‘Consideration of internal quality assurance’);
 2.6 ‘Follow-up procedures’ (now part of ESG 2015 Standard 2.3 ‘Implementing processes’);
 2.7 ‘Periodic reviews’ (now ESG 2015, Standard 1.10, Cyclical external quality assurance’).
On the ESG 2005 Standards 2.2, 2.4, 2,5 and 2,8 the review panel concluded ‘substantial
compliance’:
 2.2 ‘Development of external quality assurance processes’ (now part of ESG 2015 Standard
2.2 ‘Designing methodologies fit for purpose’);
 2.4 ‘Processes fit for purpose’ (now part of ESG 2015, Standard 2.2);
 2.5 ‘Reporting’ (now ESG 2015 Standard 2.6 ‘Reporting’);
 2.8 ‘System-wide analysis’ (now ESG 2015 Standard 3.4, ‘Thematic analysis’);
On the ESG 2005 Standard 2.3 the review panel concluded ‘partial compliance’:
 2.3 ‘Criteria for decisions’ (now ESG 2015 Standard 2.5, ‘Criteria for outcomes’).
Recommendations
Under ESG 2005 Standard 3.1 the 2012 review panel formulated a number of recommendations at
some standards in the ESG2005, which are now Standards 2.1, 2.2, 2.5, 2.6 and 3.4 in ESG 2015. In
this report these recommendations will be referred to under these five standards.

18/60

Evidence
Mission, vision and strategy
As mentioned in the Introduction (p15) NVAO’s mission emphasizes decision-making on the
accreditation of programmes and assessment of institutions in an independent manner and on the
basis of peer review by recognised experts, while at the same time promoting a quality culture
within higher education institutions in The Netherlands and Flanders. In the strategy document
2017-2020 a quality culture within institutions is seen as essential for pursuing continuous quality
assurance and quality improvement. The external assessment of the quality as exercised by NVAO is
meant to dovetail closely with the quality assurance in place at the educational institutions
themselves. NVAO’s strategy consists of:
 trust as the basic principle for quality assurance, but taking decisive action wherever
necessary;
 respecting the autonomy of institutions as regards their own quality assurance;
 a tailored approach in the assessment of educational quality, actively reducing the
administrative burden.
The strategy document 2017-2020 is a follow-up of an earlier strategy document 2013-2016, which
like its successor also emphasizes justified trust and reduction of administrative burden, but is more
topical on issues like quality of panels, reports, guidelines, cluster assessments and internal quality
assurance. Also, in a special paragraph, the relevance of good communication is mentioned and
illustrated with a number of good intentions.
Activities
Apart from analysing NVAO’s compliance with ESG Standards 2.1-2.7, which will be covered in p3244, the panel wanted to find out whether the methodologies used in NVAO’s activities, mentioned in
the INTRODUCTION (p15/16) are in line with NVAO’s core values, mission and strategy as regards
these standards.
2.1 Consideration of internal quality assurance
In NVAO’s mission the autonomy of institutions ranks highly. Representatives of institutions and
panel members informed the panel that in reviews the institution’s autonomy as regards internal
quality assurance was valued positively, while at the same time stressing the need to adhere to ESG
Standards Part 1. The frameworks for the assessment activities as mentioned above and reports
confirmed this view. The panel concludes that in practice the element of autonomy in NVAO’s
mission is well served.
Standard 2.2 Designing methodologies fit for purpose and Standard 2.3 Implementing processes
Trust and a tailored approach in the assessment of educational quality is another aspect of NVAO’s
mission, which shows itself in the design of methodologies, both at programme and institutional
level (see also INTRODUCTION, p11-14). As will be discussed later in more detail stakeholder’s
involvement in governance and actual design of methodologies is high. At the same time one of the
aims of improvement of methodologies (reduction of administrative burden) has not been realised
so far. An interesting case in methodology design and implementation is the pilot institutional
review in Flanders. This pilot came about as a sudden and substantive deviation from the Strategy
document 2013-2016, to be discussed later under ‘Current developments in Flanders’ (p22), but all
Flemish stakeholders informed the panel that they were actively engaged in formulating and
implementing this pilot so as to make it fit for the specific situation in Flanders.
Standard 2.4 Peer-review experts
The use of recognised experts is an important element of NVAO’s mission. On the basis of CV’s of
experts, and comments by panel members and stakeholders the panel concludes that this core value

19/60

is well adhered to, even though the experts could more frequently be drawn from a wider
international field than The Netherlands and Flanders only.
Standard 2.5 Criteria for outcomes
Transparency is one of the core values of NVAO and this value should express itself in clear criteria
for outcomes and in the handling of these criteria in actual reviews. During the 2012 ENQA Review
the panel had serious concerns about this aspect. As will be discussed under ESG Standard 2.5
definite improvements have been made since. However, the grading of outcomes from insufficient
to excellent remains a difficult issue, on which further reflection by NVAO and other relevant parties
is needed.
Standard 2.6 Reporting
In Strategy document 2013-2016 good communication is emphasized as an important strategy
component. Since the last ENQA-review progress is obvious, but on (initial) programme accreditation
(NL/FL, C/A/StM) progress could still be made, in particular as regards reaching the student
population.
Standard 2.7 Complaints and appeals
Respect is one of NVAO’s core values. However, while appeal procedures are in place and mentioned
in all frameworks, a solid and formal comprehensive procedure on complaint handling is not
present. The panel considers this as an omission, that would need to be addressed and solved.
Summarizing the above the panel concludes that most of NVAO’s activities are in line with its own
core values, mission and strategy. As will be shown later in this report the same holds true for
NVAO’s substantial compliance with ESG standards 2.1 - 2.7 (see p32-44). Table 2. serves as a brief
overview of the panel’s findings.
Table 2: Overview of evidence as regards ESG 2015 Standards 2.1-2.7
Activities
2.1
a. Programme accreditation NL
+
b. Programme accreditation FL
+
c. Initial programme accreditation FL, NL
+
d. Initial joint programmes accreditation
+
e. Institutional audits NL
+
f.
Institutional reviews FL
nya
g. Initial programme assessment C/A/StM
+
h. Distinctive features FL, NL
na
Standard 2.1 Consideration of internal quality assurance
Standard 2.2 Designing methodologies fit for purpose
Standard 2.3 Implementing processes
Standard 2.4 Peer-review experts
Standard 2.5 Criteria for Outcomes
Standard 2.6 Reporting
Standard 2.7 Complaints and appeals

2.2
+
+
+
+
+
+
+
+

Standards
2.3
2.4
2.5
2.6
2.7
+
+
+/+/+/+
+
+
+/+/+
+
+
+/+/+
+
+
+/+/+
+
+
+
+/+
+
nya
nya
+/+
+
+
+/+/+
+
+/+
+/+ = positive
+/- = mixed
- = negative
n(y)a= not (yet) applicable
FL = Flanders
NL = The Netherlands
C/A/StM = Curaçao/Aruba/St. Maarten

As regards the pilot institutional review (FL) only a selected number of standards is indicated in Table
2, because the results of the pilot are not yet out. Institutional audits on Curaçao, Aruba and
St.Maarten are not mentioned, as there have been none so far.
Work planning
Given the multitude of activities the panel wanted to be reassured that staff could cope with the
workload involved in these activities. This proved to be the case, as outlined under Standard 3.5, and
20/60

even better than before since the new organisational structure was established. The review panel
studied Excel-planning documents up till 2019, which showed that activities are well-planned.
Following the recommendation by the ENQA Board reviews of clusters of programmes are now also
undertaken in The Netherlands for professional higher education programmes; in NL-universities
they were already common practice. The panel saw extensive planning schemes for clustered
assessments for the years to come. However, one assessment agency told the panel that the
operational aspect of cluster reviews was difficult for institutions and assessment agencies alike, for
instance as regards the availability of panel experts.
Stakeholders
Stakeholders have input through a variety of means:
 Participation in the General Board. In the GB stakeholders from various backgrounds are
represented such as: higher education institutions, students, Flemish Inspectorate of
Education, Employers;
 Feedback from the Advisory Council. NVAO has an Advisory Council which consists of
representatives from all relevant stakeholders. The Advisory Council is asked for feedback
and consultation on NVAO policies. The NVAO Board usually sets the agenda, but the
Advisory Council can convene a meeting with the Board on its own initiative as well;
 Regular meetings with representatives of institutions and students. They come together
about every two months in so-called Resonance groups. In Flanders these groups are
supplemented with the Cabinet and the Ministry of Education and with assessment agency
VLUHR-KZ as an observer;
 Regular meetings with the assessment agencies. Assessment agencies are important
stakeholders for NVAO. NVAO regularly consults with the joint agencies and also has
meetings with each of the agencies separately;
 Regular meetings with representatives from the Dutch Ministry of Education to exchange
information on developments in the political context and policy changes that have
consequences for accreditation. There is also close cooperation with The Netherlands’
Inspectorate of higher education;
 The NVAO Board regularly (every six weeks) consults with members of the Flemish Cabinet
for Education.
During the site visit representatives of various stakeholders confirmed that NVAO takes
stakeholders‘ opinions very seriously. Stakeholders play an important role in carrying out NVAO’s
strategy and activities. The new Dutch and Flemish frameworks for quality assurance were drafted
after intensive consultation with stakeholders and reflect their input. All stakeholders appreciate the
value of NVAO in particular as regards its independence in validating conclusions drawn by
assessment panels. In spite of this some stakeholders express uncertainty about the role of NVAO as
controller vs. critical friend. NVAO staff members inform agencies and institutions in response to
questions on compliance, but also perform a formal role in handling procedures of applications.
Stakeholders explained to the panel that this sometimes causes confusion. However, the NVAO
Board assured the panel that this doesn’t reflect a structural confusion of roles within NVAO.
Engaging the interest and involvement of employers and employer organizations in the work of
NVAO is an area which the General Board is actively trying to address.
International expertise
According to the SAR NVAO’s active role in European networks on quality assurance brings
international expertise to its activities. NVAO makes use of expertise from international experts in
assessment panels, as do the assessment agencies. International experience and knowledge is a
standard requirement for all panels appointed or approved by NVAO. However, the review panel is
21/60

not impressed by the engagement of international experts in assessment panels, even though in a
sample of 2016 assessment reports the review panel noticed use of international experts in
academic reviews, and also various exchanges of Flemish and Dutch experts. Apparently, the Dutch
language used in many reviews is an obstacle. Also there is no international membership in the
General Board, but during the site visit the panel was informed that this has been noticed by the
Board, and that the Board was looking for a (Dutch-speaking) international board member.
Current developments in The Netherlands
As mentioned before, both in The Netherlands and Flanders changes are taking place. In NL new
frameworks have been put into operation in 2017, which are meant to enhance trust, respect
institutional autonomy and reduce the burden of work. The further reduction of limited programme
accreditation to two standards, which will take shape in a pilot that is intended to start in 2017, was
applauded by staff from NL professional higher education institutions. The representative of NL
universities commented that in the eyes of NL-universities the 30-year era of external programme
accreditation in NL could come to an end (‘external eyes [on programme quality] are not needed any
more’) and that institutional review would be a sufficient means to assure quality. This was
contradicted by representatives from NL-professional higher education institutions, who stressed
the relevance of programme accreditation. This view was supported by NL-student-representatives,
from both universities and higher professional education institutions. They favour programme
assessment, as it gives them more confidence in the quality of individual programmes.
Current developments in Flanders
In Flanders presently a pilot of institutional reviews of public universities and university colleges is
taking place. The pilot quickly developed into a Flanders-wide experiment in external quality
assurance and was brought to the attention of EQAR in a ‘Substantive change report’ on 20 February
2017. In the strategy document 2013-2016 the pilot wasn’t mentioned and even though a move
towards a QA system based solely on institutional reviews was envisaged already in 2013, during the
site visit one stakeholder described it as a ‘jump’. Among Flemish institutions, staff and employers,
the review panel noticed great enthusiasm about the pilot, as it had triggered new dynamics of
internal quality assurance within institutions. However, Flemish students representatives observed
mixed feelings of the student body about it, some uncertainties remaining whether this would be an
improvement in the external quality assurance of programmes. The panel also noted dissatisfaction
in NL-institutions about the lack of communication on the pilot: ‘We don’t know what is going on’.
Cyclical external quality assurance
The extensive institutional review is a baseline assessment to spearhead the introduction of a QA
system in which individual external programme assessments can be replaced by institutional
assessments. Programme assessments would be carried out internally under the responsibility of
institutions themselves, which will be considered for effectiveness in the external institutional
review. A special feature of the pilot is, that in 2017 and 2018 a total of 53 programmes are expected
to have an extension of existing accreditation beyond the regular cycle-time of external quality
assurance. For the years 2017 and 2018 this number amounts to some 4% of a total of 1201
accredited programmes in Flanders. Depending on the outcome and speed of future decisionmaking this number could go up significantly from 2019 onwards. During the pilot phase, institutions
are expected to fully take ownership of assuring the quality of the programmes as outlined in
NVAO’s Quality Code. According to the SAR the evaluation at the end of 2017 will decide on the
outline of the new accreditation system in 2020. The validity of the accreditation of existing
programmes is extended until at least the start of the new system, except for programmes in an
improvement period and those that have only undergone initial accreditation. Consequently it is
programmes which have been reviewed three times before, that are put ‘on hold’ (as mentioned in
the SAR and in the Substantive change report to EQAR). Given the review-track-record of these
22/60

programmes neither the Board nor FL-institutions considered this as a risk. Considering that the
Quality Code is designed in line with the ESG, and that new or problematic programmes are still
assessed under the current framework, the panel believes that the pilot is not likely to engender
risks in continuing assuring the quality of programmes, until a new accreditation system will be
established.
Future perspectives: coming together or drifting apart?
A frequently debated subject during the site visit was whether the diversity in pilots (NL: programme
accreditation on the basis of two standards; FL: full institutional review), together with the split of
the NVAO into an NL- and a FL-department signalled drifting apart of FL and NL. According to the
SAR, and in spite of the diverse pilots, there is a clear convergence of the systems in both countries
in the direction of external quality assurance that is more directed at institutions. For the coming
future, the institutional audit and review would gain in importance as defining assessments of
whether institutions are capable of ensuring the quality of programmes. However, during the site
visit in particular NL-stakeholders saw in the FL-pilot and the split into two departments a clear
divergence, which was regretted by all. A representative from NL- professional higher education
institutions reckoned that in the first years divergence would be unavoidable but that convergence
could occur again later.
The General Board stressed that the higher education community in Flanders is too small to have its
own independent quality assurance system and that there is one NVAO strategy for the two
countries. The task ahead would be to develop common perspectives. The Chair concluded that the
main challenge for NVAO is to serve two countries well. However, the SAR points out that it will
prove difficult to satisfy the demands of all parties: institutions want more autonomy and flexibility;
students and society demand guarantees for the quality of education; the political context,
especially in The Netherlands, carries specific and fluctuating opinions on the instrumentation of
quality assurance.
Analysis
Activities related to standards 2.1 - 2.7
The review panel is impressed by the range of activities conducted by NVAO and the way in which
work is planned and carried out by NVAO-staff. As will be discussed in this report under ESG Part 2
there is room for improvement on specific aspects of NVAO’s activities, but on the whole there is
sufficient evidence of substantial compliance on the respective standards. The review panel is also
satisfied with the way NVAO engages stakeholders in their work, although more engagement on the
part of employers would be needed. In spite of the inherent international character of a binational
organization and NVAO’s involvement in international quality assurance networks the panel has
hesitations about the international involvement in panel reviews and General Board, beyond NL and
FL.
Future developments
The panel is satisfied with the small, but consistent steps taken in The Netherlands as regards the
pilot in programme accreditation that is being planned now, in order to reduce the burden of work,
while maintaining a high level of quality assurance and to achieve even further internalization of
quality assurance within Dutch HEI’s. With regards to developments in Flanders, the panel believes
that a detailed and accurate analysis needs to be undertaken at the end of the pilot-phase, in order
to make sure that the model applied is robust enough as to assuring public trust with regards to the
new system and the maintenance of the quality of programmes. The student body in particular, as
an essential stakeholder, would need to be closely involved in the analysis and conclusions about the
effectiveness of the tested model, particularly in assuring the quality of programmes. In the absence
of accessible reports, so far the panel would only limit itself to strongly advice NVAO to develop
23/60

plans for the years to come, considering different scenarios, and to make sure all relevant
stakeholders are directly involved in the evaluation of the pilot, being able to consensively advice
decision makers on the measures to be taken in the future.
Panel recommendation
NVAO prepares coherent development plans on the future short-term and long-term development
of accreditation processes in Flanders and The Netherlands, on the basis of comprehensive
evaluations of the NL- and FL-pilots and taking into account expectations about the quality assurance
in higher education by all relevant stakeholders.
Panel conclusion: substantially compliant
ESG 3.2 OFFICIAL STATUS
Standard:
Agencies should have an established legal basis and should be formally recognised as quality
assurance agencies by competent public authorities.
ENQA Review 2012
Standard 3.2 was in ESG 2005 also Standard 3.2 ‘Official status’. The review panel concluded ‘fully
compliant’. No recommendations were given.
Evidence
NVAO was established by Treaty on 1 February 2005 by the Dutch and Flemish governments as a binational organisation. It has operated as the formal accreditation body in these countries since then.
As described under ‘QUALITY ASSURANCE’ (p11), Dutch and Flemish legislation on higher education,
explicitly describe the role and tasks of the NVAO. According to these legal documents, NVAO has
the authority to decide on accreditation of programmes and institutions of higher education in The
Netherlands and Flanders. All activities lead either to an independent decision by NVAO or an advice
to the Minister of Education in The Netherlands or Flanders, as laid down in legislation. NVAO’s
decisions comply with the requirements for formal decisions under administrative law and mention
the underlying legislation.
The official status of NVAO was strengthened in 2009 when The Netherlands and Flanders
Governments decided to recognize each other’s diploma’s provided NVAO would have accredited
the respective programmes. In 2015 this agreement was extended to the Benelux countries at large.
In their meeting of September 2016 the Dutch and Flemish Ministers reconfirmed the commitment
of both countries. In response to changes in the accreditation systems in The Netherlands and
Flanders, both governments have asked NVAO to reflect on changes in its governance structure.
However, the structural changes implemented in 2017 do not infringe on the formal status of NVAO.
The authority of NVAO to assess programmes in Curacao, Aruba and St. Maarten is based on an
agreement with the Dutch Minister of Education.
The recognition of NVAO’s decisions is not only based on its formal authority, but also on support
from the field of higher education, politics and society at large. Permanent platforms for
consultation of stakeholders in both The Netherlands and Flanders are in place.
Analysis
NVAO is the binational QA agency for The Netherlands and Flanders, established by treaty and well
founded in legislation. Its binationalism is being felt as one of the major values to assure
independence and neutrality in rather small jurisdictions. The formal recognition of NVAO as
accreditation agency by public authorities is up till now supported by the public in a larger sense.
24/60

Panel conclusion: fully compliant
ESG 3.3 INDEPENDENCE
Standard:
Agencies should be independent and act autonomously. They should have full responsibility for
their operations and the outcomes of those operations without third party influence.
ENQA-review 2012
Standard 3.3 was in ESG 2005 Standard 3.6. The panel concluded ‘fully compliant’. No
recommendations were given.
Evidence
Organisational independence
From the start of the accreditation system in The Netherlands and Flanders NVAO has been fully
independent regarding decision making. Members of NVAO’s Executive and General Board are
mandated for four years by the Committee of Ministers. They are appointed in a strictly personal
capacity and not as representatives of any organisation. In order to guarantee the independence of
board and staff members, NVAO has ruled that members of both categories cannot participate in
the decision making on or applications from institutions or programmes they have been associated
with in any form during the previous five years. Members of the Board have to be completely
independent in making decisions. If there is a specific application where this independence cannot
be guaranteed, the member of the Board will withdraw from the decision-making process for the
programme concerned. NVAO is accountable to the Committee of Ministers, which approves its
budget, the annual report and the annual accounts. In accordance with the Treaty, the Committee of
Ministers can only intervene in case of serious neglect by NVAO of its accreditation task. The
Committee of Ministers can thus only intervene in the functioning of NVAO, but not in NVAO’s
decision-making nor internal allocation of the headline budget allocation.
Operational independence
NVAO’s assessment frameworks and procedures of initial programme accreditation, institutional
reviews and the profile of external experts are clearly defined. There is a Code of Conduct outlining
requirements for independence. The value of independence is emphasized in the Internal Integrity
Code. There is a standard procedure in the assessment and approval of all panels. NVAO has control
over the recruitment, deployment and organisation of its human resource and financial resource is
allocated according to the decisions of the Board. Since the start of NVAO its operational
independence has, like its organisational independence never been put into question. During the site
visit stakeholders confirmed NVAO's decision-making is independent, and mentioned this as the
main advantage of having NVAO as accrediting body.
Independence of formal outcomes
Independence of NVAO and decisions of panels are shown in official documentation. Experts in all
reviews, coordinated by NVAO and other agencies, are checked for independence according to the
profile of NVAO. In NL NVAO approves of panels according to the guideline for the composition of
panels. In Flanders NVAO issues a non-binding advice. Interviews show that in the very few cases
NVAO's advice was not followed up this did not relate to the independence of experts.
Analysis
Independence is one of NVAO’s core values and guaranteed by law. NVAO has full responsibility for
its operations, methods and procedures. Interviews with NVAO and stakeholders confirm this.
Legally binding assessment standards are developed together with stakeholders.
25/60

Commendation
NVAO is commended for the processes it has in place to ensure independence including the Code of
Conduct and Internal Integrity Code. The professionalism and integrity of NVAO staff was highlighted
by stakeholders during the review and apparent to panel members throughout the review process.
Panel conclusion: fully compliant
ESG 3.4 THEMATIC ANALYSIS
Standard:
Agencies should regularly publish reports that describe and analyse the general findings of their
external quality assurance activities.
ENQA review 2012
Standard 3.4 is a slightly modified formulation of ESG 2005 Standard 2.8 ‘System-wide analysis’.
In the 2102 review the panel concluded on this standard ‘substantially compliant’. The panel
recommended:
‘[NVAO] gives a high priority to
a) the identification of the interested parties for system-wide analyses and of their
needs;
b) the definition of a realistic schedule of system-wide analyses; and
c) the production of system-wide analyses corresponding to the needs which were
identified’.
In a letter to NVAO by the ENQA Board, January 2013, system-wide analysis is mentioned in the
context of the recommendation: ‘.... Clustering of assessments may be beneficiary to improve
consistency in assessments and decision making, as well as to answer the stakeholders’ need for
more system-wide analysis’.
Evidence
Recent thematic analysis
In the SAR and in Annex 10 several instances of thematic analysis are indicated that were
undertaken in the past years. These include:
 Knowledge co-creation on quality culture in professional bachelor’s Communication studies;
 Accreditations of academic programmes Humanities and Communication;
 Employability of professional bachelors from an international perspective;
 Four years of new accreditation system in The Netherlands;
 The profile of Research master’s programmes;
 Teacher training programmes at universities;
 Teacher training programmes for primary schools;
 Assessment and demonstration of achieved learning outcomes.
In many of these thematic analyses information gained from review reports is actively being used.
Intentions for future thematic analysis
In the SAR also the importance of future thematic analysis is stressed several times. Examples of
intended activities are:
 System-wide analyses in the new Flemish system of institutional reviews;
 Periodically selected domains or topics for an analysis in the new system of clustered
reviews;
 Risk-based assessments and thematic analyses that complement the monitoring of the
quality of study programmes in institutions, as a complement to institutional audits and

26/60

reviews in The Netherlands and Flanders. Such analyses could also focus on common
practices or specific educational concepts.
The SAR also indicates instances in which existing activity could be extended to thematic analysis.
 cluster-analysis for large clusters of reports which primarily serve the decision-making can
be extended to more thematic analyses;
 NVAO’s annual reports which contain a full overview of all decisions taken and the scores
and provide a comprehensive account of the yearly activities of the NVAO;
 periodical publications such as the half-yearly bulletins (Jaarbericht) with an oversight of the
number of applications processed, the scores reached in assessments, and short comments
on developments and trends in the last year.
Appreciation and priorities
During the site visit a number of stakeholders, including representatives from HE organisations,
ministries and student organisations stated that they would find thematic analyses useful in
understanding and contributing to further quality developments in the higher education system.
Student organisations suggested key areas for thematic analysis, whilst other stakeholders viewed
the provision of such information important during the present period of change.
The Board informed the panel about research among stakeholders, conducted by an independent
agency, commissioned by NVAO. This research showed that stakeholders had mixed feelings on
thematic analysis. Some would like to see more of it, others less. In this respect the context of higher
education in The Netherlands and Flanders proves to be relevant, as a variety of HE-agencies
conduct thematic analyses. In Flanders VLUHR-KZ used to provide cluster-wide analyses as part of
their assessment reports. In The Netherlands the Inspectorate of Higher Education is explicitly
tasked with conducting thematic analysis, including making use of NVAO’s assessments reports.
During the site visit, the representative of the Inspectorate mentioned that NVAO and Inspectorate
are working on a joint agenda of thematic analyses. NVAO has budgeted significantly for thematic
analysis in NL from 2017 onwards. In Flanders thematic analysis is not foreseen before 2019.
Analysis
The review team considers that although thematic analysis has been carried out to a considerable
extent, there is a lack of clarity regarding the nature, role and function of thematic analysis in future
activity. NVAO has not yet solved all problems related to undertaking thematic analysis proactively
as a regular function of NVAO activities. Additionally, the processes for identifying and prioritising
thematic analysis are unclear. This is exemplified by two conflicting approaches:
 The choice to do thematic analysis depends on the demand for such an analysis. When there
is a lot of public attention on a specific domain, or when there have been important
changes, NVAO will produce a thematic analysis.
 In general, thematic analyses are produced either at the request of one of the ministries of
education in both countries, or because NVAO feels it falls within its duty of providing public
information.
The review panel concludes that there is need to find a way out of these two approaches.
Recommendation
NVAO reviews the approach to thematic analysis and in consultation with stakeholders, identifies a
common purpose and a means to regularize processes and priorities in this area.
Panel conclusion: substantially compliant

27/60

ESG 3.5 RESOURCES
Standard:
Agencies should have adequate and appropriate resources, both human and financial, to carry out
their work.
ENQA review 2012
This standard was in the ESG-2005 part of Standard 3.4 Resources. In the 2012 ENQA-review the
panel concluded ‘fully compliant’. No recommendations were given.
Evidence
Financial resources
As shown in the paragraph ‘NVAO’S Funding’ (p17) NVAO draws its financial means from subsidies of
the Dutch and Flanders governments and from fees. The SAR notes as a threat that financial support
from both the Dutch and Flemish governments was steadily decreasing. To some extent NVAO had
to economise its operations and to optimise its internal efficiency. A further decrease in financial
support could eventually lead to a diminution of its activities, especially those that are not directly
related to its statutory tasks. However, the review panel heard from the Executive Board that a
satisfactory budget settlement from the Dutch Ministry of Education was imminent. This was
confirmed by Dutch and Flanders Ministry representatives. The Flanders Ministry representatives
also confirmed that they would maintain their funding in the ratio 40:60.
Human resources
The review panel was given access to three year work plans which indicated coverage of activities
into the future. The Executive Board also advised their approaches to prioritisation of work and
reorganisation of staffing structures to ensure fulfilment of their duties. Staff of NVAO indicated that
they enjoyed a wide variety of activity within their portfolio of work and that supervision and
management ensured they were able to conduct their duties without being overloaded.
Stakeholders generally praised the professionalism of NVAO staff.
Analysis
The presented evidence creates a sense of confidence with the review panel that resources both
financial and human are safe for the years to come.
Panel conclusion: fully compliant
ESG 3.6 INTERNAL QUALITY ASSURANCE AND PROFESSIONAL CONDUCT
Standard:
Agencies should have in place processes for internal quality assurance related to defining, assuring
and enhancing the quality and integrity of their activities.
ENQA review 2012
Standard 3.6 was in the ESG 2005 part of Standard 3.8 ‘Accountability procedures’. On this standard
the ESG 2012 review panel concluded ‘fully compliant’. No recommendations were given.
Evidence
Inputs for quality assurance
Internal quality assurance at NVAO operates both at a strategic and operational level. Inputs for the
strategic level are for instance recommendations by ENQA-reviews and preferences expressed by
stakeholders in formulating the contours of new (2016) accreditation frameworks and the Strategy
document (2017-2020). Inputs for the operational level are for instance checks on NVAO’s
assessment processes by the online tool Survey Monkey; discussions with stakeholders, such as the
28/60

assessment agencies, institutions and student organisations; the NVAO resonance group, meeting
twice a year; feedback on a permanent basis in consultation with the Advisory Council; the
overviews of evaluation activities; measures of improvement as formulated in documents; weekly
discussions in the regular staff-meetings; peer-to-peer coaching of NVAO-staff and project process
coordination. On the basis of these inputs lines of work are improved wherever possible.
Internal quality assurance policy and practice
The SAR outlines various processes in place for defining, assuring and enhancing the quality and
integrity of its activities, which together form an integrated policy that covers all of NVAO’s
activities. Internal quality assurance is based on a plan-do-check-act approach. An ongoing
evaluative and iterative process monitors the effectiveness of procedures, which are adjusted, when
necessary to improve efficiency and effectiveness. There are an Integrity Code, a Code of Conduct
and clear procedures for reviews and assessments. The protocol for internal quality assurance and
the annual work plans are formally decided by the NVAO Board. A coordinator supervises the
various activities and regularly consults with the responsible Board member. The current work plan
and evaluation scheme was established in December 2014. The review team observed that the full
internal quality assurance policy was not available on the website, but this was rectified during the
site visit.
Internal quality assurance culture
The Strategy 2017-2020 mentions that NVAO promotes an organisational culture in which the
pursuit of quality constitutes a self-evident component. This is given shape not only through training
and courses, but also with peer group sessions and reinforcement of peer reviews. Interviews with
management and staff members during the site visit reflect a strong quality culture, integrity and a
supervisory management approach which fosters reflection focussed on enhancement of activities.
Appreciation of internal quality assurance
In meetings with the review panel stakeholders of the agency, which included HEIs, students,
teachers and employers, valued and trusted the approach of NVAO and indicated effective means of
communication regarding reviews and assessments. The review panel had the opportunity to look at
the internal NVAO website of the internal quality assurance, and was impressed by the range of
activities and the systematic approach. The review panel also noticed some loose loops, like notevaluating the secretaries-training, and not checking the publishing time of reports, which the staff
and internal quality assurance coordinator were ready to recognize as elements to be improved.
Analysis
NVAO has a strong commitment to internal quality assurance, which is expressed in policy-papers,
engagement by the Board, a quality assurance coordinator, an impressive system of quality
assurance and above all in a strong internal quality culture. The internal quality assurance system as
such is mature. However, the review panel also observed some omissions in the quality system
though. On balance the panel concludes to substantial compliance.
Recommendation
NVAO remains alert on loose loops in the plan-do-check-act-cycle of the quality assurance system.
Panel conclusion: substantially compliant
ESG 3.7 CYCLICAL EXTERNAL REVIEW OF AGENCIES
Standard:
Agencies should undergo an external review at least once every five years in order to demonstrate
their compliance with the ESG.
29/60

ENQA review 2012
ESG 2015 Standard 3.7 was formerly part of ESG 2005 Standard 3.8 ‘Accountability procedures’.
The 2012 review panel’s conclusion on standard ESG 2005 3.8 was ‘fully compliant’. No
recommendations were given.
Evidence
Cyclical reviews
Since its inception in 2005 NVAO has undergone ENQA reviews in 2007 and 2012. The present 2017
review follows this five-yearly routine. The ENQA review reports form NVAO’s periodic report about
its activities to the Ministers of Education, as prescribed in the Treaty between The Netherlands and
Flanders. Apart from the ENQA reviews also regular reviews by other agencies take place such as the
Dutch and Flemish Accounting Offices and (in NL) the Inspectorate of Education. These are meant to
constitute main input for the 4-yearly reports of the Dutch and Flemish Ministers of Education to
their respective Parliaments.
Follow-up of recommendations
According to the 2012-ENQA-review panel in the period 2007-2012 NVAO had paid due attention to
recommendations given in the 2007 review, even though not all 2007 recommendations had been
followed up to an extent that full compliance on all ESG 2005 Standards was achieved. As regards
the recommendations given in the 2012 ENQA-review the present review panel observes the same
for the period 2012-2017. However, in this respect it is also worth mentioning that the demands of
some ESG 2015 Standards are considerably higher than those of ESG 2005.
Analysis
NVAO is regularly reviewed by ENQA-review panels and various other external bodies.
Recommendations of reviews are taken seriously and contribute significantly to the further
development of the agency.
Panel conclusion: fully compliant

30/60

ESG PART 2: EXTERNAL QUALITY ASSURANCE
ESG 2.1 CONSIDERATION OF INTERNAL QUALITY ASSURANCE
Standard:
External quality assurance should address the effectiveness of the internal quality assurance
processes described in Part 1 of the ESG.
ENQA review 2012
Standard 2.1 was in ESG 2005 also Standard 2.1 ‘Use of internal quality assurance procedures’. The
panel concluded ‘fully compliant’, but still added the recommendation: [NVAO] makes the link
between the NVAO standards and ESG Part 1 more explicit in the assessment frameworks.
Evidence
Mapping in general
As stated in the SAR and confirmed by Board and staff during the site visit, compliance with ESG is a
basic principle of the NVAO frameworks for The Netherlands and Flanders. As regards the inclusion
of ESG part 1 Standards in the frameworks NVAO staff prepared, on request of the panel a document
‘Mapping ESG part 1 and frameworks NL-FL‘ in which all NVAO-frameworks are systematically
mapped against ESG part 1. A summary of this analysis is given in Table 3. The NVAO-analysis shows
that, apart from the framework for distinctive features, which in a sense is an addition to the other
frameworks, mapping with ESG is assured in all NVAO-frameworks: programme accreditation in NL
and FL; initial programme accreditation in NL and FL; accreditation of joint programmes, institutional
audits in NL, and institutional reviews in FL. The programme assessment in Curaçao, Aruba and St.
Maarten, which has identical frameworks as in NL, likewise reflects the same kind of mapping with
ESG. As mentioned before there is no institutional assessment in C/A/StM.
NL-frameworks
As the SAR notes NL frameworks of 2016 were specifically designed with ESG 2015 in mind. The
institutional audit complements the limited programme accreditation and covers elements of ESG
Part 1 primarily at institutional level. These are subject to an audit that takes the institutional
perspective and consists of a first visit with a general view, followed by review trails on specific
topics. The extensive framework for programme accreditation is used for programmes in institutions
that did not apply for an institutional audit or did not pass the audit. It covers all elements of ESG. In
the context of an institutional audit, some standards apply to the institution rather than to a
programme, or have a different meaning than in the context of a programme. This principle has not
changed with the new NL framework of 2016.
FL-frameworks
In the Flemish system, the elements of ESG Part 1 pertaining to quality assurance are included in the
Framework for Institutional Reviews – Flanders 2015-2017. The Quality Code is a formal supplement
to the framework for the institutional review. It focuses on those elements of the ESG that relate to
the quality of education at programme level. When assessing an institution’s conduct of assuring the
quality of programmes, the review panel verifies how these quality features have been embedded in
this conduct. The Quality Code includes an annex that outlines the relationship between the quality
features and the ESG. For lack of evidence laid down in reports the panel could not establish how the
mapping with ESG works out in practice. A remarkable feature of the pilot is that for a well-defined
number of programmes, programme accreditation is put ‘on hold’ and the envisaged cycle-time
extended.
Panels, stakeholders, staff and reports
During the site visit representatives of assessment agencies, and chairs and secretaries of panels
proved very keen on adhering to NVAO-standards. They assumed that in doing so, the ESG 2015
31/60

demands are satisfied. During the site visit the review panel checked with students and teachers
whether in practice specific elements of ESG Part 1 were covered during the reviews, which proved
to be the case. NVAO-staff assured the panel that in assessments-reports the adherence to NVAOstandards is checked and thus implicitly also whether ESG Part 1 is adhered to. In a sample of
reports the review panel saw that considerable attention to ESG Part 1 Standards was paid.
Table 3: Mapping of ESG Part 1 standards on NVAO frameworks
Institutional
audit
IA-NL/IR-FL

Quality
Code

Standaards
(1) Policy and procedures for
quality assurance
(2) Design and approval of
programmes
(3) Student-centred learning,
teaching and assessment.

Standards
1,2/1,2,3,4
(NL16: 1,2)
1,2/1,2,3
(NL16:1,2)
1,2/1,2
(NL16:1)

Standards
-

(4)
Student
admission,
progression, recognition and
certification

2/2,3
(NL16: 2)

G

(5) Teaching staff
(6) Learning resources and
student support
(7) Information management

2/2 (NL16:1,2)
2/2
(NL16:2)
3,4/3
(NL16:3,4)
2/2 (NL16:2)
3,4/3,4
(NL16:3,4)

C
D

A,B,E
B,C,D,E,F,G

Programme
accreditation
LPA-NL/EPA-NL/ PAFL/PA-NL16
Standards
*/9/4
(NL16: */9)
1,2/1,2,3,4,9/1,2
(NL16:1,2/1,2,3,4,9)
1,2,3,4/1,2,3,4,10,
11/1,2,3
(NL16:1,2,3,4/1,2,3,4,1
0,11)
1,2,4/1,2,5,9,11/1,
2,3
(NL16:1,2,4/1,2,5,8,9,1
1)
2/6/2 (NL16:2/6)
2/7,8/2
(NL16:2/7,8)
*/9/2,4 (NL16:*/9)

Initial accreditation
LIA-NL/EIA-NL/
IA-FL/PA-NL16
Standards
*/9/4
(NL16: */9)
1,2/1,2,3,4,9/1,2
(NL16:1,2/1,2,3,4,9)
1,2,3,4/1,2,3,4,10,12/1
,2,3
(NL16:1,2,3,4/1,2,34,1
0,11)
1,2,5/1,2,5,9,12/1,2,3
(NL16: 1,2,4/1,2,5,
8,9,11)
2/6/2 (NL16:2/6)
2/7,8/2
(NL16:2/7,8)
*/9/2,4 (NL16:*/9)

(8) Public information
G, H
2/8/2 (NL16:2/8,9)
2/8/2 (NL16:2/8)
(9) Ongoing monitoring and
B, D, F, H
*/9/4 (NL16:*/9)
*/9/4 (NL16:*/9)
periodic
review
of
programmes
(10) Cyclical external quality 3,4/3,4
**
*/9/4 (NL16:*/9)
*/9/4 (NL16:*/9)
assurance
(NL16:3,4)
* Covered in institutional audit. ** Falls under the institutional review.
Abbreviations: NL: The Netherlands; FL: Flanders; IA-NL: institutional audit; IR-FL: institutional review; QC:
Quality Code; LIA-NL: limited initial accreditation; EIA-NL: extensive initial accreditation; LPA-NL: limited
programme accreditation; EPA-NL: extensive programme accreditation; PA-FL: programme assessment; IA-FL:
initial accreditation; NL16: x/x: new Dutch framework (limited/extensive).

Analysis
In all NVAO-frameworks coverage of the ESG Part 1 standards is established. It is effective in actual
quality assurance at institutional and programme level in all types of accreditation. The putting ‘on
hold’ of accreditation of a limited number of programmes during the FL-institutional review pilot
stretches the cyclicity, but this is somehow counterbalanced by the intended cyclical institutional
review.
Commendation
NVAO is commended on the systematic coverage of the ESG Part 1 Standards in all frameworks that
are presently in use.
Panel conclusion: fully compliant
32/60

ESG 2.2 DESIGNING METHODOLOGIES FIT FOR PURPOSE
Standard:
External quality assurance should be defined and designed specifically to ensure its fitness to
achieve the aims and objectives set for it, while taking into account relevant regulations.
Stakeholders should be involved in its design and continuous improvement.
ENQA review 2012
Standard 2.2 formed part of ESG 2005 Standard 2.2 ‘Development of external quality assurance
processes’ and Standard 2.4 ‘Processes fit for purpose’. On both standards 2.2 and 2.4 the panel
concluded ‘substantially compliant’. The panel formulated one recommendation: [NVAO] further
strengthens the focus on quality improvement and enhancement of the HEI’s.
Evidence
New frameworks
As described in ‘NVAO’S FUNCTIONS, ACTIVITIES, PROCEDURES’ (p15-16) both in NL and FL new
frameworks have been developed over the past years in order to accommodate wishes from the
field with respect to accreditation based on justified trust and the reduction of administrative
burden.
In The Netherlands the new assessment framework was published in September 2016. It integrates
many procedures related to (initial) accreditation or the institutional audit, such as: distinctive
features, changes of name or degree, change in the duration of a programme, recognition as a
research master or joint degree, additional admission criteria, which are presently described in
separate protocols. The new framework takes the general standards as a starting point and
incorporates the additional procedure, resulting in an integrated and concise framework. The
framework creates space to institutions to demonstrate improved quality through increased
ownership, but ensures a threshold of basic quality for new and existing programmes, on which
NVAO has the mandate to decide.
In Flanders new frameworks were published in March 2015. In particular the framework for
institutional review and its corresponding quality code signals a significant step away from previous
frameworks, as they create much space for institutions’ own interpretation of quality and quality
assurance, while still remaining within the scope of ESG. The framework also includes the way by
which institutions secure the quality of programmes by means of programme evaluation.
Stakeholder’s involvement
Stakeholders confirm that they were involved in development of frameworks in both FL and NL.
The new NL framework is based on extensive evaluation and discussion with the field and the
political goals defined by the Ministry of Education. At every step in this process institutions,
students and other stakeholders were consulted and given the opportunity to comment on
proposals. In FL the framework for the pilot institutional review, in particular the quality code, were
developed in close collaboration with stakeholders. Institutions felt that the existing frameworks for
programme accreditation were not fit for purpose anymore, as they didn’t create impulses for
improvement of programmes within the institution, in particular for those programmes that had
been reviewed various times already.
An interesting outcome of the discussions during the site visit was that in assessments the
differentiation between excellent, good, satisfactory and unsatisfactory in NL regular programme
accreditation still existed, while assessment agencies and various staff of institutions maintained
that this practice was out of date and frequently resulted in unfruitful debates. To the panel the
33/60

purpose of the continuation of this grading system was unclear. The panel is aware of the legislative
context of the grading system and suggests that when a system causes too much unnecessary
confusion, NVAO should advocate a process of review of the system, together with all relevant
stakeholders to ensure fitness for purpose.
Official approval
All Dutch and Flemish frameworks have passed approval by the parliaments in both countries. This
not only applies to (initial) accreditation and institutional audits and reviews, but also to the design
of processes in assessments related to accreditation such as distinctive features, combination of
programmes, extension of the duration of programmes, changes in name and degree, additional
admission criteria etc.
Analysis
The panel considers that over the past years NVAO has made considerable progress in making
frameworks fit for purpose. Stakeholders’ participation is obvious and contributes to better finetuning, even though some specific elements of old frameworks remain unsatisfactory in the eyes of
some stakeholders. The recommendation of the 2012 ENQA review panel to strengthen the focus on
quality improvement and enhancement of the HEI’s has definitely been followed-up properly.
Justified trust as regards institutions has gained in importance. The panel noticed during the site visit
high expectations of the FL-HEIs towards the institutional reviews and the enthusiasm for this
approach was palpable. NVAO recognizes that the management of expectations would be a relevant
policy issue, also for the sake of continued public trust in the new system.
Panel conclusion: fully compliant
ESG 2.3 IMPLEMENTING PROCESSES
Standard:
External quality assurance processes should be reliable, useful, pre-defined, implemented
consistently and published. They include:
- a self-assessment or equivalent
- an external assessment normally including a site visit
- a report resulting from the external assessment
- a consistent follow-up
ENQA review 2012
Standard 2.3 was formerly ESG 2005 Standard 3.7. On this standard the panel concluded ‘fully
compliant’. No recommendation was given. In a letter of 29 January 2013, the ENQA Board gave a
recommendation on Standard 3.7. referring to elements presently falling under ESG2015 Standard
3.4 and 2.5. This recommendation is discussed under these standards.
Evidence
Self-assessment, site visit and report
As the SAR indicates all assessments for (initial) accreditation of programmes and institutional audits
and reviews are based on a self-evaluation report. A site visit is part of the assessment procedures
for (initial) accreditation and institutional audit or review and related procedures. All assessments
for (initial) accreditation of programmes and institutional audits and reviews, regardless of the final
decision, lead to a public assessment report by the panel and a decision by NVAO according to
prescribed formats. The procedures for assessments are laid down in the accreditation frameworks
and are compulsory for assessment agencies. NVAO checks if all steps have been followed for each
application. The SAR describes that this applies to all accreditation processes, whether they are

34/60

directed to initial accreditation (NL, FL, C/A/StM), existing programmes (NL/FL), institutional audits
or reviews (NL, FL), joint degrees or distinctive features.
Quality of assessment reviews
The review panel noted the focus on open dialogue between peers and ways to encourage this,
including the inclusion of an open consultation, during which students, teachers or others can
approach the panel anonymously with questions or complaints about the programme. The structure
and elements of assessments are continuously improved in response to feedback from stakeholders
and panels. NVAO ensures consistency between the handling of this procedure by different panels
and process coordinators by developing internal guidelines for coordinators, training panel chairs
and secretaries, by clustering analyses, and by critically monitoring the practices of assessment
agencies.
Follow up
The SAR noted that the follow up of recommendations and improvements is primarily the
responsibility of the institution. NVAO monitors the follow up of recommendations from previous
assessments in its analysis of applications. Follow up after two years is formally foreseen in the case
of conditionally positive decisions and extended accreditation during improvement periods. The
institutional audit looks specifically at internal mechanisms for implementing and following
improvements after programme assessments.
The review panel wished to test the robustness of the processes and clarify the situation regarding
follow up, which did not seem compliant with EQAR interpretation. The team tested the robustness
of the processes outlined by:
a) review of the public website;
b) requesting further information regarding the documentation for the process in each
jurisdiction;
c) undertaking a limited analysis of reports; and
d) clarifying points in site meetings with key stakeholders.
The review panel found that NVAO propose recommendations both in instances where institutions
receive accreditation for the full period and also when institutions must undergo a recovery period,
normally lasting two years. Follow up is undertaken in the case of conditionally positive decisions
and extended accreditation during improvement periods. In cases where there is no recovery period,
recommendations will not be reviewed until the next application. Although NVAO maintains that the
distinction between conditions and recommendations is clear, in developing a common European
terminology and interpretation, the use of this distinction (and the different websites where
conditions and recommendations are meant to be published, see Standard 2.6) may be considered
problematic.
Issues in implementing processes
During the site visit various issues concerning implementing processes were discussed:
 Clustering NL reviews of existing programmes
In 2016 cluster assessment for NL existing programmes, already realized for universities, is
also implemented for Dutch higher professional education. However both assessment
agencies and institutions perceive an increase of workload due to the complexity of the
evaluation and sometimes unfavourable operational conditions.
 Reporting in FL institutional review
While generally speaking implementation of the FL pilot institutional review meets with
great enthusiasm and appears to work well, one feature of the implementation of the pilot is
that reports of the pilot will only be published after the pilot is over. Single institutions
35/60



undergoing institutional review do get their report at the end of the procedure. However, it
is not published -and therefore publicly accessible- until the pilot is over. NVAO informed
the panel that FL institutions had pleaded for this arrangement. During the site visit one
institution told the panel that waiting for a report for over a year was too long to work on
further improving quality.
Grading in NL programme accreditation
The implementation of the grading in existing NL programme accreditation appears to cause
difficulties even many years after introduction of this system.
Administrative burden
So far none of the institutions and staff with whom the review panel spoke had experienced
a noticeable reduction of administrative burden because of new frameworks and processes.
At the same time it was pointed out that in an NL-institution by now the ‘burden’ of quality
assurance was caused more by internal quality assurance (70%) than by external quality
assurance (30%), which may also be a fair measure of the extent to which quality assurance
has been internalized in institutions.

Analysis
The implementation processes as developed by NVAO over the last years follow ESG-standards and
good practice closely, even though at times elements need further consideration. However the
phrasing of follow up by NVAO remains to some extent difficult to understand by outsiders. On
balance the panel decides on substantial compliance.
Recommendation
NVAO remains sensitive regarding issues of implementation as experienced by stakeholders and
adopts a clearer terminology to differentiate between substantively different approaches to follow
up of decisions.
Panel conclusion: substantially compliant
ESG 2.4 PEER-REVIEW EXPERTS
Standard:
External quality assurance should be carried out by groups of external experts that include (a)
student member(s).
ENQA review 2012.
This standard was in ESG2005 part of Standard 3.7 External quality assurance criteria and processes
used by the members. In ESG 2015 it has become a separate standard.
The 2012 review panel concluded on ESG 2005 standard 3.7 ‘fully compliant’. No recommendations
were given.
Evidence
Panels in general
NVAO has clear guidelines for the composition of panels and the procedure of panel approval.
Panels should consist of at least four members (at least five for the Dutch institutional audit),
including a student member. The guidelines provide criteria for profile (skills/expertise) of experts
including knowledge of the discipline, of the professional field, of education in the domain, of the
international dimension, experience in reviewing or auditing and knowledge of assessment. There is
a code of conduct for panel members and panel members sign a no conflict of interest document.
The same criteria for selection of peers are used for evaluations in the Caribbean, with the addition
that one of the peers should be from the Caribbean. Also for joint degree programme international
expertise is taken care of.
36/60

For the composition of NVAO-panels for initial programme accreditation and institutional audits
NVAO uses its binational status to have Flemish people in the Dutch evaluation, and vice versa.
Other international experts are occasionally used, but less frequent. For institutional audits NVAO
looks for experienced administrators.
NVAO has a register of experts that have previously participated in reviews. This also applies to
students who participate in NVAO-led accreditations. NVAO always consults the institution before
appointing a panel. NVAO tries to avoid engaging the same experts as those of the assessment
agencies in order to obtain a variety of perspectives in the assessments.
Panels in Flanders
In line with a legal prescript in Flanders panel members for accreditations following the current
framework are selected by VLUHR-KZ. NVAO assesses panel proposals and gives advice. In a limited
number of cases the NVAO gave a negative advice on a panel, which was ignored by VLUHR-KZ.
These cases referred to a combination of skills, not to the independence of the panel.
NVAO can only advise on the composition of the panel. The panel concluded that this advisory
function is sufficient to balance the autonomy of VLUHR-KZ, national legal requirements and the
requirements of NVAO. There are adequate procedures post accreditation to ensure that the
process is conducted effectively and within guidelines. A recent development is that other
assessment agencies than VLUHR-KZ are allowed to carry out reviews, provided they are registered
with EQAR. In the pilot institutional review in Flanders, NVAO selects panel members and decides on
all panel compositions.
Panels in The Netherlands
For regular programme accreditation in NL institutions propose peer experts for NVAO to approve,
almost always in collaboration with assessment agencies. NVAO decides on the proposals for
composition of the assessment panels. In the case of distinctive features NVAO assesses whether the
panel has sufficient specialist expertise for the added procedure.
Training and quality of panel members
For initial programme accreditation and institutional audits NVAO provides training for experts.
However, during the site visit representatives of employers informed the panel that they were
briefed only, without further training. NVAO does provide training for secretaries of initial
programme evaluations, but during the site visit there were mixed opinions about these trainings. In
any case panel chairs were very positive about the quality of both secretaries and process
coordinators. Assessment agencies provide training to experts for regular programme assessment
and inform them of the procedures.
There are feedback mechanisms (as described under Standard 3.6) to monitor the quality of chairs,
secretaries and process coordinators. During the site visit various external evaluators of NVAO
praised the quality of chairs and experts and their judgements.
Analysis
The selection of panel members and the composition of panels is taken great care of both in NL and
FL. There are mechanisms for a balanced panel composition and an expertise that is suitable for the
specific tasks (initial accreditation, institutional audit, distinctive features joint degrees), even
though international input in panels still is a point of attention.
Panel conclusion: fully compliant
37/60

ESG 2.5 CRITERIA FOR OUTCOMES
Standard:
Any outcomes or judgements made as the result of external quality assurance should be based on
explicit and published criteria that are applied consistently, irrespective of whether the process
leads to a formal decision.
ENQA review 2012
Standard 2.5 was in ESG 2005 Standard 2.3 ‘Criteria for decisions’. The panel concluded ‘partially
compliant’. The recommendations formulated were:
 [NVAO] refines the descriptions of the aims and objectives, ensures that they are prominent in
the frameworks and show how the various elements of the frameworks contribute to the aims
and objectives;
 [NVAO] formulates explicit and public criteria about how it reaches its accreditation decisions
in relation to the programme assessments in all cases;
 [NVAO] establishes a clear procedure on how to handle cases where the conclusions in the
assessment report are not accepted by NVAO
 [NVAO] strengthens the predictability of the timeframe and efficiency of its decision-making
process;
In its letter to NVAO of 29 January 2013 the Board of ENQA made an additional recommendation ‘to
improve the consistency of accreditation judgments reached’ also addresses the issue of decisionmaking.
In the Terms of Reference for the 2017-Review special mention was made of ESG2015, Standard 2.5
(‘Criteria for outcomes’): ‘It should be addressed whether NVAO has clarified (within its own
operational documents and without questioning the holistic nature of its judgements) the criteria and
decision-making process used in making decisions on the accreditation of existing programmes in The
Netherlands, including the way in which it ensures consistency of its decisions based on reviews
undertaken by different agencies’.
Evidence
Developments since the 2012 ENQA-review
Since the previous 2012 review NVAO has taken a number of steps to improve the decision-making
process. According to the SAR these are:
 introduction of cluster-based accreditation and project based handling of applications;
 further clarification of criteria and decision rules for the judgements;
 a project to make process coordinators more aware of their responsibilities as regards
application of criteria;
 a training programme for panel chairs and secretaries to enhance the consistent use of
criteria;
 approval (NL) or formal advice (FL) on panel compositions;
 a protocol for the assessment of students’ final assignments;
 detailed internal manuals for NVAO-staff for programme accreditations;
 a ‘four-eye and two-step’ decision making process on the Board;
 asking for additional information when a judgement in an assessment report is not
convincing;
 regular consultation with assessment agencies;
 measures (‘fixed and explicit rules’) to ensure that the possibility of small differences in
interpreting generic frameworks doesn’t stand in the way of consistency.
These measures were judged positively in the follow-up evaluation in 2014.

38/60

Conditions and improvement periods
In the Dutch system, a (conditionally) positive decision in the initial accreditation procedure entitles
the programme to award valid degrees and diplomas. In the case of a conditional decision, the
programme is granted up to two years to meet the conditions set in the assessment. Failure to do so
leads to the immediate loss of accreditation. In the case of a negative decision on accreditation for
existing programmes, NVAO can grant an improvement period. It means that the duration of the
accreditation is extended for up to two (NL) or three (FL) years, allowing the programme to
implement measures to improve the weaknesses or deficiencies indicated by the panel, and have
these assessed. When the result of the assessment is negative, accreditation is withdrawn. While the
Flemish system contained an improvement period from the start, it became an effective tool in The
Netherlands only in 2010. A (conditionally) positive outcome of the Dutch institutional audit allows
institutions to make use of the framework for limited programme assessments with four standards:
intended learning outcomes, the curriculum, the assessment system and the achieved learning
outcomes (when available for new programmes). The fulfilment of conditions is subject to
reassessment within two years. In Flanders, there are no formal consequences to the institutional
review in the current pilot round.
Grading
The frameworks contain precise guidelines for calibrating the scores used in the judgements. The
initial programme accreditation or assessment (NL, FL, C/A/StM), the institutional audit (NL) and the
regular programme accreditation (FL) is straightforward with a minimum of grading nuances:
pass/fail/conditional. There is little discussion on this type of decisions, with hardly problematic
cases over the past years. The calibration in regular programme accreditation (NL) and distinctive
features (NL) ranges from insufficient to excellent. This system is perceived by a number of NLstakeholders as an unnecessary complication, which affects consistency, in particular as regards the
interpretation of the yes-decisions.
‘Difficult’ cases
In a stratified sample of NVAO decisions the review panel detected some ‘difficult cases’ in which the
Board took decisions, which extended standard procedures. These cases appear to be a tiny
minority, for which general rules may be difficult to establish, although here again the NL-grading
system for regular programmes complicated one such case.
Analysis
Given the evidence provided and the opinions of stakeholders the panel is convinced that there has
been good progress in clarifying criteria for outcomes, as far they are concerned on the
yes/no/conditional decision. However, the grading of outcomes from insufficient to excellent
remains a difficult issue, on which further reflection by NVAO and other relevant parties is needed.
Panel recommendation
NVAO reflects on the grading system as presently applied and considers making it as straightforward
as necessary for the purpose of yes/no/conditional accreditation.
Panel conclusion: substantially compliant
ESG 2.6 REPORTING
Standard:
Full reports by the experts should be published, clear and accessible to the academic community,
external partners and other interested individuals. If the agency takes any formal decision based
on the reports, the decision should be published together with the report.

39/60

ENQA-review 2012
Standard 2.6 was in ESG 2005 Standard 2.5 ‘Reporting’. On this standard the panel concluded
‘substantially compliant’. The recommendation given was: ‘[NVAO] clarifies
a) the purpose of every kind of report;
b) the readership; and
c) the needs of the various kinds of readers in order to enhance the readability’.
In the Terms of Reference for the 2017-ENQA-review special mention was made of ESG2015,
Standard 2.6 (‘Reporting’): ‘Issues related to the readability of its reports for its defined target
audience should be addressed’.
Evidence
Publishing reports
All NL and FL reports on initial and regular programme accreditation and all institutional reports are
published at the NVAO website, including those that resulted in a negative decision or conclusion.
This applies also to NVAO assessments like joint degrees, special features or assessment of
programmes and institutions at the Dutch Caribbean islands. However, in applications for initial
accreditation and distinctive features, institutions may withdraw their application in case of a
negative decision. Reports are then not published. In Flanders the reports on the institutional review
will be published together when the pilot is over.
Accessibility of reports at NVAO website
Reports are easily accessible on the NVAO website, once published. However, the publication of
reports is sometimes delayed for various months after the decision has been taken. In an extended
stratified sample of the NVAO decisions made in 2016, which the panel took in the beginning of
March 2017, it appeared that at least 10 percent of the reports and decisions made in 2016 was not
available. The review panel was told that the delay was caused by a temporary lack of personnel in
the communication unit, due to a position that became suddenly vacant and needed to be filled
following the typical timing of any transparent recruiting procedure.
Contents of reports
Compulsory elements in assessments reports include (among others) information on the institution
and context, procedural aspects, programme site visit, composition panel, description of findings,
considerations and judgements; recommendations and – if relevant – aspects related to a
conditional judgement. There is no specific format for organizing the content of the report.
Assessment agencies, which are commissioned by institutions, may choose their own format.
In the NVAO decision reports the name of assessed unit, the decision, detailed judgement standard
by standard, the legal basis of the decision, the result and the validity period, and a list of panel
members are mentioned, including a summary. This summary is normally taken literally from the
panel assessment report.
Conditions and recommendations
Under the 2014 NL framework, recommendations in reports leading to a positive accreditation are
published at the NVAO website in the reports. NVAO can also accentuate recommendations in the
decision report. In the 2016 NL-framework, for procedures resulting in conditional accreditation the
conditions are mentioned in the reports and at the website. However, following NL-governmentregulations, and as explained under Standard 2.3, if there are recommendations in reports leading to
a positive accreditation, these recommendations are not published at the NVAO website. They have
to be published at the institutions’ website. Thus a distinction is made between assessment and
enhancement. The enhancement part is not published at the NVAO website.
40/60

Heterogeneous character of reports
Assessment reports on institutional reviews (FL) and or institutional audits of internal quality
assurance (NL) are written by secretaries commissioned by NVAO. This also goes for assessment
reports on initial accreditations in NL and FL. In most cases, the assessment reports on programmes
in the Dutch Caribbean are written by NVAO staff. Reports on re-accreditation of programmes are
offered to NVAO by institutions. They are generally written by assessment agencies. All experts are
involved in producing the report, the secretary being responsible for most editing, although final
reports are the result of a consensus among all panel members. Secretaries receive a common NVAO
training, in addition to assessment agencies’ own trainings.
In spite of compulsory elements in all reports, but due to the fact that external assessments are
done by various agencies, assessment reports and summaries are very heterogeneous. The full
experts’ reports can largely differ in terms of length and style. The summary of the whole procedure
and outcomes can range from an executive summary of one page up to twenty pages.
Readability and readership
NVAO stresses the importance of the readability of reports and trains secretaries accordingly. Chairs
are trained as well, and tuned to writing reports. The executive summary is meant to be oriented
towards a general audience. Also conditions for a positive initial accreditation and the issues to be
addressed in a recovery period for re-accreditation should be clearly marked and taken over in
NVAO-decisions, when they deal with important aspects of the programme.
During the site visit NVAO staff told the review panel that an analysis on the needs of different
readerships has not yet taken place. Representatives from institutions informed the review panel
that reports are clear and had greatly improved over the past years. Also staff from within
institutions found reports readable and useful. However, students informed the panel that they do
not read reports, not even the summaries. Also employers do not know of reports. As regards the
general public, NVAO staff told the review panel that journalists appear to be able to read reports
well, and write about them accordingly when there is an element of news in it. The panel was
informed that in Flanders the NVAO-reports (or conclusions thereof) will feature at a new
government-led website with comprehensive information on programmes and institutions in
Flanders.
Analysis
Stakeholders have experienced significant positive progress since the 2012 review, due to concrete
measures implemented by NVAO: regular common training of secretaries and chairs, regular ‘tuning
meetings’ with assessment agencies, the introduction of a summary. Whilst reports can be
considered clear if gathered by an expert readership, readability is hindered by the heterogeneity of
style, which is alright for decision-makers, but less “user friendly” to a large public. The ‘summary’
appears to be open to wide interpretation and requires further professionalization in order to reach
a wider readership. Neither reports nor summaries appear to appeal to a readership among students
and employers. Possibly short abstracts with a maximum of 10 lines, like in scientific papers, are
more suitable to students, in particular on individual programmes. Reports on institutional reviews
could remain directed at expert institutional readership only, as these are generally speaking less
interesting to students, employers and the general public.
Panel recommendation
NVAO analyses the actual and potential readership of its reports and their needs, and develops new
means to reach a readership among students and employers.

41/60

Panel conclusion: substantially compliant
ESG 2.7 COMPLAINTS AND APPEALS
Standard:
Complaints and appeals processes should be clearly defined as part of the design of external
quality assurance processes and communicated to the institutions.
ENQA review 2012
Standard 2.7 was in the ESG-2005 part of Standard 3.7 ’External quality assurance criteria and
processes used by members’. The panel concluded ‘fully compliant’. No recommendations were
given.
Evidence
NVAO has appeal procedures in place. These are described in the SAR and in the assessment
frameworks. An institution can lodge an internal appeal on the basis of General Administrative Law
against any decision by NVAO. In Flanders an internal appeal may be lodged regarding an intended
decision by NVAO. Appeals are being dealt with by NVAO’s independent appeal committees. NVAO
has separate appeal committees for Flanders and The Netherlands, with an overlap of one member.
The Flemish committee deals with remarks, complaints and internal appeals, as outlined in the
Regulations on Governance principles approved by the Flemish Parliament. Both Dutch and Flemish
appeal committees are appointed by NVAO and consist of independent experts with legal and
educational backgrounds. After receipt of an appeal, the committee advises the board of NVAO,
which will take a final decision taking into account this advice. In case an internal appeal is rejected,
an institution can lodge an external appeal against this decision with (in The Netherlands) the
Administrative Jurisdiction Department of the Council of State [Afdeling Bestuursrechtspraak van de
Raad van State] and (in Flanders) with the Flemish Government and finally with the Council of State
of Belgium [Raad van State].
In the SAR a track record of appeal cases from 2012-2016 is mentioned. During the site visit an
update including 2017 was supplied. Over this period in Flanders three internal appeals were lodged.
Two of them were rejected; one appeal was withdrawn after an agreement with the complainants.
In The Netherlands two internal appeals were lodged. One was rejected, but in the following
external appeal accepted, because of an imperfection in the Dutch law (which is meanwhile
rectified). Another internal appeal regarding the grading in assessment scores by the panel is still
pending.
NVAO has no formal comprehensive complaint procedure as regards conduct or lack of
professionalism, but there is a general route of complaint handling via the national ombudsman.
Apart from the Regulations on Governance Principles mentioned above, for procedures (in FL) for
which NVAO is responsible stakeholders, such as panel members, staff or students, may report to
NVAO matters arising during the assessment that could affect the independence of tbe assessment.
The SAR doesn’t provide information on complaints, nor could information be provided during the
sitevisit. Panel chairs generally praised the professionalism of NVAO staff and NVAO staff told the
panel that complaints about conduct and lack of professionalism are rare. As mentioned before a
widely advertised ‘open hour’ in which stakeholders who were not invited for the site visit could
speak with the review panel during the site visit was cancelled for lack of complainants. However,
during the site visit one of the stakeholders informed the panel about a complaint, which, to his
dissatisfaction, was dealt with in an informal way only.

42/60

Analysis
The panel is satisfied with the procedures on appeals against decisions, but considers the lack of a
solid and formal comprehensive complaint procedure an omission, even if some elements of
complaint-handling are there and informal handling of complaints by NVAO normally suffices.
Panel recommendation
NVAO develops a complaint procedure and opens a section ‘Complaints and appeals’ at its website
with appropriate formats for complaints and appeals.
Panel conclusion: partially compliant

43/60

SUMMARY OF COMMENDATIONS
Commendation on Standard 2.1
NVAO is commended on the systematic coverage of the ESG Part 1 Standards in all frameworks that
are presently in use.
Commendation on Standard 3.3
NVAO is commended for the processes it has in place to ensure independence including the Code of
Conduct and Internal Integrity Code. The professionalism and integrity of NVAO staff was highlighted
by stakeholders during the review and apparent to panel members throughout the review process.
Apart from these two standard-linked commendations NVAO is congratulated with the superb
preparation of the review and the smooth running of the site visit.
OVERVIEW OF JUDGEMENTS AND RECOMMENDATIONS
F = Fully compliant; S = Substantially compliant, P = Partially compliant, N = Non-compliant
STANDARD
ESG
2.1
CONSIDERATION
OF
INTERNAL

JUDGEMENT

RECOMMENDATION

F

QUALITY

ASSURANCE

ESG

2.2

DESIGNING

F

METHODOLOGIES FIT FOR
PURPOSE

S
ESG 2.3 IMPLEMENTING
PROCESSES

ESG 2.4 PEER-REVIEW

NVAO remains sensitive regarding issues of implementation as
experienced by stakeholders and adopts a clearer terminology to
differentiate between substantively different approaches to follow
up of decisions.

F

EXPERTS

ESG 2.5 CRITERIA FOR

S

OUTCOMES

S
ESG 2.6 REPORTING
ESG 2.7 COMPLAINTS

P

AND APPEALS

S
3.1 ACTIVITIES, POLICY
AND

PROCESSES

FOR

QUALITY ASSURANCE

3.2 OFFICIAL STATUS
3.3 INDEPENDENCE
3.4 THEMATIC ANALYSIS

F
F
S

NVAO reflects on the grading system as presently applied and
considers making it as straightforward as necessary for the purpose
of yes/no/conditional accreditation.
NVAO analyses the actual and potential readership of its reports and
their needs, and develops new means to reach a readership among
students and employers.
NVAO develops a complaint procedure and opens a section
‘Complaints and appeals’ at its website with appropriate formats for
complaints and appeals.
NVAO prepares coherent development plans on the future shortterm and long-term development of accreditation processes in
Flanders and The Netherlands, on the basis of comprehensive
evaluations of the NL- and FL-pilots and taking into account
expectations about the quality assurance in higher education by all
relevant stakeholders.

NVAO reviews the approach to thematic analysis and in consultation
with stakeholders, identifies a common purpose and a means to
regularize processes and priorities in this area.

44/60

3.5 RESOURCES
3.6 INTERNAL QUALITY
ASSURANCE

F
S

AND

NVAO remains alert on loose loops in the plan-do-check-act-cycle of
the quality assurance system.

PROFESSIONAL CONDUCT

3.7 CYCLICAL EXTERNAL

F

REVIEW OF AGENCIES

In light of the documentary and oral evidence considered by it, the review panel is satisfied that, in
the performance of its functions, NVAO in substantial compliance with the ESG.

45/60

ANNEX 1: 2012 AND 2017 EXTERNAL REVIEWS : A COMPARATIVE OVERVIEW OF THE FINDINGS
F = Fully compliant; S = Substantially compliant, P = Partially compliant, N = Non-compliant
2012 review

2017 review

ENQA Criterion/ESG
Level of
compliance
ESG

2.1

CONSIDERATION

INTERNAL QUALITY
(FORMERLY ESG 2.1)

F

[NVAO] makes the link between the NVAO
standards and ESG Part 1 more explicit in the
assessment frameworks;

F

S, S

[NVAO] further strengthens the focus on quality
improvement and enhancement of the HEI’s;

F

OF

ASSURANCE

2.2
DESIGNING
METHODOLOGIES FIT FOR PURPOSE

Level of
compliance

Recommendation(s)

ESG

(FORMERLY ESG 2.2 AND ESG 2.4)

ESG 2.3 IMPLEMENTING PROCESSES
(FORMERLY ESG 2.6 AND 3.7)

F, F

S

ESG 2.4 PEER-REVIEW EXPERTS
(FORMERLY ESG 3.7)

F

F

ESG 2.5 CRITERIA FOR OUTCOMES
(FORMERLY ESG 2.3 AND 3.7)

P,S

- [NVAO] refines the descriptions of the aims and
objectives, ensures that they are prominent in
the frameworks and show how the various
elements of the frameworks contribute to the
aims and objectives;
- [NVAO] formulates explicit and public criteria
about how it reaches its accreditation decisions
in relation to the programme assessments in all
cases;
- [NVAO] establishes a clear procedure on how

46/60

S

Recommendation(s)

NVAO remains sensitive regarding issues of
implementation as experienced by stakeholders and
adopts a clearer terminology to differentiate
between substantively different approaches to
follow up of decisions.

NVAO reflects on the grading system as presently
applied and considers making it as straightforward as
necessary for the purpose of yes/no/conditional
accreditation.

to handle cases where the conclusions in the
assessment report are not accepted by NVAO
- [NVAO] strengthens the predictability of the
timeframe and efficiency of its decision-making
process;
- To improve the consistency of accreditation
judgments reached. Clustering of assessments
may be beneficiary to improve consistency in
assessments and decision making, as well as to
answer the stakeholders’ need for more systemwide analysis.

ESG 2.6 REPORTING (FORMERLY
ESG 2.5)

ESG 2.7 COMPLAINTS AND APPEALS
(FORMERLY ESG 2.7 AND 3.7
[GUIDELINE])

3.1

ACTIVITIES,

POLICY

S

[NVAO] clarifies
a) the purpose of every kind of report,
b) the readership, and
c) the needs of the various kinds of readers in
order to enhance the readability;

F

S

P

AND

PROCESSES
FOR
QUALITY
ASSURANCE (FORMERLY ESG 3.1,
3.3, AND 3.5)

S,F,F

S

3.2 OFFICIAL STATUS (FORMERLY
ESG 3.2)

F

F

3.3 INDEPENDENCE
ESG 3.6)

F

F

(FORMERLY

3.4 THEMATIC ANALYSIS (FORMERLY
ESG 2.8)

S

-[NVAO] gives a high priority to a)the
identification of the interested parties for
system-wide analyses and of their needs b) the

47/60

S

NVAO analyses the actual and potential readership of
its reports and their needs, and develops new means
to reach a readership among students and
employers.
NVAO develops a complaint procedure and opens a
section ‘Complaints and appeals’ at its website with
appropriate formats for complaints and appeals.
NVAO prepares coherent development plans on the
future short-term and long-term development of
accreditation processes in Flanders and The
Netherlands, on the basis of comprehensive
evaluations of the NL- and FL-pilots and taking into
account expectations about the quality assurance in
higher education by all relevant stakeholders.

NVAO reviews the approach to thematic analysis and
in consultation with stakeholders, identifies a
common purpose and a means to regularize

definition of a realistic schedule of system-wide
analyses and c)the production of system-wide
analyses corresponding to the needs which were
identified.
-To improve the consistency of accreditation
judgments reached. Clustering of assessments
may be beneficiary to improve consistency in
assessments and decision making, as well as to
answer the stakeholders’ need for more systemwide analysis.
3.5 RESOURCES (FORMERLY ESG
3.4)

processes and priorities in this area.

F

F

CONDUCT

F

S

3.7 CYCLICAL EXTERNAL REVIEW OF
AGENCIES (FORMERLY ESG 3.8
[GUIDELINE])

F

F

3.6 INTERNAL QUALITY ASSURANCE
AND

PROFESSIONAL

(FORMERLY ESG 3.8)

48/60

NVAO remains alert on loose loops in the plan-docheck-act-cycle of the quality assurance system.

ANNEX 2: PROGRAMME OF THE SITE VISIT
Schedule ENQA Review NVAO 29-31 March 2017

Timing

Topics/organisations

14.00-16.00

Review panel’s kick
off meeting and
preparations for Day
1
A pre-visit meeting
with the agency
contact person to
clarify
elements
related to the overall
system and context
Meeting with SAR
team NVAO
Break
Meeting with NVAO
Executive Board and
General Manager

16.00-16.30

Session 1
16.30-17.00
15 minutes
Session 2
17.15-18.15

15 minutes
Session 3
18.30-19.15
19.30

Timing
8.30-9.00
Session 4
9.00-10.00

15 minutes
Session 5
10.15-11.00

15 minutes
Session 6
11.15-12.00

Break
Meeting
with
General Board of
NVAO
Dinner for panel

Topics/organisations

29 March 2017
Persons for interview

Issues to be
discussed

Panel and ENQA-coordinator
NVAO Policy Officer as contact person, Mr.
Thomas de Bruijn

Standards
3.1

Mr. Thomas de Bruijn/Mr. Henri Ponds, policy
officers NVAO

SAR, Standards
2.6, 2.7, 3.6

NVAO Board:
- Mr. Anne Flierman (Chair)
- Mrs. Ann Verreth (Vice Chair)
- Mr. Paul Zevenbergen (Member)
- Mr. Marc Luwel (Member)
- Mr. René Hageman, Head of department NL
- Mr. Axel Aerden, General Manager FL.

Standards 2.3,
3.1, 3.2, 3.3, 3.5

- Mr. Paul van Roon, Mrs. Lieteke van Vucht
Tijssen, Mrs. Mia de Schamphelaere, Mr. Bart
Maes

SAR, Standards
2.4, 2.5, 2.6,
3.1, 3.3

30 March 2017
Persons for interview

Review Panel preparation Day 2
Meeting with NVAO Policy Officers:
Staff
Mrs. Lisette Winsemius/ Mrs. Lieve Desplenter/
(Policy advisors and Mr. Pieter Caris/ Mr. Frank Wamelink/ Mr. Dagmar
Management Team)
Provijn/ Mrs. Nancy van San/ Mr. Henri Ponds
Break
Meeting
with - Mr. Feite Hofman, director of higher education,
representatives
of Dutch Ministry of Education
the Dutch Ministry of - Mr. Dirk Haen, policy advisor, Dutch Ministry of
Education,
Culture Education
and Science (OCW) - Mr. Gerben Kerkhof, Dutch Ministry of Education
and Inspectorate of - Mr. Bert Jaap van Oel, Inspectorate of Education,
education
The Netherlands
Break
Meeting
with - Mr. Koen Pelleriaux, General Director, Flemish
representatives
of Ministry of Education
the Flemish Ministry - Mrs. Sophie Landuyt, policy advisor, Flemish
of Education, Culture Ministry of Education
and Science

49/60

2.4,

Issues to be
discussed
Standards 2.1,
2.4, 2.5, 2.6,
3.1, 3.4, 3.6

Standards 2.5,
2.6, 3.1, 3.2,
3.3, 3.4, 3.5,

Standards 3.1,
3.2, 3.3, 3.4, 3.5

12.00-13.00
Session 7
13.00-13.45

Lunch panel
Meeting with Dutch
Universities
of
Applied Sciences and
Dutch
universities
(Vereniging
Hogescholen/ VSNU)

15 minutes
Session 8
14.00-14.30

Break
Meeting with Dutch
private institutions
for higher education
(NRTO)
Meeting with Flemish
professional
education
and
universities
(VLIR/VLOHRA)

Session 9
14.30-15.15

15 minutes
Session 10
15.30-16.15

15 minutes
Session 10a
16.30-17.30

Break
Meeting with Dutch
and
Flemish
assessment agencies
NQA,
Hobéon,
AeQui,
Certiked,
QANU, and VLUHRKZ
Break
Meeting with panel
members
and
secretaries

Delegation of VSNU
- Mr. Karl Dittrich, Chair of the VSNU
- Mrs. Katinka Eikelenboom, domain coordinator
education, VSNU
Delegation of VHS:
- Mr. Leendert Klaassen, Board member of VH,
Chair of the Board of Stenden University of
Applied Sciences, Leeuwarden
- Mr. Roeland Smits, policy advisor Vereniging van
Hogescholen.

Standards 2.1,
2.3, 2.5, 3.1, 3.4

- Mr. Eric Verduyn, director of education
Hogeschool NCOI, Board member of NRTO,
responsible for higher education

SAR, Standards
2.5, 2.6, 3.1

Delegation of VLIR:
- Prof. dr. Jan Danckaert, vicerector education
VUB, member of the Working Group Education of
VLIR, and member of the Klankbordoverleg
- Prof. dr. Didier Pollefeyt, vicerector education KU
Leuven, Chair of the Working Group Education of
VLIR, member of the evaluation group for the new
Flemish system
- Prof. dr. Rosette S’Jegers, secretary-general of
VLIR
Delegation of VLHORA:
- Mrs. Hilde Sels, QA-manager Thomas More
Hogeschool
- Mr. Ignace Van Dingenen, QA-manager,
Erasmus-hogeschool Brussels

SAR, Standards
2.5, 3.1, 3.2

- Mr. Willem van Raaijen, Hobéon
- Mr. Paul Esveld, Certiked
- Mr. René Kloosterman, AeQui
- Mr. Sietze Looijenga, QANU
- Mr. Paul Thijssen, NQA
- Mrs. Marleen Bronders, VLUHR-KZ

Standards
2.4, 2.5,
2.7, 3.1

2.3,
2.6,

-Dr.mr. Eric van de Luijtgaarden, lector European
Law, Zuyd Hogeschool (panel member for agencies
and NVAO)
- Drs. Raoul van Aalst, auditor, AeQui (panel chair
in many assessments)
- Prof. dr. Ton van Haaften, professor of Dutch
Discourse Studies, Leiden University (panel chair
and member in various institutional audits NL)
- Prof.dr. Frank Witlox, professor of Economic
Geography, Ghent University (panel member in
Flemish assessments by VLUHR-KZ and also in
Dutch research master’s assessments)
- Prof. dr. Joost Lowyck, em. professor of
Educational technology, KU Leuven (panel
member in Dutch and Flemish assessments)

Standards
2.2, 2.4,
2.6, 2.7

2.1,
2.5,

50/60

17.30-18.00

18.00- 19.00
19.00

Timing
8.30-9.00
Session 11
9.00-9.45

15 minutes
Session 12
10.00-10.45

15 minutes
Session 13
11.00-11.45

11.45- 12.30
12.30-13.00
13.00-13.30

13.30-15.30
15.30-16.00

- Prof.dr. Paul de Knop, professor of Sociology of
Sport, Vrije Universiteit Brussel (panel member
various assessments FL/NL)
- Mrs. Drs. Johanneke Braaksma, Secretary
for NVAO, NQA, VLUHR (until 17.15)
- Mrs. Dr. Jetje de Groof, Secretary and process
coordinator for QANU, NVAO, VLHOVLHORA, VLIR
Mr. Thomas de Bruijn and Mrs. Irma Franssen

Meeting with NVAOcontactperson and
NVAO-specialist
Dutch Caribbean
Review panel meeting on preliminary findings
Panel dinner

Topics/organisations

31 March 2017
Persons for interview

Review panel preparation Day 3
Meeting with Dutch - Mr. Remco Barendregt, mr. Robin de Bles, ISO
and Flemish Student - Mr. Ben Windey, Mr. Lorenzo Ego, VVS
organisations: LSVB,
ISO and VVS
Break
Meeting
with - Mrs. Annick van Kerckhove, lic., lecturer
teachers from Dutch Communication, Artevelde Hogeschool, Antwerp
and Flemish higher - Mr. Kevin Picalausa, lic., lecturer professional
education
bachelor ICT, Odisee Hogeschool, Ghent
institutions
- Mr. Jan Loisen, lic. lecturer Communication
Sciences, Vrije Universiteit Brussel
- Mr. Rik Vanderhaege, lic. lecturer Communication.
Thomas More Hogeschool, Antwerp
- Prof. dr. Wieger Bakker, professor of Public
Governance and Management, Utrecht University
- Dr. Gerard van der Bunt, associate professor of
Sociology, Vrije Universiteit Amsterdam
- Drs. Margriet Plasman, educational advisor and
lecturer HRM and Applied Psychology, Saxion
Hogeschool
Break
Meeting
with - Mr. Louis Spaninks, coordinator Human Capital
representative
of Agenda ICT (HCA ICT)
employers
- Mr. Frans Berntsen, former general manager
Slachtofferhulp Nederland, member of workfield
advisory committee
- Mrs. Annette Geirnaert, former general manager
of Kwinta, Flemish Centre for Quality Management
- Mr. Patrick Audenaerde, Prevention Advisor,
Ghent Harbour
Lunch for panel
Meeting of panel to decide on issues to be clarified
Meeting with NVAO Board of NVAO
board
to
clarify
pending issues
Concluding meeting of panel
Final
de-briefing NVAO-Board, staff
meeting with Board

51/60

Dutch
Caribbean and
Joint Degrees

Issue to be
discussed
Standards 2.1,
2.4, 2.6, 3.1,
3.4

Standards 2.2,
2.5, 3.1, 3.4

SAR,
Standards 2.2,
2.4, 3.1

Standards 2.3,
2.7, 3.1, 3.4

n.a.

16.00

members and staff to
inform
about
preliminary findings
End of site visit

52/60

ANNEX 3: TERMS OF REFERENCE OF THE REVIEW
1. Background and Context
NVAO is the independent and binational accreditation organisation set up by the Dutch and Flemish
Parliaments on 1 February 2005, whose primary goal it is to provide an expert and objective
judgement of the quality of higher education in Flanders and the Netherlands. Its operations are
supervised by the Committee of Ministers of Education of both the Netherlands and Flanders.
NVAO’s main activities are related to its legal tasks and include drafting and maintenance of the
frameworks for accreditation, deciding on applications for accreditation for existing programmes,
coordinating assessments and deciding on initial accreditation and institutional audits or reviews in
the Netherlands and Flanders. In addition, NVAO can be asked to advise the ministries of education
in both countries regarding specific topics or carry out assessments related to quality assurance.
NVAO operates in two different systems of higher education which have both implemented the
Bologna reforms on the basis of shared principles for quality assurance. Compliance to the European
Standards and Guidelines has been a cornerstone for the Dutch and Flemish systems of quality
assurance since their inception. The systems of quality assurance in both the Netherlands and
Flanders enter a third phase of their development, which is characterized by reduction of
administrative burden on institutions and increasing their autonomy with regard to quality
assurance, while maintaining guarantees for the quality of study programmes. NVAO further
maintains a strong international presence in European and global networks for external quality
assurance, which enables NVAO to channel back new developments in higher education to
institutions in both countries.
NVAO has been a full member of ENQA since 2003. With the review of 2017 NVAO aims to renew its
ENQA membership. NVAO has been registered on EQAR since 2008 and aims at renewing its
registration in September 2017.
2. Purpose and Scope of the Evaluation
This review will evaluate the way in which and to what extent NVAO fulfils the Standards and
Guidelines for Quality Assurance in the European Higher Education Area (ESG). Consequently, the
review will provide information to the ENQA Board to aid its consideration of whether membership
of NVAO should be reconfirmed and to EQAR to support NVAO application to the register.
The review panel is not expected, however, to make any judgements as regards granting
membership.
2.1 Activities of NVAO within the scope of the ESG
In order for NVAO to apply for ENQA membership and for registration in EQAR, this review will
analyse all activities of NVAO that are within the scope of the ESG, i.e. reviews, audits, evaluations or
accreditation of higher education institutions or programmes that relate to teaching and learning
(and their relevant links to research and innovation). This is regardless whether these activities are
carried out within or outside the EHEA, and whether they are obligatory or voluntary.
The following activities of NVAO have to be addressed in the external review (self-assessment report
and the external review report):

53/60

1. Programme accreditation in Flanders, the Netherlands and the so called “Caribbean
Netherlands”;
2. Initial programme accreditation in Flanders and the Netherlands;
3. Accreditation of joint programmes;
4. Institutional audits/reviews in the Netherlands and Flanders;
5. Programme assessment in Curaçao, Aruba and St. Maarten;
6. Institutional assessment in Curaçao, Aruba and St. Maarten
7. Assessments of distinctive (quality) features for programmes and institutions in the
Netherlands and Flanders.
In addition, while the following activities are separate parts of external quality assurance activities,
they may be relevant in relation to a number of the ESG standards as follows, and should thus be
addressed appropriately in the self-assessment report and the external review report:
8. Approval of/advising on panels for programme assessments - ESG 2.4;
9. Production of thematic analyses or evaluations on the basis of assessments - ESG 3.4;
10. Drafting and maintaining frameworks for accreditation for programmes and institutions in
higher education - ESG 2.2 and ESG 2.3.
NVAO carries out evaluations of various features of programmes, in most cases as part of
assessments for (initial) accreditation. These result in either advise to the Dutch of Flemish Minister
of Education or an independent decision by NVAO. The ENQA review panel will consider on the basis
of NVAO’s self-evaluation report whether these evaluations are part of the regular external quality
assurance activities of NVAO or a separate evaluation of study programmes in relation to learning
and teaching in higher education, and thus whether they are within the scope of the ESG. The
evaluations include:
1. Advising the Minister of Education on applications for combining two or more existing
programmes into broad programmes (NL);
2. Advising the Minster of Education on applications for extending (or reducing in Flanders) the
formal duration of programmes, including research masters of 120 EC (advise in Flanders,
initial accreditation in the Netherlands);
3. Deciding on the changes of names of programmes and degrees (NL);
4. Advising the Minister of Education on allowing additional admission criteria for programmes
related to specific educational concepts (NL).
Furthermore, the self-evaluation report and external review report should also address (1) NVAO’s
requirements for the recognition of other agencies and (2) how NVAO ensures that the decisions
taken on the basis of reviews carried out by other agencies are in line with the ESG, especially in case
the agency is not registered on EQAR.
The following issues that were flagged when NVAO‘s registration in EQAR was last renewed and
should be addressed in NVAO’s self-evaluation report and in the external review report:
ESG 2.5 – Criteria for outcomes [ESG 2005: standard 2.3]
It should be addressed whether NVAO has clarified (within its own operational documents and
without questioning the holistic nature of its judgements) the criteria and decision-making process
used in making decisions on the accreditation of existing programmes in the Netherlands, including
the way in which it ensures consistency of its decisions based on reviews undertaken by different
agencies.

54/60

ESG 2.6 – Reporting [ESG 2005: standard 2.5]
Issues related to the readability of its reports for its defined target audience should be addressed.
3. The Review Process
The process is designed in the light of the Guidelines for ENQA Agency Reviews and in line with the
requirements of the EQAR Procedures for Applications.
The evaluation procedure consists of the following steps:











Formulation of the Terms of Reference and protocol for the review;
Nomination and appointment of the review panel;
Self-assessment by NVAO including the preparation of a self-assessment report;
A site visit by the review panel to NVAO;
Preparation of the final evaluation report by the review panel;
Check of the report for factual accuracy and grave misunderstandings by the agency;
Completion of the final evaluation report by the review panel;
Scrutiny of the final evaluation report by the ENQA Review Committee;
Analysis of the scrutiny by the ENQA Board and their decision regarding ENQA membership;
Follow-up of the panel’s and/or ENQA Board’s recommendations by the agency, including a
voluntary follow-up visit.

3.1 Nomination and appointment of the review team members
The review panel consists of four members: one or two quality assurance experts, an academic
employed by a higher education institution, student member, and eventually a labour market
representative (if requested). One of the members will serve as the chair of the review panel, and
another member as a review secretary. Two of the reviewers are nominated by the ENQA Board on
the basis of proposals submitted to ENQA by the member national agencies. The third external
reviewer is drawn from a nomination provided by the European University Association (EUA) or the
European Association of Institutions in Higher Education (EURASHE). The nomination of the student
member comes from the European Students’ Union (ESU).
In addition to the four members, the panel will be supported by the ENQA Secretariat review
coordinator who will monitor the integrity of the process and ensure that ENQA expectations are
met throughout the process. The ENQA staff member will not be the Secretary of the review and will
not participate in the discussions during the site visit interviews.
Current members of the ENQA Board are not eligible to serve as reviewers.
ENQA will provide NVAO with the list of suggested experts with their respective curriculum vitae to
establish that there are no known conflicts of interest. The experts will have to sign a non-conflict of
interest statement as regards NVAO review.
3.2 Self-assessment by NVAO, including the preparation of a self-assessment report
NVAO is responsible for the execution and organisation of its own self-assessment process and shall
take into account the following guidance:


Self-assessment is organised as a project with a clearly defined schedule and includes all
relevant internal and external stakeholders;
55/60








The self-assessment report is broken down by the topics of the evaluation and is expected to
contain, among others: a brief description of the national HE and QA system; background
description of the current situation of the Agency; an analysis and appraisal of the current
situation; proposals for improvement and measures already planned; a SWOT analysis; each
criterion (ESG part II and III) addressed individually. All agency’s QA activities (whether
within their national jurisdiction or outside of it, and whether obligatory or voluntary) will be
described and their compliance with the ESG analysed.
The report is well-structured, concise and comprehensively prepared. It clearly
demonstrates the extent to which NVAO fulfils its tasks of external quality assurance and
meets the ESG and thus the requirements of ENQA membership.
The self-assessment report is submitted to the ENQA Secretariat who has 4 weeks to prescrutinise it before forwarding the report to the panel of experts. The purpose of the prescrutiny is to ensure that the self-assessment report is satisfactory for the consideration of
the panel. The Secretariat will not judge the content of information itself but whether the
necessary information, as stated in the Guidelines for ENQA Agency Reviews, is present. For
the second and subsequent reviews, the agency is expected to enlist the recommendations
provided in the previous review and to outline actions taken to meet these
recommendations. In case the self-assessment report does not contain the necessary
information and fails to respect the requested form and content, the ENQA Secretariat
reserves the right to reject the report and ask for a revised version within 4 weeks. In such
cases, an additional fee of 1000 € will be charged to the agency.
The report is submitted to the review panel a minimum of six weeks prior to the site visit.

3.3 A Site Visit by the Review Panel
NVAO will draw up a draft proposal of the schedule for the site visit to be submitted to the review
panel at least two months before the planned dates of the visit. The schedule includes an indicative
timetable of the meetings and other exercises to be undertaken by the review panel during the site
visit, the duration of which is 2,5 days. The approved schedule shall be given to NVAO at least one
month before the site visit, in order to properly organise the requested interviews.
The review panel will be assisted by NVAO in arriving in Hague, Netherlands.
The site visit will close with an oral presentation and discussion of the major issues of the evaluation
between the review panel and NVAO.
3.4 Preparation and completion of the final evaluation report
On the basis of the review panel’s findings, the review secretary will draft the report in consultation
with the review panel. The report will take into account the purpose and scope of the evaluation as
defined under articles 2 and 2.1. It will also provide a clear rationale for its findings with regards to
each ESG. A draft will be first submitted to the ENQA review coordinator who will check the report
for consistency, clarity and language and it will be then submitted to NVAO for comment on factual
accuracy. If NVAO chooses to provide a statement in reference to the draft report it will be
submitted to the chair of the review panel within two weeks after the receipt of the draft report.
Thereafter the review panel will take into account the statement by NVAO, finalise the document
and submit it to NVAO and ENQA.
The report is to be finalised within three months of the site visit and will not exceed 40 pages in
length.

56/60

When preparing the report, the review panel should also bear in mind the EQAR Policy on the Use
and Interpretation of the ESG, so as to ensure that the report will contain sufficient information for
the Register Committee for application to EQAR.
NVAO is also requested to provide a letter addressed to the ENQA Board outlining its motivation
applying for membership and the ways in which NVAO expects to contribute to the work and
objectives of ENQA during its membership. This letter will be discussed along with the final
evaluation report.
4. Follow-up Process and Publication of the Report
NVAO will consider the expert panel’s report and will publish it on its website once the ENQA Board
has made its decision. The report will also be published on the ENQA website, regardless of the
review outcome and decision by the ENQA Board. NVAO commits to preparing a follow-up plan in
which it addresses the recommendations of the review panel and to submitting a follow-up report to
the ENQA Board. The follow-up report will be published on the ENQA website, in addition to the full
review report and the Board’s decision.
The follow-up report will be complemented by a small-scale visit to the agency performed by two
members of the original panel (whenever possible). This visit will be used to discuss issues, based on
the ESG, considered as of particular importance or challenge by NVAO. Its purpose is entirely
developmental and has no impact on the judgement of membership and/or compliance of the
agency with the ESG. Should the agency not wish to take advantage of this opportunity, it may opt
out by informing the ENQA Review Coordinator about this.
5. Use of the report
ENQA shall retain ownership of the report. The intellectual property of all works created by the
expert panel in connection with the review contract, including specifically any written reports, shall
be vested in ENQA.
The review report is used by the Board of ENQA for the purpose of reaching a conclusion on whether
NVAO has met the ESG and can be thus admitted/reconfirmed as a member of ENQA. The report will
also be used for registration on EQAR, and is designed so as to serve these two purposes. However,
the review report is to be considered final only after being approved by the ENQA Board. Once
submitted to NVAO and ENQA and until it is approved by the Board the report may not be used or
relied upon by NVAO, the panel and any third party and may not be disclosed without the prior
written consent of ENQA. NVAO may use the report at its discretion only after the Board has
approved of the report. The approval of the report is independent of the decision on membership.
The Chair of the panel shall remain available to respond to questions of clarification or further
information from the EQAR Register Committee provided that the ENQA Secretariat is copied in all
such requests.
6. Budget
NVAO shall pay the following review related fees:
Fee of the Chair
Fee of the Secretary
Fee of the 2 other panel members
Fee of 2 panel members for follow-up visit
57/60

4,500 EUR
4,500 EUR
4,000 EUR (2,000 EUR each)
1,000 EUR (500 EUR each)

Administrative overhead for ENQA Secretariat
Experts Training fund
Approximate travel and subsistence expenses
Travel and subsistence expenses follow-up visit

7,000 EUR
1,400 EUR
6,000 EUR
1,600 EUR

This gives a total indicative cost of 30,000.00 EUR VAT excl. for a review team of 4 members. In the
case that the allowance for travel and subsistence expenses is exceeded, NVAO will cover any
additional costs after the completion of the review. However, the ENQA Secretariat will endeavour
to keep the travel and subsistence expenses in the limits of the planned budget, and will refund the
difference to NVAO if the travel and subsistence expenses go under budget.
The fee of the follow-up visit is included in the overall cost of the review and will not be reimbursed
in case the agency does not wish to benefit from it.
In the event of a second site visit required by the Board and aiming at completing the assessment of
compliance, and should the agency accept a second visit, an additional fee of 500 EUR per expert, as
well as travel and subsistence costs are recoverable from the agency.
7. Indicative Schedule of the Review
Agreement on terms of reference
Appointment of review panel members
Self-assessment completed
Pre-screening of SAR by ENQA coordinator
Preparation of site visit schedule and indicative timetable
Briefing of review panel members
Review panel site visit
Draft of evaluation report and submitting it to ENQA
coordinator for pre-screening
Draft of evaluation report to NVAO
Statement of NVAO to review panel if necessary
Submission of final report to ENQA
Consideration of the report by ENQA Board and response
to NVAO
Publication of the report

58/60

By September 2016
November/December 2016
By 1st November 2016
November 2016
January/February 2017
February/March 2017
March/early April 2017
May/early June 2017
June 2017
June/July 2017
By Mid-August 2017
September 2017
September 2017

ANNEX 4: GLOSSARY
ENQA
ESG

European Association for Quality Assurance in Higher Education
Standards and Guidelines for Quality Assurance in the European Higher Education Area,
2015
HE
higher education
HEI
higher education institution
QA
quality assurance
SAR
self-assessment report
NL
The Netherlands
FL
Flanders
C/A/StM Curaçao, Aruba, St. Maarten (Dutch Caribbean)
GB
General Board NVAO
EB
Executive Board NVAO
CV
curriculum vitae
HRM
human resource management
n(y)a
not (yet) applicable

59/60

ANNEX 5. DOCUMENTS TO SUPPORT THE REVIEW
DOCUMENTS PROVIDED BY NVAO
 NVAO strategy documents 2013-2016 and 2017-2020;
 Assessment frameworks as valid from 1 Jan 2017;
 An elaboration on p55 and 56 of the SAR as regards the concordance between NVAO’s
assessment frameworks and ESG part 1;
 An overview of the approx. 500 decisions made by NVAO in 2016, on programmes, including
Dutch Caribbean programmes and Joint Degrees;
 An overview of the assessment agencies carrying out reviews of institutions and
programmes;
 An overview of the correspondence between NVAO and EQAR in 2016 and 2017 on
substantive changes in the accreditation routine in Flanders up till 27 March 2017;
 An overview of new developments as regards developments in accreditation in The
Netherlands, and NVAO organisational structure;
 An overview of subsidies and fees (2014-2016);
 NVAO quality policy paper (Jan. 2017);
 All documents relating to internal quality assurance
 Excel-sheets with work plans for NVAO-NL and NVAO-FL;
 A selection of reports on Joint Degrees;
 Overview on the context of higher education and NVAO in The Netherlands and Flanders;
 Update on developments in Dutch higher education and in the structure of NVAO, including
list of members of the General Board (March 2017);
 An overview of types of decisions in Flanders (2015-2018);
 An overview of complaints and appeals NL and FL (2012-2017).
OTHER SOURCES USED BY THE REVIEW PANEL
 European Approach for Quality Assurance of Joint Programmes (October 2014, approved by
EHEA ministers in May 2015);
 ENQA review QANU 2016;
 ENQA review VLUHR-KZ 2014 and follow up.

60/60

THIS REPORT presents findings of the ENQA Agency Review of the Accreditation Organisation of the Netherlands and Flanders (NVAO),
undertaken in 2017.

2017 ENQA AGENCY REVIEW