Evaluation of EFSF and
ESM financial assistance:
Technical Appendices
15 June 2017

Contents

Summary ��������������������������������������������������������������������������������������������������������������������������� 3
A. Terms of Reference ������������������������������������������������������������������������������������������������������ 4
B. Methodology ��������������������������������������������������������������������������������������������������������������� 17
1.
Organising the evaluation ��������������������������������������������������������������������������� 17
2.
Evaluation strategy �������������������������������������������������������������������������������������� 22
3.
Components of the evaluation ������������������������������������������������������������������� 24
3.1 Collection of evaluation data ������������������������������������������������������������� 24
3.2 Desk studies ���������������������������������������������������������������������������������������� 26
3.3 Analysis strategy for interviews and surveys ���������������������������������� 35
4.
Conclusion ���������������������������������������������������������������������������������������������������� 38
Sources ��������������������������������������������������������������������������������������������������������� 39
C. Intervention logic ������������������������������������������������������������������������������������������������������� 40
D. Vulnerability analysis ������������������������������������������������������������������������������������������������ 48
E. Timing of request ������������������������������������������������������������������������������������������������������� 52
F. Choice of instrument ������������������������������������������������������������������������������������������������� 55
G. Crisis timeline ������������������������������������������������������������������������������������������������������������ 56

Summary

These technical appendices were compiled to provide further background to the first
ESM evaluation. Their purpose is to enhance transparency on the technical aspects of
the organisation of the evaluation exercise, and the various strands of research. The evaluation was designed to ensure maximum impartiality, within the available organisational set-up,
and methodological rigour. By describing the process and the various strands of analysis the
appendices support the credibility of the evaluation.
The terms of reference have been made public in their original form in Appendix A, which was
approved by the ESM Board of Governors on 24 October 2016. Appendix B describes the topology and process of the evaluation. The design and conduct of the assessment was guided by
a theoretical framework that resulted from a facilitated brainstorming process, summarised in
Appendix C. Appendix D provides technical background to the assessment of the evolution of
vulnerabilities in the relevant economies, which the ‘relevance’ part of the evaluation largely
drew on.
Appendices E and F contain background material for the analysis of the timing of programme
requests in section 3.1 of the evaluation report and the choice of instrument in section 3.2. In
Appendix G, the key crisis-related events from the perspective of the EFSF/ESM are summarised
in a timeline.
These appendices were compiled by the ESM evaluation team and are available online only. The
Independent Evaluator's report is available online and in print.

3

A. Terms of Reference

The European Stability Mechanism will perform an evaluation of the key activities as is customary practice in international institutions. These Terms of Reference are a statement of the background, objectives, and purpose of the first ESM evaluation exercise; the individual duties and
responsibilities of the actors in the project; and timelines for the main deliverables.
The purpose of the evaluation is to assess the relevance, effectiveness, and efficiency1
of EFSF and ESM financial assistance in safeguarding financial stability of the euro area
and its Member States.
Background
Context
The European Financial Stability Facility (EFSF) and European Stability Mechanism (ESM) were
established to provide financial assistance programmes for euro area Member States in response
to the global financial crisis and the European sovereign debt crisis. The two institutions provided financial assistance to five countries in six separate programmes, either alone or as part
of a broader international effort. They have disbursed in total more than €260 billion since 2011.
More specifically, the financial assistance programmes are:2

Institution

Country

Formal
request

Financial assistance
facility agreement
entered into

Completion

Partners

EFSF

Ireland

21 Nov 2010

22 Dec 2010

8 Dec 2013

EFSM, IMF and
the UK, Sweden
and Denmark
as lenders; EC
and ECB

EFSF

Portugal

7 Apr 2011

27 May 2011

18 May 2014

EFSM, IMF
as lenders; EC
and ECB

EFSF

Greece

8 Feb 2012

15 Mar 2012

Expired
30 Jun 2015

EC, ECB, IMF

ESM

Spain

25 Jun 2012

24 Jul 2012

31 Dec 2013

EC, ECB, and IMF
(not financing)

ESM

Cyprus

25 Jun 2012

8 May 2013

31 Mar 2016

EC, ECB, IMF

ESM

Greece

8 Jul 2015

19 Aug 2015

Ongoing

EC, ECB, IMF

Source: ESM

1
2

4

See definitions in the Annex 1.
The Facility agreements can be found at www.esm.europa.eu/financial-assistance.
(Updated 8 June 2017)

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Four countries completed their programmes with a ‘clean exit’ and no follow-up arrangement.
The second Greek programme ended after some prolongation, and a subsequent ESM Greek
programme is on-going. Neither EFSF nor ESM activities were evaluated during the first years
of operation. Hereafter references in this document to the ESM also apply to the EFSF, where
relevant, even when not specified.
The organisational setup in which the programmes are run involves various institutions, which will
have to be taken into account in the design and scoping of the EFSF and ESM related evaluation:
• relevant substantive policy discussions take place in the Eurogroup and Eurogroup Working
Group matching closely the EFSF and ESM governing bodies;
• programme conditionality is negotiated by the European Commission in liaison with the
European Central Bank;
• the programmes have been conducted as joint programmes with the IMF whenever possible;
and
• post-programme monitoring is done by the European Commission and in parallel the Early
Warning System is conducted by the ESM.
These programmes have been political and economic cornerstones of European crisis management and have, therefore, rightly been subjected to substantive public scrutiny and debate. This
justifies a thorough evaluation to collect lessons for the future.
The Chairman of the Board of Governors asked the ESM in early 2016 to develop a mandate for
programme evaluations. To prepare the ESM performed a desk study on the relevant evaluations
and policy reviews conducted by ESM peer institutions to gain a better understanding of how an
ESM evaluation could add value.
A proposal for the potential scope of the exercise was discussed twice with the ESM Board of
Directors. The Board of Governors approved the exercise on 16 June 2016.
Purpose and scope
Mandate
The mandate covers both EFSF and ESM financial assistance programmes. The mandate is primarily to look into past programme activities to draw lessons for future activities. These lessons
should help to promote the relevance, effectiveness, and efficiency of the programmes and further improve decision-making processes.3
The evaluation therefore strengthens institutional learning and memory. The primary audience
for the report consists of the EFSF/ESM members represented in the governing bodies and ESM
management. As a secondary objective, the evaluation aims to provide comfort to external stakeholders, including the general public, by providing transparency and validated information on the
organisations’ activities.

3

Given the focus on concluded programmes, the current, ongoing third Greek programme will not
be part of the scope.
5

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Scope and limitations
The Board of Directors and the Board of Governors have discussed and provided guidance on
the scope of the ESM evaluation exercise. The evaluation period covers the negotiations for each
programme and runs through the post-programme period up to end-June 2016. The main focus
will be on the relevance, effectiveness and efficiency of EFSF/ESM programme activities, concentrating on design, financing and cooperation issues.
The exercise will cover activities relevant to EFSF and ESM programmes for Ireland, Spain,
Cyprus, and Portugal. The second Greek programme will be addressed where specific events
offer relevant insight, but that programme will not be evaluated overall.4
This initial exercise is an outcome evaluation5 taking into account ESM procedural aspects in the
overall institutional framework.6 It can also be characterised as a formative evaluation7 given its
focus on learning.
The report will deliver cross-cutting analysis of the programmes rather than country-by-country
conclusions. This should ensure greater clarity in the resulting policy lessons. The scope of the
evaluation is limited in that it will not evaluate the political aspects of the Eurogroup negotiations
or certain aspects of programme design – e.g. conditionality – that are the specific responsibility
of other institutions. Such issues outside of the evaluation’s formal scope may nevertheless be
addressed if they serve as explanatory factors.
Evaluation questions
The discussions with the governing bodies yielded the following set of more specific evaluation
questions, which can be broadly assigned to three programme phases (negotiation, programme
execution, and post-programme monitoring) and to standard evaluation criteria.

4

5
6
7
6

After discussion, the Board of Governors advised the ESM to cover all programme countries as
appropriate. To avoid compromising current activities under the third Greek programme, the
intention is to evaluate relevant aspects of the second Greek programme related to financing
effectiveness and efficiency, for example in bank recapitalisation, contingency and disbursement
planning, and programme governance, until end-December 2014, the initial expiry date of the
programme. The entire second Greek programme and the third programme could be evaluated
after completion in August 2018.
See definition in Annex 1.
In this exercise, because of the relatively recent completion of the programmes, the ESM is not
expected to prepare an impact evaluation (see definition in Annex 1).
See definition in Annex 1.

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Negotiation

Relevance

Did the EFSF/ESM comply with its mandate to safeguard the financial stability of the
euro area and its Member States by deploying its programmes? How did their roles
evolve over time?
How were financing needs
defined?
Was the need for buffers
considered appropriately in
the financing envelope?

Effectiveness

Programme execution

Post-programme
monitoring

Period of:

Were the EFSF/ESM
financial assistance
activities consistent with
the prime objective?

Did ESM instruments
and agreed policies
work, and under what
type of country-specific
Was the granted assistance circumstances to ensure
lasting effects?
instrument appropriate
and the programme period
adequate?

How can the ESM best support effective programme implementation, including in the
post-programme period?
Were members treated even-handedly in light of the size of assistance relative to each
country’s financing needs, lending terms, or disbursement practices?
What were the major
factors affecting successful
implementation?
Did programmes target
effectively the financing
needs of a country and
account for the specific
requirements of bank
recapitalisation?
Was the financing
planning and disbursement
strategy conducive to
effective programme
implementation?
Were the financing terms
sufficiently flexible in view
of uncertainty?
Were programme
contingencies assessed
appropriately?

Does the lending
framework set out an
appropriate creditor
position, including on
proportionality and the
ability to monitor and assert
rights?
How has the need
for possible follow-up
arrangements been
assessed?
How did borrowing costs
and maturity structure
support programme
achievement and
programme sustainability?
Did benefits prove
sustainable?
Were there unintended
effects?

If the original
implementation strategy
was modified, were the
compensating measures
effective?
Efficiency

Did requests for financial
assistance programmes
and the assessment by
institutions comply with the
requirements of the ESM
Treaty and Guidelines?
Did the information
provided ensure the
efficient conduct of
assistance?

Collaboration

What measures were taken
to ensure that resources
were used efficiently?
Were resources adequate?
Have activities (e.g. fund
raising and disbursements)
taken place in a timely
manner?

What was the nature of the relationships with the relevant local, European, or
international authorities and bodies, and what could be improved?
How effective was the
programme governance
structure?

7

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Roles and responsibilities
Key actors
The project actors are the high level evaluator (evaluator), ESM project sponsor, ESM evaluation
team and evaluation manager. The team will be supported by external advisors.
The ESM Board of Governors’ Chairperson has nominated Gertrude Tumpel-Gugerell, an Austrian
economist who is a former vice-governor of the Austrian National Bank and a former member of
the Executive Board of the European Central Bank, as the evaluator.
The main tasks of the evaluator consist of:
• agreeing on the Terms of Reference,
• reviewing the draft report,
• participating in the consultation with the Board of Directors on the draft report,
• ensuring balanced final evaluation judgements,
• reporting to the ESM Board of Governors, and
• promoting outreach.
The evaluator would address her conclusions or assessments to the ESM Board of Governors’
Chairperson.
The report will be published under the evaluator’s auspices on the ESM website.
Based on the findings, the evaluator is expected to submit a set of recommendations to the ESM
Board of Governors to improve the functioning of the ESM and the programme activities.
The ESM project sponsor will ensure that the evaluator receives full support, meaning that
the evaluator’s instructions and guidance are implemented, the evaluation exercise has sufficient resources to execute its tasks in a timely manner, the evaluation team receives unhindered
access to information,8 and cooperative relations with the relevant institutions, such as the ESM
partner institutions for the programmes, are maintained. Both the ESM project sponsor and the
evaluation manager are responsible for keeping the project timetable on track.
The evaluation manager will organise and coordinate the daily work of the ESM evaluation
team, striving to ensure the relevance and credibility of the process. He will promote a common
understanding of the basic objectives, values, and assumptions underlying the analysis among the
evaluation team and ensure that evaluators protect the anonymity and confidentiality of the information and data gathered, where not publicly available. Moreover, the evaluation manager may
liaise with the evaluator and key stakeholders within the remit of the evaluation plan. External advisors will be consulted to ensure effective use of the best available expertise during the process.
8

8

In line with the documents outlining the scope of the exercise, which were submitted to the
governing bodies in May and June 2016, the evaluation team will have access to relevant EFSF/
ESM filing systems, documents and data within the confines of confidentiality arrangements.
ESM Members are expected to facilitate access as appropriate.

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

The evaluation team working under the coordination of the evaluation manager will:
• design and execute background studies and interviews,
• collect the necessary secondary data,
• draft a report for review by the evaluator,
• organise a consultation of the Board of Directors on the draft report,
• incorporate feedback into the final report in cooperation with the evaluator, and
• support the evaluator in reporting to the Board of Governors.
When designing the methodology, the team will pay due attention to avoid burdening the ESM
beneficiary members excessively with requests while ensuring their appropriate participation.
ESM internal services will support in particular with the editing of the final documents, and
with administrative tasks, such as contracting and procurement, and travel and meeting arrangements. Where necessary, the ESM may enlist external assistance to help, for example, with
initial training, preparation of interview transcripts, and logistics for fieldwork or further analytical
support. ESM Communications team will handle relations with the public and specific external
stakeholders.
The evaluation team, drawn from ESM staff members with differing areas of expertise and experience, will deploy a qualitative data analysis software package to support the reliability and validity of the mixed methods approach9 on which the evaluation is based.
Governance of evaluation
ESM Members as represented in its governing bodies and ESM management are the addressees
of the evaluation report. The evaluator will present her report to the ESM Board of Governors’
Chairperson who will submit it to the ESM Board of Governors. The report will be published following the discussion of the Board of Governors.
These Terms of Reference are approved by the ESM Board of Governors in written procedure
following guidance from the evaluator and discussion by the ESM Board of Directors.
The ESM Management Board will ensure appropriate follow-up to the evaluation report and the
recommendations with the governing bodies.
Management of potential conflicts of interest
The appointment of an external evaluator reflects the ESM’s intention to avoid any potential conflict of interest with ESM staff or management with respect to the evaluation exercise.
Together with the evaluator, the project sponsor, in facilitating the implementation of the
evaluator’s task, will support an independent and impartial evaluation process free from undue

9

See definition in Annex 1.
9

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

management influence. Advice from external advisors who have relevant experience of evaluation at international institutions will further be deployed to support independence.
ESM staff members who participated directly in the design and implementation of EFSF/ESM
programmes should be excluded from the evaluation team, although their input will be gathered and integrated into the analysis.
Quality assurance
These Terms of Reference have been prepared in cooperation with the evaluator. The evaluator
will review the draft report for consultation and ensure balanced judgements for the final report.
As a further quality assurance method, a former deputy director of the IMF Independent
Evaluation Office and the evaluation service of the European Investment Bank will support the
evaluation team in the design and analysis phases.
In addition, in accordance with the standard practice for such evaluations, the evaluation team
may simulate an evaluation reference group10 by involving in specific occasions further experienced evaluators mainly from selected inter-governmental institutions. While they would provide
expert guidance to the evaluation process and intermediate products, their engagement would
be purely advisory.
Consideration will be given to geographical representation within the evaluation team.
Deliverables and schedule
The main external deliverables are:
• A draft evaluation report for a stakeholder consultation facilitated by the evaluation team to
discuss preliminary evaluation findings and conclusions; and
• A final evaluation report, which will be made public.
These will be preceded by desk studies that will provide the basis for a series of interviews structured around the key evaluation questions outlined earlier in these Terms of Reference.
The broad timeline for the evaluation team’s work will be from September 2016 to April 2017. The
main steps foreseen in the high-level work plan are detailed in the following table.

10
10

See definition in Annex 1.

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Some of the tasks will nevertheless take place in part in parallel with others. An indicative timeline is as follows:
Phase

Timing

Pre-planning

June-August 2016

Nomination of the evaluator and terms of reference

July-August 2016

Practical work procedures, team on-boarding

End-August – mid-September 2016

Desk study phase

Mid-September – October 2016

Interviews

November 2016

Preparing draft report

December 2016 – mid-February 2017

Review by the evaluator

End-February 2017

Key stakeholder consultations

March 2017

Preparing final report and review by the evaluator

End-April 2017

Submission to the Board of Governors

May 2017

Presentation to the ESM Annual Meeting

June 2017

Source: ESM

The complexity of the subject (with multiple programmes, multiple programme objectives, partner institutions, and financial systems), together with the challenges related to accessibility and
availability of information, are the most likely risks to compliance with the projected schedule.
Approach and methodology
This section specifies in broad terms the methodological building blocks of the evaluation. Data
will be collected from a variety of sources (internal and external documents, surveys, interviews,
and focus group discussions). In line with best practice, this information will be analysed with various methods aiming at an in-depth review that considers context and relationships appropriately.
Under this approach, the objective is to use various analytical methods to obtain robust results.
While the mandate excludes in particular policy conditionality from the direct scope of this evaluation, the evaluation team may want to mention them if they represent potential explanatory factors. The team will consider good practice standards and evaluation guidelines (e.g. by the IMF,
OECD and networks such as the Evaluation Cooperation Group of the IFIs and United Nations
Evaluation Group) where applicable and available.
Desk studies
As part of the work plan preparation, the evaluation team will develop an intervention logic leading from inputs and activities to outcomes.11 This will provide a foundation for the information
collection and further analysis, together with the key evaluation questions identified in section 2.
This will involve document research, and analysing inputs, activities and outcomes as well as
developing indicators on efficiency and effectiveness.
In addition, the team will prepare a web survey on ESM country team coordinators’ experiences
with potential follow-up interviews, and interview ESM management. The survey will support the
development of the external interviews and guidance for interviewers.

11

See definition in Annex 1.
11

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

This phase will also help further familiarise the team with the programmes.
Interviews
The evaluation team will make a field visit to each programme country to supplement the findings
of background studies and to conduct interviews with both open and closed questions and potential focus group discussions on the key evaluation questions. Small teams will be drawn from
the overall evaluation team to conduct the interviews to ensure a sufficiently broad experience
base. The interviewing team and at least one additional team member will analyse the interview
transcripts. Mission reports and transcripts will be drawn from the key parts of the interviews to
facilitate a profound cross-sectional analysis. The interviews may be refined based on a parallel
analysis of the first interview experiences.
Interviewees will be selected based on their knowledge and roles in the relevant country programme and availability (purposive sampling12). Interviewees will have the opportunity to suggest
further interviews with people relevant for specific evaluation questions.
In addition to the programme countries, the intention is to schedule interviews with the institutional partners as far as they are disposed and able to contribute to the analysis within the
relevant timeframe.
All information will be stored or, if not otherwise possible, cross-referenced in a secure database
to which access is restricted to the evaluation team to ensure accuracy, protect the confidentiality
of participants, and generate conditions for an open exchange of views. This will also enable the
archiving of the project as a single file at completion.
References
EFSF and ESM financial assistance programmes are joint programmes conducted with the IMF
whenever possible. The ESM Treaty assigns certain tasks to the European Commission and the
European Central Bank. The institutions – previously called the Troika – operate jointly in the execution and monitoring of the programmes.
Other institutions have published different types of programme evaluations on the basis of their
own mandates and rules. Key reports listed in Annex 2. provide further contextual material for
this evaluation. Some evaluations, e.g. by the European Central Bank, have not been disclosed.
To draw upon rather than duplicate the work of the other evaluations, where appropriate, the
reports will be (i) examined for evidence for evaluation questions, in particular, on the role of the
ESM, and (ii) to identify additional specific questions of relevance to this evaluation.

12
12

See definition in Annex 1.

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Annex 1. Definitions for the purposes of these Terms of Reference
The following definitions provide broad guidance on the general evaluation terminology used in
these Terms of Reference.
Term

Definition

EC

European Commission

Effectiveness*

The extent to which the financial assistance programme’s objectives were
achieved, or are expected to be achieved, taking into account their relative
importance.

Efficiency

A measure of how economically resources/inputs (funds, expertise, time, etc.)
are converted into results. (OECD-DAC definition (2002))

EFSM

The European Financial Stabilisation Mechanism is a lending arrangement
guaranteed by the budget of the European Union.

ESM BoD

ESM Board of Directors

ESM BoG

ESM Board of Governors

ESM partner
institutions

Institutions directly involved in programme design and financing, as
referenced in EFSF/ESM governing documents.

Evaluation reference
group

An informal group of experienced evaluators, selected by the evaluation
team in cooperation with the sponsor and the external advisors, representing
primarily inter-governmental organisations. Their roles are advisory.

External advisers

Advisers contracted to contribute to quality assurance of the evaluation
process and assist the evaluation team in the design and analysis phases.
Their responsibilities are advisory.

External stakeholders

Institutions other than the key stakeholders that are affected, including
academia and the general public

Finding

A finding uses evidence from one or more sources to allow for a factual
statement.

Formative evaluation

An evaluation intended to improve performance by drawing lessons for the
future.

IMF

International Monetary Fund

Impact*

Positive and negative, primary and secondary long-term effects produced
by a financial assistance programme, directly or indirectly, intended or
unintended.

Impact evaluation*

An impact evaluation quantifies the net change in outcomes that can be
attributed to a specific project or programme, usually by the construction of
a plausible counterfactual.

Intervention logic

Such logic explains how a programme or a policy is understood to contribute
to a chain of results that produce the intended or actual outcomes or impacts.
There are various ways, such as a flow chart or a results chain, to illustrate it.
Also known as programme theory or causal model.

Key stakeholders

ESM Members and partner institutions

Lessons learned*

Generalisations based on evaluation experiences with financial assistance
programmes and related policies that abstract from specific circumstances to
broader situations. Frequently, lessons highlight strengths or weaknesses in
preparation, design, and implementation that affect performance, outcome,
and impact.

Mixed methods
approach

Research approach using quantitative, qualitative, and blended approaches
to information collection and analysis, widely accepted as a good-practice
strategy for the evaluation of complex programmes.

Outcome evaluation*

Evaluation of the likely or achieved short-term and medium-term effects of
a financial assistance programme’s outputs.

Output*

The support and services which result from a financial assistance programme;
this may also include resulting changes which are relevant to the achievement
of outcomes.

Primary audience

The addressees of the evaluation report who include ESM Members, as
represented in its governing bodies, and management of the ESM.

13

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Term

Definition

Process evaluation*

An evaluation of the internal dynamics of implementing organisations, their
policy instruments, their service delivery mechanisms, their management
practices, and the linkages among these. It may further expand to cooperation
with partner institutions and member states.

Purposive sampling

Non-probability sampling typically associated with qualitative research in
which a relatively small number of persons, cases or other units of analysis
are selected because they can provide particularly valuable information
related to the research questions. Also known as judgemental, selective, or
subjective sampling.

Relevance*

The extent to which the objectives of a development intervention are
consistent with country needs, the institutional mandates, and partners’
policies or, where relevant, global priorities.

Stakeholders

Key stakeholders and external stakeholders

Sources: ESM, *Derived from OECD-DAC definition (2002)

14

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Annex 2. References
Reference evaluations
IMF IEO (2016) The IMF and the Crises in Greece, Ireland, and Portugal—An Evaluation by
the Independent Evaluation Office, EB/EVC/16/13, July 2016 (www.ieo-imf.org/ieo/pages/
CompletedEvaluation267.aspx)
EC (2016) Evaluation of the Financial Sector Assistance Programme. Spain, 2012-2014, Institutional
paper 019, European Commission, January 2016. (ec.europa.eu/economy_finance/publications/
eeip/pdf/ip019_en.pdf)
EC (2015) Ex post evaluation of the economic adjustment programme, Ireland, 2010-2013,
Institutional paper 004, European Commission, July 2015 (ec.europa.eu/dgs/economy_finance/
evaluation/pdf/ex-post_ireland_en.pdf)
ECA (2015) Financial assistance provided to countries in difficulties, Special report no 18/2015,
European Court of Auditors, January 2016 (www.eca.europa.eu/en/Pages/DocItem.
aspx?did=35016)
IMF IEO (2014a) IMF leadership and coordination in the response to the Global Financial and
Economic Crisis, IEO Background paper, BP/14/06, 8 October 2014 (www.ieo-imf.org/ieo/files/
completedevaluations/BP1406%20-%20Bernes.pdf)
IMF IEO (2014b) IMF Macroeconomic policy advice in the financial crisis aftermath, IEO
Background paper, BP/14/07, 8 October 2014 (www.ieo-imf.org/ieo/files/completedevaluations/
BP1407%20-%20Dhar.pdf)
IMF IEO (2014c) Aspects of IMF financial sector surveillance during the crisis, IEO Background
paper, BP/14/08, 8 October 2014 (www.ieo-imf.org/ieo/files/completedevaluations/BP1408%20
-%20Levine2.pdf)
IMF IEO (2014d) Review of crisis management programs supported by IMF Stand-By
Agreements, 2008-11, IEO Background paper, BP/14/12, 8 October 2014 (www.ieo-imf.org/ieo/
files/completedevaluations/BP1412%20-%20Takagi%20and%20Others_crisis2.pdf)
IMF IEO (2013) The Role of the IMF as Trusted Advisor, Evaluation report, February 2013 (www.
ieo-imf.org/ieo/files/completedevaluations/RITA_-_Main_Report.pdf)
IMF (2015) Crisis program review, 9 November 2015, (www.imf.org/external/np/pp/
eng/2015/110915.pdf)
IMF - Ex post evaluations on exceptional access for programme countries (Greece, June 2013;13
Latvia, January 2013;14 Ireland, January 201515), IMF country reports

13
14
15

www.imf.org/external/pubs/ft/scr/2013/cr13156.pdf.
www.imf.org/external/pubs/ft/scr/2013/cr1330.pdf.
www.imf.org/external/pubs/ft/scr/2015/cr1520.pdf.
15

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Other references
Independent Evaluation Group (2011) Writing Terms of Reference for an Evaluation: A how-to
guide, World Bank, (siteresources.worldbank.org/EXTEVACAPDEV/Resources/ecd_writing_
TORs.pdf)
OECD–DAC (2002) Glossary of Key Terms in Evaluation and Results Based Management, OECD
Development Assistance Committee, Paris. (www.oecd.org/dac/evaluation/2754804.pdf)
UNEG (2010), Quality Checklist for Evaluation Terms of Reference and Inception Reports, United
Nations Evaluation Group, (www.iom.int/jahia/webdav/site/myjahiasite/shared/shared/mainsite/
about_iom/eva_techref/UNEG_TOR.pdf)
UNEG (2016) Norms and Standards for Evaluation, United Nations Evaluation Group, June 2016
(www.uneval.org/document/detail/1914)

16

B. Methodology

Introduction
This note summarises the ESM evaluation from the perspective of process and methodology.
It is a general practice among international institutions that policy interventions should be evaluable and evaluated. The ESM evaluation exercise aimed at improving future financial assistance
activities by actively seeking feedback on lessons learned and by making the information public.
The purpose of this methodology note is to conceptualise the evaluation exercise conducted by
the ESM evaluation team. It contextualises the exercise, explains the types of data used to establish findings, and answers the evaluation questions. Moreover, it addresses the composition of
the evaluation team and substantiates the credibility, transparency and impartiality of this evaluation exercise. Arrangements were made to ensure the exercise was as impartial and transparent
as possible, as guided by the Terms of Reference.
This methodology note should be read in conjunction with the Terms of Reference and the note
on the Intervention logic. The Terms of Reference lay out the mandate and the main objectives
of the exercise. Conducting an evaluation in a reliable and credible fashion implies the use of
a robust evaluation design that reduces the risk of irrelevant or invalid findings. The exercise
encountered several data and document availability limits. Some relevant respondents could not
be reached within the designated mission dates. As a matter of transparency, these limits will be
addressed in the following.
The evaluation process was built on six phases, starting from planning, data collection and analysis, going through various iterations to inference and drafting the report. The draft was submitted
for consultation at the ESM Board of Directors and to peer institutions. Following the feedback,
the Independent Evaluator formulated final recommendations.
Chapter 1 of this Appendix covers the planning phase. Chapter 2 explains the overall strategy and
Chapter 3 explains how the team proceeded to define the evaluation questions and conduct the
analytical work. Chapter 4 concludes.

1.	

Organising the evaluation

Roles and responsibilities
The evaluation project was conducted by a multitude of actors with different roles and responsibilities. These actors include the Independent Evaluator (Gertrude Tumpel-Gugerell), the ESM
project sponsor, the ESM evaluation manager, the ESM evaluation team as well as external advisers (a former deputy director of the IMF Independent Evaluation Office and the director of the
evaluation service of the European Investment Bank). The two external advisers were appointed
to advise the team, ensure quality control and raise potential concerns about conflicts of interest.
Moreover, the former IMF evaluator actively advised on the design of the background studies
as well as transition from the background studies to the draft report. The EIB evaluator played
a deeper role in the planning phase of the evaluation exercise. Additionally, two external consultants were integrated into the evaluation team to further support credibility and independence of
17

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

the evaluation project. Finally, when editing the report, an external editor was made available for
the production process.
The Independent Evaluator Ms Tumpel-Gugerell provided direction and performed specific tasks,
such as reviewing the draft texts and leading work on the recommendations. The main tasks
were defined in the Terms of Reference. Based on the findings, Ms Tumpel-Gugerell formulated
a set of recommendations to the ESM Board of Governors to advance future ESM financial assistance. The report was published under her auspices on the ESM website.
The members of the multi-disciplinary team were selected on the basis of their expertise in
a particular sector (ALM, banking, back office, legal, corporate governance, etc.). In order to minimise external concerns over potential conflicts of interests, each member of the evaluation team
(including the external consultants) was asked to identify any conflict of interest regarding their
participation. As a consequence, when an ESM member had been directly involved in a country
programme (design or implementation), they did not participate in the interviews in that country. A certain flexibility was, however, allowed with regard to post-programme monitoring. This
meant the composition of the ESM interview teams varied from one country to another.
Planning phase
The Chairperson of the Board of Governors’ Jeroen Dijsselbloem initiated the preparations for
assessing the relevance, effectiveness and efficiency of EFSF and ESM financial assistance in
safeguarding financial stability of the euro area and its Member States. In the pre-planning phase,
ESM staff conducted preliminary examinations on evaluations and policy studies made by other
institutions in order to propose a scope that would limit overlaps with the relevant existing evaluations and ensure value-added to the primary audience.
The Board of Directors twice discussed the scope of the exercise, and the Board of Governors
approved its mandate at its Annual Meeting on 16 June 2016. The Independent Evaluator was
formally appointed on 22 September 2016.
Following consultation with the Independent Evaluator, the Board of Governors approved the
Terms of Reference (Appendix A) in a written procedure on 24 October 2016. The Terms of
Reference is a key document in the EFSF/ESM Evaluation process as it advised how the evaluation should be conducted. It presented the objectives of the evaluation, the role and responsibilities of the actors involved, the timeline and the resources available. The Quality Checklist for
Evaluation Terms of Reference and Inception Reports by United Nations Evaluation Group was
used as a reference when developing the Terms of Reference for this exercise.
The evaluation questions
Assessing the relevance, efficiency, effectiveness and collaboration relevant for EFSF/ESM financial assistance was the core objective of the report. The Terms of Reference assigned evaluation
questions to four criteria (See Table 1) and the appropriate evaluation period (negotiation, execution, post-programme monitoring). Some evaluation questions concerned the entire programme
period, others were directly linked to a particular period of the programme process.

18

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Table 1: Definition of the evaluation criteria
Criterion

Definition

Relevance

The extent to which the objectives of a development intervention are consistent
with country needs, the institutional mandates, and partners’ policies or, where
relevant, global priorities.

Effectiveness

The extent to which the financial assistance programme’s objectives were achieved,
or are expected to be achieved, taking into account their relative importance.

Efficiency

A measure of how economically resources/inputs (funds, expertise, time, etc.) are
converted into results.

Collaboration

The extent to which the objectives and process of collaboration, the methods of
transparency for the membership base and the larger community functioned.

Each evaluation question falls under an evaluation criterion. Three questions were not addressed.
More information is given in section 1.1 of the evaluation report.
A theoretical intervention logic (Appendix C) for the EFSF/ESM was developed by the evaluation
team to frame the analysis and guide the final definition of the interview strategy. This helped
develop the topics and background questions for the semi-structured interviews that were conducted between October 2016 and January 2017.
The scope of the evaluation was limited by the instruction not to evaluate the following:
• The political aspects of the Eurogroup negotiations or certain aspects of programme
design that are the specific responsibility of other institutions, such as conditionality, even if
the ESM Treaty establishes a fundamental link between financial assistance and the compliance with conditionality.
• Individual supervisory decisions;
• The Greek programme in its entirety. To avoid compromising current activities under the
third Greek programme, the evaluation team was asked to evaluate relevant aspects of the
second Greek programme until end-December 2014, the initial expiry date of the programme.
The entire second Greek programme could be evaluated together with third after completion
of the latter in August 2018, as mentioned in the Terms of Reference.
• The role of the partner institutions was not evaluated per se, but they were taken into
account when designing and scoping the evaluation as they feed into the organisational setup
of the programmes. In particular the following were considered:
– relevant policy discussions take place in the Eurogroup and Eurogroup Working Group
closely matching the EFSF and ESM governing bodies;
– programme conditionality is negotiated by the European Commission in liaison with the
European Central Bank;
– programmes were conducted as joint programmes with the IMF whenever possible; and
– post-programme monitoring is done by the European Commission and the Early Warning
System is conducted by the ESM in parallel.
These issues, which lie outside the evaluation’s formal scope, were addressed when they served
as potential explanatory factors. As an example, the EFSF/ESM disbursement processes were
analysed as a function of the conditionality setting process.

19

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Both the Board of Directors and Board of Governors discussed the evaluation period. The agreed
period covered the negotiations for each programme and ran through the post-programme period
up to 30 June 2016. During the exercise, the Independent Evaluator decided to widen the evaluation to encompass the pre-negotiation period. Greece was nevertheless subject to the above
limitation.
Because of the relatively recent completion of the programmes under evaluation, the exercise was
not expected to prepare an impact evaluation quantifying the net change in outcome attributed
to a specific project or programme, usually by the construction of a plausible counterfactual. For
the same reason, certain evaluation questions such as the sustainability of programme benefits
were not assessed.
Figure 1 illustrates the main steps of the evaluation.

20

STEP 1

Team
on-boarding and
Nvivo training

October
2016

STEP 2

STEP 3

STEP 4

STEP 5

STEP 6

Source: ESM

Meeting with the Independent Evaluator

Project formally
approved on
22 September 2016

Planning / team on-boarding

June/September
2016

Figure 1
Evaluation process
December
2016

Important events

Interviews

Desk study phase

Data collection and analysis

November
2016

February
2017

Review by Independent Evaluator

Drafting

Preparation of draft report

Coding // qualitative analysis

January
2017
Consultation

March
2017

May
2017

Review by
Evaluator

Drafting

Additional
qualitative
analysis

Finalise the
report

Finalisation/Production

April
2017

ESM Annual
Meeting in
June 2017

Publication

June
2017

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

BoD
Comments

BoD
Consultation

21

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

2.	

Evaluation strategy

The Terms of Reference guided the team to adopt a mixed methods approach for this evaluation entailing the use of both qualitative (QUAL) and quantitative (QUANT) methods in various
combinations. The purpose was to profit from the advantages of both quantitative and qualitative approaches and to overcome their drawbacks by mixing them. The objective of using
mixed methods was to strengthen the reliability of the data and the validity of the observations
and recommendations as well as to broaden and deepen the understanding of the processes
leading to the programmes’ results, including how the context surrounding the programmes’
implementation affects these results. Another value of the strategy arises from its potential to
strengthen the ability to explain when mixing happens at multiple stages of the evaluation process. (Woolcock, 2003; Creswell, 2006)
Cross-country analysis, instead of country-by-country analysis, was chosen because it was felt
this would yield greater clarity in the resulting policy lessons. The secondary objective was to
support transparency.
Given the broad scope of the Terms of Reference, the strategy built on several analytical strands
and used triangulation to ensure reliability. Triangulation was used to obtain complementary data
on the topics under study and strengthen the inference process. This took place at several levels
of the exercise:
• Data triangulation: involving the collection and use of a variety of data sources (for example,
macroeconomic forecasts and statistics, financial market data, ratings data, surveys, interviews, news databases, and programme and policy documents).
• Methodological triangulation: engaging in multiple methods to study a single problem
(for example, interview responses verified against survey responses and document analysis;
financing needs data and vulnerability analysis interpreted on the basis of interview findings);
• Investigator triangulation with several team members in a single study (for example in analysis of key events, pricing policy and terms of lending, timing of request and exit strategies).
(Morse, 1991; Teddlie, Rashakkori, 2009)
In addition, data was in many cases collected, compiled and analysed by small teams separate
from the team making inferences and drafting the report. Interview summaries or transcripts
were always verified by a minimum of two persons from the relevant interview team. At least
one of the two team leaders coded each interview.
The topology (Figure 2) varied between the analytical strands, which were largely run in parallel.
In some cases, the qualitative phase preceded a quantitative study used to confirm findings.
In other cases, quantitative analysis was enriched by qualitative information or by transforming quantitative data to qualitative (convergence model). The use of the convergence model,
which represents the traditional model of a mixed methods triangulation design, consists of the
separate collection and analysis of quantitative and qualitative data on the same phenomenon.
Transforming one data type into another is accomplished by either quantifying qualitative findings
or qualifying quantitative results (Cresswell, 1999; Tashakkori & Teddlie, 1998). This transformation allowed the data to be mixed during the analysis stage and facilitated the comparison, interrelation, and further analysis of the data sets. The results converged during the interpretation.
This method is often used to validate, confirm or corroborate quantitative results with qualitative
findings.

22

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

In a few cases the study involved two qualitative phases (e.g., interview results combined with
document analysis).

Figure 2
Topology of evaluation studies
ESM staff Survey
Pricing
Policy

Budgetary
savings

Vulnerability

Financing
needs

Financial
sector
repair

Timing of
request

Exit
strategies

QUAL

QUANT
// Market

QUANT

QUANT
Database

QUAL

QUANT

QUAL
Interviews

Timeline

Disbursement

CTC

Cooperation

Choice of
instrument

QUAL

QUAL
ICF Charts

QUAL

QUAL

QUAL
QUANT

QUANT
// IMF

QUANT

QUAL
Document
studies

QUANT

QUAL
Interviews

QUANT
Indicators

QUAL
Interviews

QUAL
Interviews

QUAL

QUAL
News
database

QUAL
Document
studies

QUAL
QUANT

QUAL

Repayment
ability

QUAL
CTCs

QUANT

QUAL

QUAL
QUAL
Interviews Management

Interviews
Early Warning System reports and summaries from political meetings
Board survey

INFERENCE

Triangulation

Used to identify specific themes to focus on when using quantitative data

Transformation

Source: ESM

The exercise features both as an outcome and formative evaluation. As it aimed to take into
account ESM procedural aspects in the overall institutional framework, it was characterised as an
outcome evaluation. As such, the purpose was to evaluate the likely, or achieved, short-term and
medium-term effects of a financial assistance programme’s output. As a formative evaluation, it
intended to support performance improvement by drawing lessons for the future.
Conclusions were drawn both at the level of individual desk studies and subsequently as part
of the iterative drafting process between the evaluation team and the Independent Evaluator.
Inference should be understood both as a process of drawing conclusions and the conclusions
themselves. The complexity of the financial assistance activities and the study questions required
a broad approach involving consideration of contextual factors, process and underlying meanings
as part of the inference process.
Report drafting was structured around iterative reviews by the Independent Evaluator to ensure
her full ownership. This helped eliminate alternative interpretations together with the consultations and team member checking.

23

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

The inference process culminated in the formulation of recommendations as called for by the
Terms of Reference. The value of the recommendations can be assessed in relation to their relevance and usefulness to the policymakers over time. In this context, the Independent Evaluator
and the team felt guided to discuss broad future avenues, for example, on the basis of the evaluation questions dealing with collaboration and on conditions necessary for most effective support to potential future programmes countries or in periods of heightened stress. Indeed a large
majority of the interviewees spontaneously raised the topic of future role of the ESM.

3.	

Components of the evaluation

3.1

Collection of evaluation data

To answer the evaluation questions, the evaluation team needed to determine diverse types of
evidence (quantitative and qualitative data) that were collected for each criterion. A plan was
set-up to collect such evidence, coherent with a mixed methods approach.
Numerical studies and document studies
The findings of the evaluation report were based on diverse qualitative sources of data: an
exploratory staff survey (containing 32 responses from members of senior management, country team coordinators and experts), a Board survey (to which 24 Board members responded)
and 79 semi-structured expert interviews conducted between October 2016 and January 2017
(a total of 130 respondents from programme countries, partner institutions and ESM governing
bodies). Various kinds of written documents were used as sources, keeping in mind that they
may contain errors or omissions, like other forms of evidence.
Numerical studies focused mainly on the relevance of financial assistance, the efficiency of the
disbursement process, the effectiveness of financial sector repair and planning for exit from the
programmes. During the evaluability assessment, the evaluation team realised that the governance framework for European financial assistance was missing a public database, in line with
IMF practice, which would disseminate harmonised data on the country programmes. The creation of such a database for the purposes of the evaluation team has been an important step for
the analysis and collection of the findings. Also collection of the relevant disbursement data had
to be organised from the ESM administrative systems. For financial market data, commercial data
sources were used.
Sampling of interviewees and survey respondents
The driving sampling strategy was purposive, also known as selective sampling. It’s a type of
non-probability sampling, when cases are selected on a judgement basis. The phase can also be
characterised as reputational sampling whose purpose is to extract information from individuals
who have particular expertise.
The exploratory staff and management surveys were targeted at the members of internal country
programme mailing lists and supplemented by some previously involved staff. The Board survey
was addressed to all current ESM Directors with a request to invite also previous directors and
governors to respond.

24

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Box 1 describes the steps for the selection of the cases for interviews.
Box 1: Selection of interviewees
Step 1: ESM management and staff survey in September 2016 – request to highlight the most
relevant persons for interviews in each programme country and partner institution;
Step 2: Verify against country teams in programme documents;
Step 3: Contact the target institution or finance ministry with a ‘wish list’ of interview requests
that they were able to complement;
Step 4: The evaluation team arranged additional ad hoc interviews of reputable experts who
were no longer working in the targeted institutions. The Independent Evaluator conducted separately several background discussions which informed her guidance and review.
All persons proposed by the authorities were interviewed. A small number of persons on the
wish list were not available during the limited interview missions (2-3 days per country) despite
efforts to keep daily schedules flexible and accommodative. In some cases, it was not possible
to reach experts who had moved to the private sector or abroad. Given time limitations, it was
decided to concentrate on public sector interlocutors.
Figure 3 and Figure 4 characterise the distribution of interviewees for this evaluation.

Figure 3
Number of interviews, by relevant
programme

Figure 4
Number of interviews, by type of
institution

25

40
35

20

30
25

15

20
10

15
10

5

Other

Central bank

Ministry

0

Peer institution

Multiple

Cyprus

Spain

Greece

Portugal

Ireland

0

5

Source: ESM

As a rule, the respondents received the Terms of Reference and a list of focal topics in order to
prepare for the interview. Respondents were free to address the topics they thought were most
relevant. This allowed them to reflect on their own priorities in the context of the assistance
programmes.

25

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

The interviewers were briefed on pre-mission discussions and through a guidance note containing instructions on how to conduct the interviews. The evaluation team was guided by a set of
more detailed questions that were not shared with the respondents. Effort was made to adapt
these questions to each interviewee’s profile. ESM Interview teams included 3-4 persons who
took turns in leading the discussion and taking notes. They operated under two team leaders,
which allowed for some overlap in the missions to save time.
Interview teams prepared summaries of each interview applying the four-eye principle (meaning
approval was required by two people who were present). Where the team was allowed to record
discussions, a verbatim transcript was produced with a similar verification principle. About 55%
of all the interviews were recorded and transcribed, ranging from 10-100% depending on the
country (and averaging 65% among national authorities).
Initially, the evaluation team planned to conduct focus groups after the interviews in order to
scope the respondents’ attitudes or reactions in a way which would not be feasible during the
one-to-one interviews or questionnaire surveys. However, this was not possible in the timeframe. In most cases, the respondents chose to be accompanied by one or more colleagues,
which may have led some of them to put emphasis on institutionally accepted views. Although
the interviewee sometimes asked colleagues to complement his or her message, it was always
clear who was the main respondent.
3.2

Desk studies

In the early stages, multiple desk studies were conducted by the ESM evaluation team with the
support of other internal ESM employees. The desk studies had two key objectives: ensuring the
reliability of evidence by triangulation, and helping team members familiarise themselves with
key topics, which finally fed into the report.
These desk studies reflect the mixed methods approach of the evaluation exercise by their qualitative or quantitative aspects. The following tables describe the main features of the desk studies. Some of them encountered hurdles, including unhelpful processes, data collection issues, as
well as factors or events not under the evaluation team’s control. These hurdles were taken into
account during the analysis and inference phases.
Vulnerability analysis
A study on evolution of the sovereign vulnerabilities was chosen as the backbone of the background studies, in particular in relation to the relevance criteria. Appendix D presents technical
background for the study.

26

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Timeline of the programmes and crisis resolution
Purpose

Scope

To establish a collection of events
relevant to the evaluation, together with
their exact date, document reference,
category, source type and key words, to
allow fact checking for the draft report
and to support other desk studies.

EFSF and ESM programme activities; and key external
events.

Type of data

Method(s) used

Quantitative: amounts disbursed, ratings

QUAL-QUAL strategy. Drawn up from internal and external
databases and sources.

Qualitative: websites, newspapers, press
releases, official documents
Sources

Problem/bias

Quantitative: ESM data, Rating agencies

Some dates turned out to be incorrect but given that
various sources were used, cross-comparison was
possible.

Qualitative: national, European and
international (IMF) websites programme
documents, evaluation reports, press
releases
Additional comment

The timeline partly relies on internal, non-published sources, in particular Board decisions and lending
documentation. For a summary presentation see Appendix G.

Timing of request
Purpose

Scope

To study the pre-negotiation period to
determine timeliness of country requests
for financial assistance and review
indicators that could signal increase in
financing needs over a relevant period.

Programmes for Ireland, Portugal, Greece, Spain and Cyprus.

Type of data

Method(s) used

Qualitative: Interviews, web search

QUANT-QUAL-QUAL strategy. For the quantitative phase we
determined indicators that could have impact on the size of
financing needs. The second step reviewed the evolution of
the indicators over 12 months (matching normal surveillance
period of economies) prior to the official request.

Quantitative: Market and financial data;
Issuance data

In the first qualitative phase, we drew on expert interview
questions to identify potential alternative dates for
commencement of programmes. In the second qualitative
phase, we cross-checked the expert views with press
reports. Relevant dates were superposed to charts and the
evolution of the indicator compared with the actual date for
request.
Sources

Problem/bias

Quantitative: Bankscope, Bloomberg,
Dealogic, ECB, European Commission,
Haver, SNL. Issuance data in common
with the study on exit strategies.

Part of the data was of annual frequency while interviewees
indicated months of relevance.

Qualitative: Interviews, FACTIVA new
database, Google.

No control group was used to eliminate general trends.
This simplified analysis simulated no change in the trajectory
of the indicators when assuming earlier request.
Informality of the request process.

Evaluation question(s)
Did the information provided in the negotiation phase ensure the efficient conduct of assistance?
Indicators
Evolution of bond spread, capital, portfolio inflows and NPLs compared to time of request and expert
views on earlier timing for potential need for assistance. Appendix E charts the relevant indicators by
country.
Additional comment
Analysis involved inputs from three investigators, each phase executed largely separately.
27

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Assessment of the choice of instrument
Purpose

Scope

To support the analysis of effectiveness. Clarify
pertinence of the financial assistance instrument
requested and granted for the five programme
countries. Scoping the roles played by various
stakeholders in initiating negotiations.

EFSF/ESM programmes for Ireland, Portugal,
Greece (second programme), Spain and Cyprus.

Type of data

Method(s) used

Qualitative and quantitative

QUAL-QUAL strategy. Following interviews,
we studied available documents and the board
survey to corroborate findings; and clarified the
instrument criteria in Appendix F.

Sources

Problem/bias

Quantitative: Financing needs analysis

All five evaluated programmes were requested
under the EFSF, which does not have precise
criteria for instrument selection other than
contagion risk. This impacted the relevance of
the question in the Terms of Reference “Did
requests for financial assistance programmes and
the assessment by institutions comply with the
requirements of the ESM Treaty and Guidelines,”
and therefore it was not prioritised.

Qualitative: Official requests, interviews, EFSF/
ESM legal texts; vulnerability analysis; evaluations
and programme reviews by other institutions;
Board survey.

Evaluation question(s)
Was the granted assistance instrument appropriate and the programme period adequate?
Indicators
Instrument indicated in the request for assistance
Arguments and circumstances surrounding the choice of instrument
Correspondence between the choice and the requirements for the particular financial assistance
instrument
Additional comment
The study is linked to the study on timing of request. Appendix F summarises EFSF/ESM instruments.

28

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Sizing of the financial assistance (financing needs)
Purpose

Scope

To examine how financing needs were
derived and how they evolved over
time. Examination whether programme
contingencies, and the need for buffers,
were adequately taken into account.

Financial assistance granted to Ireland, Portugal, Greece
(second programme), Spain and Cyprus by the EFSF/ESM
and other programme partners.

Type of data

Method(s) used

Quantitative data complemented by
qualitative information from reports and
interviews.

QUANT-QUAL-QUAL strategy. Comparative analysis:
Comparison across programme countries and comparison
between initial programme planning and programme
outcome.

Sources

Problem/bias

Programme reviews and evaluation
reports, mostly from the European
Commission, complemented by sources
from national authorities, central banks,
and other institutions. Interviews with
authorities in member countries and
international partner institutions.

Country comparisons were hampered because data
reporting was not fully harmonised data (e.g. reporting of
financing needs including or excluding amortisation of shortterm debt). In some cases lack of concluding programme
reviews and publicly available end-programme data (e.g.
data on “programme outcome” for Greece is based on the
fourth review which contains estimations, particularly for
the final programme year.

Evaluation question(s)
How has the size of financial assistance been defined?
To what extent have contingency buffers been taken into account and communicated?
Has the assessment of the required size of financial assistance changed over time (evolution of
financing needs, planned vs. actually disbursed amounts)?
Identification of common patterns across programme countries.
Indicators
Gross financing needs and its subcomponents.
Financing sources and its subcomponents.
Size of financial assistance granted by all creditors.

29

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Efficiency of the disbursement process
Purpose

Scope

To assess the efficiency of the EFSF and
ESM disbursement process.

Disbursements made under EFSF/ESM programmes for
Ireland, Portugal, Greece (second programme), Spain and
Cyprus.

Type of data

Method(s) used

Quantitative and qualitative data:
qualitative data was transformed into
quantitative data for analysis then
cross-checked with qualitative data
(interviews).

QUAL-QUANT-QUAL strategy. Comparison and
manipulation of various dates extracted from the timeline
database.
We measured the average number of days between
the date a disbursement was requested (the so-called
Request for Funds) and the date the payment was made.
This measure proved to be flawed, since in some cases,
one request for funds covered several tranches (EFSF) or
instalments (ESM). In these cases, only the first of the
series was taken into account, as the subsequent ones
were dependent on the completion of additional action on
the part of the programme country and therefore did not
measure the efficiency of the EFSF/ESM.
This still left us with several outliers which we were able
to explain with help from experts. These related to other
dependencies such as state aid approval, and to the fact
that Requests for Funds were sometimes sent long before
Board approvals were obtained (pointing again to the fact
that the measure was flawed; however, a more accurate
measure could not be found).

Sources

Problem/bias

Quantitative: EFSF and ESM Board
approvals; ALM & Lending database.

The disbursement process changed over time to increase
flexibility; this complicated comparison over time.

Qualitative: interviews
Evaluation question(s)
Were members treated even-handedly in light of the size of assistance relative to each country’s
financing needs, lending terms, or disbursement practices?
Was the financing planning and disbursement strategy conducive to effective programme
implementation?
Have activities (e.g. fund raising and disbursements) taken place in a timely manner?
What measures were taken to ensure that resources were used efficiently?
Were the financing terms sufficiently flexible in view of uncertainty?
Did programmes target effectively the financing needs of a country and account for the specific
requirements of bank recapitalisation?
Indicators
Frequency of disbursements
Number of days between the date on which a programme country sends a Request for Funds and the
date the disbursement is made
Average ratings of programme countries
Cumulative share of total disbursed amount in %

30

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Evolution of EFSF/ESM’s pricing policy
Purpose

Scope

To assess appropriateness and evenhandedness of the EFSF/ESM’s pricing
policy

EFSF/ESM programmes for Ireland, Portugal, Greece
(second programme), Spain and Cyprus.

Type of data

Method(s) used

Quantitative, qualitative

QUAL-QUANT strategy. Comparative analysis

Sources
Quantitative: Lending rates
Qualitative: Pricing policy, funding
strategy, Board decisions
Evaluation question(s)
Were members treated even-handedly in light of the size of assistance relative to each country’s
financing needs, lending terms, or disbursement practices?
Were the financing terms sufficiently flexible in view of uncertainty?
How did borrowing costs and maturity structure support programme achievement and programme
sustainability?
Indicators
Evolution of EFSF/ESM lending rates

Budgetary savings from the EFSF/ESM’s lending terms
Purpose

Scope

To estimate the budgetary savings from
EFSF/ESM’s financial assistance.

EFSF/ESM programmes for Ireland, Portugal, Greece
(second programme), Spain and Cyprus.

Type of data

Method(s) used

Quantitative

QUANT-QUANT strategy. Counterfactual and comparative
analysis.
As far as EFSF/ESM financial assistance is concerned,
the simplest way to estimate the savings achieved over
the past years is to compare the effective interest rate
payments on EFSF/ESM loans with the interest rate that
these countries would have paid had they been able to
cover their financing needs in the market in the absence
of disruption. The proposed approach values every single
disbursement in the past at the average market 10-year
bond yield in a year.

Sources

Problem/bias

Quantitative: ESM lending rates, IMF
lending rates, market rates

Calculation of budgetary savings from EFSF/ESM’s lending
terms is only a rough approximation to illustrate order of
magnitude as alternative financing scenarios (market or IMF
financing) are only hypothetical.

Qualitative: ESM and IMF pricing policies
Evaluation question(s)

Were members treated even-handedly in light of the size of assistance relative to each country’s
financing needs, lending terms, or disbursement practices?
How did borrowing costs and maturity structure support programme achievement and programme
sustainability?
Indicators
Comparison between the budgetary savings from EFSF/ESM financing and the hypothetical
alternatives in which the additional gross financing needs would be met by market borrowing or from
the IMF

31

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Financial sector repair
Purpose

Scope

To summarise and analyse the EFSF/ESM
participation in the financial sector parts of the
concluded economic adjustment programmes.
The study recalls the different ways of designing
the financial sector programmes throughout
the crisis. It also looks at the success of the
programs in light of the objectives of financial
sector stabilisation and efficiency. The purpose
of the desk study is to form a background for the
Independent Evaluator and the interviews with
stakeholders.

The study focuses on repair of the financial sector
in all programme countries, irrespective of the
type of instrument used (i.e. loan to the sovereign
or indirect recapitalisation). It discusses the
vulnerabilities built up in the pre-crisis period, the
way they were addressed during the programme
and the remaining issues that still have to be
addressed. The assessment has been performed
from the perspective of the ESM’s primary goal, i.e.
to maintain financial stability in the euro area.

Type of data

Method(s) used

Both quantitative and qualitative. The time horizon QUAL-QUANT-QUAL strategy. The approach
covers the period of 2007 Q4 – 2016 Q3.
comprises a comparative analysis of specific
themes across programme countries based on
qualitative and quantitative information. Themes
included bank solvency, liquidity, asset quality
and governance. Quantitative data are used in
a standardised format in order to filter out the
differences in levels. Qualitative information was
used to highlight specific themes for focus.
Sources

Problem/bias

Quantitative data were used for the comparative
analysis. The main sources of data collected are
financial databases such as SNL and BankScope
as well as documentation such as programme
reports. Qualitative data includes ESM staff level
discussions with programme participants via the
internal survey and review of internal and external
documents and studies.

Conclusions have been drawn from the sole
perspective of whether financial stability was
restored or maintained. As such, they do not
take into account other considerations, such as
maximizing the value of public investments etc.
Furthermore, the quantitative databases cover
only the systemically important banks, therefore
it provides a comprehensive, but not full picture
of the entire banking sectors. Difficulties were
identified during cross programme comparison due
to timing and crisis driver differences.

Evaluation question(s)
Have activities (e.g. fund raising and disbursements) taken place in a timely manner?
Were resources adequate?
Were the EFSF/ESM financial assistance activities consistent with the prime objective?
Did programmes target effectively the financing needs of a country and account for the specific
requirements of bank recapitalisation?
Indicators
Level/intensity of drivers that led to the need for financial sector repair
Evolution of private sector deposits, change in CET1 capital, change in net NPLs, change in the
leverage ratios,change in banks’ gross non-performing loans and change in banks’ operating profitability
before and after the initiation of the programme
Comparison between the initial planned envelopes vs. actual funding needs in the financial sector
Percentage of public and private sector involvement in bank recapitalisation
Time lag between the disbursement of financial assistance and the approval of state aid restructuring
plans

32

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Exit strategies
Purpose

Scope

To study the circumstances surrounding
the exit from programmes and how
programme countries prepared for them;
what strategies, including possible
follow-up arrangements, were discussed
with the institutions? Which criteria
affected the choices made? Finally, to
review the role of technical assistance for
the exit preparations.

Clarifying the process for defining exit strategies for the five
programme countries. Searching evidence on the timing
and key contents of the strategies as well as the roles
played by various stakeholders in initiating the process of
definition.

Type of data

Method(s) used

Qualitative and quantitative

QUAL-QUANT-QUAL strategy. Following interviews,
indicators for the quality of market access were identified
(issuance activity and secondary market conditions). Finally,
document analysis and the Board survey was used to
corroborate views collected from the interviews.

Sources

Problem/bias

Quantitative: Market data for indicators
of market access (Bloomberg, Dealogic),
Debt management offices.

Issuance patterns of non-programme countries were not
compared with those of the five programme countries.
Diverse definitions of cash buffers in documents
complicated the analysis.

Qualitative: Interviews, programme
reviews and evaluations, annual reports
of DMOs, Board survey.

It was not possible to assess the role of the programme
reviews for exit as was initially hoped.

Evaluation question(s)
What were the major factors affecting successful implementation?
Does the lending framework set out an appropriate creditor position, including on proportionality and
the ability to monitor and assert rights?
How has the need for possible follow-up arrangements been assessed?
Indicators
Date of discussions on exit strategies, follow-up arrangements
Maturity structure prior to and following extensions
Volume and maturity of new issuances and bid-ask spreads
Additional comment
The analysis approached issuance activity around programme start and exit simultaneously,
contributing also to the timing of request analysis. Analysis involved inputs from three investigators
working largely separately.

Monitoring and enforcement of repayment ability
Purpose

Scope

To assess the adequacy of the ESM’s
capacity to monitor and enforce
repayment by programme countries.

ESM legal framework.

Type of data

Method(s) used

Qualitative

An assessment of the legal instruments available under
the ESM Treaty, guidelines and loan documentation as well
as of the Early Warning System; relevant EU law; Board
surveys, staff surveys and interviews with programme
country representatives and partner institutions.

Sources
Qualitative: legal and official documents.
Evaluation question(s)
Does the lending framework set out an appropriate creditor position, including on proportionality and
the ability to monitor and assert rights?
Indicators
Confidence in capacity to monitor
Confidence in capacity to control
33

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Internal processes and role of country team coordinators (CTCs)
Purpose

Scope

To describe and assess the internal
process for ESM programme-related
policy input, review and coordination of
country teams’ work, as a measure of
ESM efficiency.

The desk study summarises key findings regarding the
internal process for programme-related policy input and
review based on a review of the existing written guidelines,
an internal survey, and interviews with Country Team
Coordinators and ESM Management Board Members.

Type of data

Method(s) used

Qualitative – Interviews, internal surveys
and document review

QUAL-QUAL strategy. Mainly empirical analysis:
Examination of written guidance and semi-structured
interviews with current Country Team Coordinators and
select Members of the Management Board. Review of the
internal survey and the intervention logic.

Sources

Problem/bias

Qualitative: Interviews with Country
Team Coordinators and ESM
Management Board Members

Only the current ESM staff and management members
were interviewed.

Evaluation question(s)
What measures were taken to ensure that resources were used efficiently?
Indicators
Adequacy of existing guidance for policy work by country team coordinators
Additional comment
The existing internal process for policy input and review with regard to the work of CTCs was found
broadly appropriate and in line with the ESM size and current mandate.

Cooperation
Purpose

Scope

To assess the evolution of the EFSF/
ESM’s involvement in programme work
and engagement of with the national
authorities.

EFSF/ESM programmes for Ireland, Portugal, Greece
(second programme), Spain and Cyprus.

Type of data

Methods used

Qualitative – interview summaries and
transcripts

QUAL-QUANT transformation strategy.

Sources

Problem/bias

Qualitative - ESM evaluation interviews

The analysis does not differentiate between quality and
intensity of engagement. A specific analysis on EFSF/
ESM cooperation with national authorities was retained for
the report as it supported a broader perception from the
interviews with the national authorities.

Nvivo experimental analysis tool extracted respondents’
sentiment based on a standard linguistic examination of
text (interview summaries and verbatim transcripts of the
interviews) specifically coded for a number programme
related cooperation themes. Then automatic filters were
applied, looking at the sentiment of words in isolation
regardless of context. The filters categorised text by “very
negative”, “moderately negative”, “moderately positive”
and “very positive”. The percentages were calculated
based on the number of words without assigning specific
weights and aggregated by programme country.

The sentiment autocoding tool of the Nvivo software
package is an experimental tool. For more information,
please see: http://help-nv11.qsrinternational.com/desktop/
concepts/How_auto_coding_sentiment_works.htm
Evaluation questions
What was the nature of relationships with the relevant local, European and international authorities and
bodies, and what could be improved?

34

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

3.3

Analysis strategy for interviews and surveys

Coding and analysis of interviews
The Terms of Reference prescribed full interlocutor confidentiality, which was closely preserved
in reporting. Summaries and transcripts of the interviews were uploaded to a common secure
database accessible only to team members and coded for specific themes. Coding corresponds
to cataloguing or indexing of data in a hierarchical thematic structure.
In order to analyse the interviews and surveys, the evaluation team used a recognised qualitative
data analysis software (Nvivo 11 Plus for teams) that “allows researchers to organize and analyse
a wide variety of data, including but not limited to documents, images, audio, video, questionnaires and web/social media content”1. The software enabled simultaneous access to sources
and tools by team members.
Coding was advised by two training sessions. Initially coding was experimental with some coordination for the themes relevant for the evaluation questions. A structured scheme (parent and child
nodes) was developed gradually, while allowing investigators to create personal nodes when the
existing structure was not adequate to express some key messages from the interviews.
Coding took place as a team exercise, to enable efficient use of limited time and limit personal
bias in interpretation. Figure 5 shows how many investigators coded for each top level node.

1

Edhlund, B., Mcdougall, A. (2016).
35

Source: ESM

36

Technical assistance

Success elements

Strategic positioning

Repayment ability

Relevance of country programme

Programme length

Programme exit

Programme description

Programme cycle

Pricing strategy

Post programme period

Interesting quotes

Instrument choice and timing

Financial sector repair

Envelope size

DSA

Disbursements

Cooperation (looking back)

Contribution to sustainability

Technical assistance

Success elements

Strategic positioning

Repayment ability

Relevance of country programme

Programme length

Programme exit

Programme description

Programme cycle

Pricing strategy

Post programme period

Interesting quotes

Instrument choice and timing

Financial sector repair

Envelope size

DSA

Disbursements

Cooperation (looking back)

Contribution to sustainability

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Figure 5
Number of users coding
12

10

8

6

4

2

0

Note: DSA stands for Debt Sustainability Analysis.
Source: ESM

Following an initial set of interviews, the coding scheme was enriched but the coders were also
allowed to create additional nodes in personal folders as they found relevant topics. One of the
two interview team leaders coded each interview.

Figure 6
Number of sources coded

90

80

70

60

50

40

30

20

10

0

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Once all but a small number of interviews were analysed, the core team assessed the personal
nodes and consolidated the number of nodes as appropriate. The remaining personal nodes were
kept in a separate folder. This coding exercise can be characterised as vertical: each source was
entirely coded. Figure 6 shows the number of sources coded for each top level node.

Figure 7
Number of coding references
600
500
400
300
200
100

Technical assistance

Success elements

Strategic positioning

Repayment ability

Relevance of country programme

Programme length

Programme exit

Programme description

Programme cycle

Pricing strategy

Post programme period

Interesting quotes

Instrument choice and timing

Financial sector repair

Envelope size

DSA

Disbursements

Cooperation (looking back)

Contribution to sustainability

0

Source: ESM

Additionally, horizontal coding took place: all the sources were coded at a time for one node
which was considered as one of the main themes. A number of investigators received a task to
code all the sources for his/her theme. Figure 7 shows the number of coding references for each
top level node.
The analytical software incorporates an audit trail. It also provided two experimental features:
autocoding and sentiment analysis. These were used as support tools, while taking into account
their mechanistic nature. Autocoding helped identify key words and concepts from the material.
Autocoding for sentiment was used to analyse cooperation-related views based on interviews.
These results were discussed against the interviewers’ perceptions in the drafting process. One
chart was used in the report.
The report drafting was conducted by a smaller core team to reduce coordination challenges.
When drafting the report, the team used topic-specific nodes to access interview material,
board survey results, and a variety of programme documentation. This allowed for a documented
cross-sectional view on the material. Evaluators queried the database to a varying degree for
further cross-evidence. Some academic literature was used to corroborate findings.

37

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Treatment of surveys
As it has been underlined above, the evaluation team conducted a Board survey based on openended questions which was an add-on to the initial design to ensure relevance. This survey
introduced non-programme country directors’ views and was analysed in a similar manner to the
interviews. The responses were uploaded to the database, coded and analysed as part of the
thematic inquiry of the interview material. In addition, the responses were summarised question
by question in a specific note.
An exploratory ESM management and staff survey was conducted prior to the installation of the
analytical tool. It comprised both closed and open ended questions. It was summarised by two
team members and the results were used as guidance to refine the general approach and definition of operational evaluation questions.

4.	Conclusion
The Terms of Reference set the mandate and the main objectives of the evaluation exercise. This
note summarised the evaluation process and methodology, with the aim of showing that the
Terms of Reference have been adhered to.
The Independent Evaluator took position on each of the evaluation questions addressed. She
furthermore led the work on recommendations. An external advisor and consultants oversaw the
technical process with comments and guidance. The distinctiveness of the EFSF/ESM activities
required a team composed of various experts from the ESM. A mixed method approach using
quantitative and qualitative data analyses was necessary to address the evaluation questions.
Many findings are also broadly aligned with findings of other relevant evaluations.
The team faced some time and evaluability challenges, but overall the mixed methods approach
and a large sample of respondents provided a sufficiently broad pool of evidence to conclude the
exercise with confidence about its relevance and credibility. The conclusions and recommendations are transferable to similar situations. The findings are directly relevant considerations for
potential future euro area programmes. Although most other regional financing arrangements
beyond the euro area do not operate in a fixed exchange rate environment, they would likely face
similar challenges due to multiple stakeholders and cooperation requirements.
Finally, this exercise identified the need for future evaluations, namely of the Early Warning System
and the EFSF/ESM programmes for Greece. In the ESM Board consultation future work was also
advised to take account of the social impacts of EFSF/ESM type of adjustment programmes.

38

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Sources
Creswell, J. W. (1999), “Mixed-method research: Introduction and application”, in C. Ciznek
(Ed.), Handbook of educational policy, San Diego, CA: Academic Press.
Creswell, J. W., Plano Clark, V. L. (2006), “Designing and Conducting Mixed Methods Research”,
Thousand Oaks, CA: Sage.
Edhlund, B., Mcdougall, A. (2016), “Nvivo11 Essentials”, February 2016.
European Commission (2012), “Impact evaluations counterfactual”, October 2012.
Goldsbrough, D. (2016), “Use of Evidence in Evaluations: example of prolonged use of IMF
resources”, Internal ESM presentation, September 2016.
Guest, G et al. (2016), “How Many Focus Groups Are Enough? Building an Evidence Base for
Nonprobability Sample Sizes”, Field Methods, Volume 29, Issue 1, February 2017.
Lance, P.M., Guilkey, D. K., Hattori, A., Angeles, G. (2014), “How Do We Know If a Program
Made a Difference? – A guide to statistical Methods for Program Impact Evaluation”, Chapel Hill,
North Carolina: MEASURE Evaluation, May 2014.
Morse, J. (1991), “Approaches to qualitative-quantitative methodological triangulation”, Nursing
Research, Volume 40, Issue 2, pp.120-123, March/April 1991.
Onwuegbuzie, A. J., Combs, J. P. (2011), “Data Analysis in Mixed Research: A Primer”,
International Journal of Education, Volume 3, No. 1: E13, April 2011.
Onwuegbuzie, A. J., Combs, J. P. (2011), “Describing and Illustrating Data Analysis in Mixed
Research”, International Journal of Education, Volume 2, No. 2: E13.
Organization for Economic Co-operation and Development (1998), “Methodological Note,
Indicator Methodology Sheets”, OECD, DCD/DAC(98)6/ADD.
Rao, V., Woolcock, M. (2003), “Integrating Qualitative and Quantitative Approaches in Program
Evaluation”, World Bank, June 2003.
Silver, C. (2011),”Moving on Beyond Initial Coding Using CAQDAS Packages”, web publication,
https://www.surrey.ac.uk/sociology/research/researchcentres/caqdas/support/analytictasks/
moving_on_beyond_initial_coding_using_caqdas_packages.htm
Tashakkori, A., Teddlie, C. (1998), “Mixed methodology: Combining qualitative and quantitative
approaches”, Thousand Oaks, CA: Sage.
Teddlie, C., Tashakkori, A (2009), “Foundations of Mixed Methods Research”, Thousand Oaks,
CA: Sage Publications.
United Nations Evaluation Group (2016), “Norms and Standards for Evaluation”, New York:
UNEG, June 2016.
Vijayendra, R., Woolcock, M. (2003), “Integrating qualitative and quantitative approaches in program evaluation. In The impact of economic policies on poverty and income distribution”, ed.
Francois J. Bourguignon and Luiz Pereira da Silva. New York: Oxford University Press.

39

C. Intervention logic

The framework described in this note shows how EFSF/ESM resources and activities are understood by the evaluation team to be combined to deliver the results (outputs and outcomes) in
the ESM institutional setting, and how this process may be affected by various external factors.
Purpose of intervention logic
Intervention logic is a tool used in the conceptual phase of the evaluations, which in this case
took place in the autumn of 2016. The tool helps specify how and why the activities evaluated
produce results. Evaluation needs a contextual framework that enables critical thinking while
clarifying the process reviewed, its objectives and the underlying assumptions. While it simplifies
the operation and the key relationships, it should reflect the complexity of the real world object
of evaluation in a useful way.
The process of developing the intervention logic helped maintain a consistent evaluation process,
particularly as a relatively large team was involved. It facilitated building a shared understanding
of the scope and objectives and supported onboarding of the team established for the exercise.
The purpose of developing the intervention logic was also to show how EFSF/ESM activities
contributed to the overall European and global crisis resolution and economic governance activities. Attribution is not straight forward, given the joint-nature of the country programmes and the
complexity of the economic and financial policy framework in the euro area.
The core concepts introduced are explained in Box 1.

Box 1: Core concepts
Inputs: all the resources that contribute to the production and delivery of outputs. These resources
are “what we use to do the work”. They include financing, personnel, equipment, etc.
Activities: the processes or actions that use a range of inputs to produce the desired outputs and
ultimately outcomes. Basically, activities describe “what we do”.
Outputs: the final products, or goods and services produced for delivery. Outputs may be defined
as “what we produce or deliver”.
Outcomes: results in the medium-term for specific beneficiaries that are the consequence of
achieving specific outputs. Intended outcomes are “what we want to achieve”, but there may
also be unintended outcomes. Medium-term could be understood as three-to-five or more years.
Impacts: the results of achieving specific outcomes, such as maintaining the financial stability or
the integrity of the euro area, which emerge over a lengthy period of time – generally considered
to extend over a decade – after the conclusion of a financial assistance programme.

40

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

The main tool adopted by the team was the Kellogg model for intervention logic.1 In addition, the
evaluation office of the European Investment Bank supported the team by challenging the initial
versions. The team also consulted ESM management.
It is recognised that there is a number of schools of thought on the use of the conceptual frameworks such as the intervention logic and results chains, and that these can be developed to varying extents. The ESM team considered that the visual presentations developed in brainstorming
and group discussions help communicate key elements of the object.
High-level dimensioning
As European crisis resolution mechanisms, the EFSF/ESM are deeply embedded in the
institutional framework of European and national decision-making processes for financial
assistance. Programmes are discussed at the political level in the Eurogroup and the Eurogroup
Working Group. For the EFSF, these two bodies also represent the same countries that formally adopt financial assistance programmes and decide upon disbursements. For the ESM, the
relevant decisions are formally taken by the ESM Board of Governors and Board of Directors.
Decisions in the EFSF/ESM governing bodies require appropriate authorisation at the national
level, which often involve national parliaments. Figure 1 illustrates the high level framework.
The role of the EFSF/ESM in designing and implementing programmes has evolved over
time. The initial role of the EFSF as a pure provider of financing has been significantly broadened.
With the creation of the ESM, responsibilities for different programme elements were allocated
as follows: The ESM is in charge of the overall programme governance and the design of financial
assistance. The negotiation of policy conditionality is attributed in the ESM Treaty to the European
Commission in liaison with the ECB. In addition, the IMF is involved in programme conditionality
and financing whenever possible. There is nevertheless no obligation for the decision-making
bodies to follow the IMF’s policies. The EFSF and ESM cooperate closely with authorities in beneficiary Member States or other relevant European institutions, such as the Single Supervisory
Mechanism, during the programme design, implementation and post-programme phases.
The EFSF/ESM strategy to provide financial assistance is guided by their mandates and
takes into account international and national factors. These mandates are to provide financial assistance to Member States to safeguard the financial stability of the euro area and its
Member States. The EFSF and ESM provide liquidity support to a country while it is undergoing
reforms to address the weaknesses that led to the loss of market access at affordable rates.
The EFSF/ESM strategy in addressing a country’s weaknesses and safeguarding financial
stability operates against the backdrop of either an adverse or supportive international
economy. It complements other initiatives taking place at European and national level to address
a crisis situation, in particular the role of monetary policy in stabilising the euro area. The support
is geared towards the domestic factors that caused a programme country’s economic and financial vulnerability. In the European crises, problems often emerged in the banking system.

1

W.K. Kellogg Foundation: Logic Model Development Guide, January 2004.
41

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Figure 1
High level intervention logic of EFSF/ESM programmes
Strategies

Assumptions

Financial assistance with conditionality buys time for
reform implementation. Euro area institutional reforms,
national fiscal consolidation, structural reform and
financial sector repair to restore confidence.

Structural reforms and fiscal consolidation leads to
sustainable public debt levels and regenerates market
confidence, growth potential; Three-year programme
is long enough to return to market financing. Solidarity
while no debt mutualisation, nor permanent transfers.

Influential factors
International economy and financial
system.
Monetary and fiscal policy stance
in the international economy.
Global financial safety net and role of
the IMF.
Regulatory environment.
Degree of private risk sharing.
Economic and banking sector linkages.
Political context.

Problem or issue
Lost market access at affordable rates.
Threat to the financial stability of euro
area and its Member States (contagion).

Community needs/assets
Ensure continued private and public
financing of the economy and
international creditors.
Guarantee a stable and predictable
operating environment.
Ensure the integrity of euro area.

Desired results
(output, outcomes and impact)
Sustainable debt service burden
and structure.
Financial stability.
Correction of imbalances.
Restoration of sustainable economic
growth.

Source: ESM staff application of Kellogg model

It is important to understand the role of the underlying assumptions (Figure 1) shaping the evaluation. These are aspects of the operating environment that are taken for granted.
In this model, the existence of influential factors which are out of EFSF/ESM control implies that
there can be unintended consequences.
Having agreed on a high-level intervention logic, the team considered more specifically the processes delivering outcomes with different timeframes.
Operational logic
The EFSF/ESM strategy in providing financial assistance follows an intervention logic
comprising three key elements:
• First, a number of inputs (some of which are external) allow the ESM to constitute a firewall
for the euro area to safeguard financial stability;
• Second, each individual financial programme involves various activities that the ESM develops
in line with its mandate;
• Finally, the combination of these inputs and activities leads the ESM to deliver outputs, to
achieve outcomes, and to contribute to a (positive) impact on the sustainability and economic
integrity of supported member states and the euro area as a whole.
This chain of elements can be called the intervention logic of the ESM in providing financial assistance. Articulating the different elements of the intervention logic allows us to conceptualise or
capture the ESM financial assistance strategy and further develop a results chain.

42

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

As is shown in Figure 2, the strategy consists of planned inputs and activities that are expected
to lead to expected outputs. The strategy is geared towards achieving the desired policy outcomes in the short (one-to-three years) and medium term, which in turn are expected to achieve
the desired long-term impact(s) of the programme.
Inputs that the ESM can use in its strategy consist of:
1. financial resources (paid-in and callable capital; sovereign guarantees for the EFSF),
2. human resources (expertise) and management systems,
3. governance framework,
4. collaboration with partner institutions, and
5. political context.
ESM activities include:
1. designing and negotiating financial assistance,
2. mobilising funding and managing liquidity,
3. executing lending transactions,
4. monitoring repayment capacity, and
5. providing technical assistance to programme countries.
Concrete outputs of the ESM strategy can include:
1. financing proposals, MoUs, financial assistance agreements,
2. ESM market financing (bonds, bills, credit lines),
3. disbursements (bonds and cash) under lending programmes,
4. EWS reporting, and
5. advice and reports to Members to support repayment capacity.
In addition, the European Commission’s activity of reviewing compliance with policy conditionality
(resulting in compliance reports) forms a framework that supports EFSF and ESM programmes
and the desired outcomes.

43

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

As for the outcomes, we can consider in the short term:
1. using programme financing to close financing gaps,
2. correcting fiscal and structural policies,
3. stabilising the banking sector,
4. stabilising depositors’ confidence, and
5. reducing contagion to the euro area member states.
And, in the medium term:
1. economic rebalancing (fiscal, external),
2. public and private deleveraging,
3. improved competitiveness,
4. restored confidence, regained market access,
5. adequate governance of economic policy, and a
6. sound banking system, adequate supervisory and resolution frameworks.2
Regarding the potential impacts, considered in the long term, they relate to:
1. sustainable growth and job market,
2. financial stability,
3. sustainable public finances,3 and the
4. integrity of the euro area.

2
3
44

Including recapitalisation, asset quality, and business plans.
Including robust expenditure controls, sound revenue policies, and contained contingent liabilities.

Acute financing need(s)

•

•
•

•

•

•

Relevance

Paid-in
and callable
capital
EFSF guarantees
Governance
framework
Management
systems
Internal expertise
Collaboration with
partners

Source: ESM staff application of Kellogg model

Efficiency

Development and
maintenance of
assistance
capacity
Funding and
liquidity
management
Lending or
recapitalisation:
Design and
negotiation of
financing terms
and instruments
Public outreach
Review
programme
implementation
Technical
assistance

Activities

* Shared responsibility / participation
*External input

•
•

•

•
•

•

Inputs

•
•

•

•

•

•

•

•

•

Financing
proposal
Financial
Assistance
Facility
Agreement
Issuance to
market
Disbursements
(cash and bonds)
Design of debt
structure
Assessment of
beneficiary
repayment
capacity (EWS)
Programme
monitoring
reports
MoU
Advice for
capacity building

Outputs

DELIVER

Programme financing
closes financing gaps
and improves debt
structure
Stabilised banking
sector and restored
confidence of
depositors
Demand balancing
Reduce spillovers to
euro area members

Collaboration

•
•

•

•

Immediate Crisis
Management

Short term

Restore confidence

Declining debt path
and refinancing risks
Rationalised spending
and revenue policies
Improved
competitiveness
Restructured banks
Market confidence and
credible market
financing
Adequate governance
of economic policy
Adequate supervisory
and resolution
framework

Public Policy
reforms

Medium term

Effectiveness

•

•

•
•

•

•

•

Outcome

ACHIEVE

Figure 2
Operational intervention logic in the financial assistance programme context

•

•
•
•

•

•

•

Sustainable public
finances
Banks finance the
economic activity
efficiently
Major rigidities of the
economy removed
Sustainable growth
Political unity
Investors trust the integrity
of the euro
Stable and predictable
operating environment

Euro area
Financial Stability

Impact (Long term)

CONTRIBUTE

•

•
•

EURO: threat to stability
Country: loss of market
access at realistic rates
Risk to the integrity of the
Euro area

At stake: problem-solving

SOLVE

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

45

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

The ESM strategy in providing financial assistance is designed to lead to defined outcomes. The final programme impact is a result of this combined with the activity of other
partner institutions and programme implementation by member state authorities. This
presupposes that the underlying programme assumptions were correct, that the impact of influential factors was assessed correctly, and that these factors did not change. If any of the external
factors changes, this will affect the expected impact and the ESM will need to revise its strategy
to mitigate and adjust for this so that expected outcomes are still achieved. In the broader programme context, this should happen in coordination with other institutions, the authorities, and
in line with the political decision-making process.
The desired outcomes and impact of EFSF/ESM programmes derive directly from its
mandate of safeguarding financial stability. The outputs from operational activities include:
• Safe investment of the paid-in capital and a robust procedure to call additional capital, if
needed, from the member states. This financial backing supports a high credit rating and
therefore a capacity to raise funds from the capital market. A robust guarantee structure similarly supports the EFSF’s capacity to mobilise funding at affordable rates.
• Expertise enables the EFSF/ESM to design financial assistance based on operational instruments and a lending framework endowed by the governance framework. These are formalised
in a financing proposal and financing terms embodied in a financial assistance agreement.
A Memorandum of Understanding on policy conditionality, negotiated by the European
Commission, sets a path for agreed reforms that materialise as outcomes of the programme.
• The two institutions’ lending framework and liquidity management enable timely disbursements to address the immediate financing needs and smoothening of a beneficiary Member’s
debt structure.
The combination of the above outputs allows the programme country to close its financing gap,
as well as design and execute strategies that increase confidence in banks. In essence, the
financial assistance buys time for the programme country to implement necessary reforms and
therefore smoothens the adjustment need.
In the short term, desirable outcomes centre on crisis management issues, for example filling
the financing gap and starting the gradual correction of fiscal and structural policies. In many
cases, the financing strategy also aims to stabilise the banking sector by allowing the programme
country to ensure that viable banks can receive the required capital support and thus stem potential deposit outflows.

46

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

The medium-term outcomes focus on putting different elements of public policy in the concerned programme country back on a sustainable track. This can reflect, for example, reforming the fiscal framework to generate surpluses and increase competitiveness, which help put
debt on a downward trajectory, and containing contingent liabilities from the financial sector. To
achieve the latter, market confidence needs to improve, for example, by shoring up supervision
and resolution frameworks, which gradually reduces the contingent liabilities of the government.
Monitoring a programme country’s repayment capacity takes place in congruence with the
European Commission’s post-programme surveillance. The ESM’s Early Warning System monitors the reduction in the programme country’s refinancing risks and is maintained when a programme country has outstanding liabilities towards either the EFSF or ESM. In addition, technical
assistance by the ESM or the partner institutions may include advice that supports administrative
capacity and reforms that ultimately improve the programme country’s repayment capacity.
In the longer term, the main impact of EFSF/ESM programmes and the programme partners'
activities will be to put the economy as a whole back on a sustainable path, building up buffers
and securing long-term stability. This includes removing rigidities that could hinder the economy
from adapting to change and could thus lead to a renewed build-up of imbalances.

47

D. Vulnerability analysis

The vulnerability analysis employs the comprehensive scorecard for assessing sovereign vulnerabilities developed by Lennkh et al. (2017). It is based on (i) a wide set of indicators categorised
along six dimensions, (ii) a scoring system based on thresholds provided by the literature where
available or determined by the percentiles from the historical distribution of a pool of OECD and
non-OECD EU countries, (iii) an aggregation scheme, underpinned by correlation and principal
component analysis (PCA) and (iv) a system of traffic lights.
Sovereign vulnerability is assessed across the following six dimensions: (i) government borrowing needs, conditions and debt structure; (ii) economic strength; (iii) fiscal position; (iv) financial
sector and other contingent liabilities; (v) institutional parameters, and (vi) private leverage, credit
and real estate. Each dimension includes a set of indicators. The selection of the indicators is
based on the related literature, while taking into account results of the correlation and principal
component analysis.
For each indicator, we define three time-invariant thresholds that allow per country and point
in time to assign a 1 to 4 score. Where available, the thresholds for the scoring of the indicators
were taken from the existing literature. When this was not possible, the thresholds were set
in line with quartiles of the historical distribution of OECD and EU countries in 2002-2016. The
quartiles can be interpreted as follows: If a country is among the 25% of best-performers, it is
assigned the score 4 (most resilient); countries in the second quartile are assigned the score 3;
third quartile 2; and bottom quartile 1 (most vulnerable).
The aggregate scores are calculated as a weighted average of the scores of individual indicators (Table 1). The weights are based on correlation analysis, PCA, literature, and expert judgement. First, based on correlation analysis, we assign higher weight to indicators which are more
strongly correlated to proxies of financial or economic stress, such as future GDP growth rates,
rating actions or changes in government bond spreads. Second, based on the PCA, we identify
groups of indicators which follow similar trends and are thus deemed to explain the same underlying vulnerability. We then lower the weights of individual indicators in these groups to avoid
overrepresentation of vulnerabilities captured by more individual indicators.
To improve visualisation, the quantitative assessment is complemented with a system of traffic
lights. Given the uncertainty surrounding the calculations as well as limitations of any one-sizefits-all approach, the analysis should not be interpreted as an exact numerical exercise. Therefore,
a country can be classified into one of four broad vulnerability categories: low vulnerability (green,
overall score ≥3), moderate vulnerability (yellow, 2.5-3.0), elevated vulnerability (orange, 2.0-2.5)
and high vulnerability (red <2). The three cut-off values are set in a way that they broadly correspond to the following average ratings of the sovereigns assigned by Fitch, Moody’s and S&P
over the 2007-16 time period: the ESM’s General Eligible Asset List (AA-), the ECB’s Credit
Quality Steps 1&2 (A-), and the agencies’ investment-grade status (BBB-). We assume that the
ratings were on average accurate, albeit sometimes with a delay. The cut-off values of the traffic lights are nevertheless somewhat more conservative. One can argue that a country has to
outperform the average score in the respective rating range to qualify for the corresponding
vulnerability category.

48

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Table 1: Overview of the approach and the aggregation scheme
Dimension

Borrowing
needs,
conditions and
debt structure

Economic
strength

Fiscal position

Financial sector
and other
contingent
liabilities

Institutional
parameters

Private leverage,
credit and real
estate

Weight

25%

20%

15%

10%

15%

15%

Indicator

Weight

Gross financing needs (% of GDP)

20.0%

Change in gross financing needs (p.p.)

10.0%

10-year bond yield spreads to Germany (p.p.)

15.0%

10-year bond yield volatility (std)

15.0%

Credit rating and outlook (avg Moody's, S&P and Fitch)

0.0%

Share of short-term debt (%)

15.0%

Change in share of short-term debt (p.p.)

10.0%

Share of debt held by non-residents

0.0%

Share of foreign currency-denominated debt (%)

15.0%

Potential GDP growth (%)

15.0%

Real GDP growth (%)

10.0%

Volatility of real GDP growth (std)

10.0%

GDP per capita (PPS thousands)

15.0%

WEF Competitiveness Index

10.0%

Inflation volatility (std)

5.0%

Current + capital account balance (% of GDP)

15.0%

Unit labour cost (%)

15.0%

Unemployment rate (%)

5.0%

Government debt-to-GDP ratio (%)

10.0%

Government debt-to-government revenue ratio (%)

10.0%

Change in government debt-to-GDP ratio (p.p.)

10.0%

Net debt (% of GDP)

10.0%

Interest-growth differential (p.p.)

10.0%

Primary balance (% of GDP)

15.0%

Structural balance (% of pot. GDP)

15.0%

Longest period of positive primary balance (years)

10.0%

Highest average structural balance over 8 years (% of GDP)

10.0%

ESM's Bank Viability Index

33.3%

Increase in ageing costs (% of GDP)

16.7%

Stock of government guarantees (% of GDP)

16.7%

Net international investment position (% of GDP)

33.3%

WB Governance Effectiveness

15.0%

WB Regulatory Quality

15.0%

WB Rule of Law

15.0%

WB Doing Business Rank

25.0%

Commission's fiscal rule index

15.0%

OECD EPL

0.0%

OECD PMR

0.0%

TI Corruption perception index

15.0%

Non-financial corporations' debt (% of GDP)

20.0%

Household debt (% of GDP)

20.0%

Credit growth (%)

15.0%

Credit flow to non-financial sector (% of GDP)

15.0%

House price growth - nominal compensation growth (p.p.)

30.0%

Note: The ESM’s Bank Viability Index assesses the fundamental strength of euro area banks and thus captures the potential risk posed by its banking sector for the respective sovereign.

49

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Case study: Vulnerabilities prior to and after the EFSF/ESM programme intervention
This study shows the development of sovereign vulnerabilities for the five programme countries
for the period before and since the inception of the programme. Due to annual data, and in the
case of Cyprus the delay in negotiations, the first year of the intervention differs in some cases
from the year of the request (Table 2).

Table 2: Dates of request for EFSF/ESM financial assistance
Country

Programme request

Year of intervention (analytical purposes)

Ireland

December 2010

Portugal

May 2011

2011

Greece (2nd program)

February 2012

2012

Cyprus

June 2012

2013

Spain

June 2012

2013

2011

As can be observed in the charts below, the developments in terms of vulnerability before the
EFSF/ESM intervention were unfavourable in all the five countries according to most vulnerability dimensions, and improved following the programme intervention. The two exceptions in the
post-intervention period are the borrowing conditions for Greece, driven mostly by the increasing
share of short-term debt, which were not offset by a decline in government bond yields, and
Spain’s slightly weakened institutional parameters as measured by the World Bank.

Figure 1
Evolution of Vulnerabilities
Euro area 2016

Ireland

IE 2011-16

1 Govt. borrowing needs,
conditions and debt structure

2.9

2 Economic strength

3.3

3 Fiscal position

3.1

4 Financial sector and other
contingent liabilities

2.2

5 Institutional parameters

3.5

6 Private leverage,
credit & real estate

1.2
1.0

2.0
Euro area 2016

Portugal

3.0
PT 2008-11

4.0
PT 2011-16

1 Govt. borrowing needs,
conditions and debt structure

1.7

2 Economic strength

2.3

3 Fiscal position

2.1

4 Financial sector and other
contingent liabilities

2.0

5 Institutional parameters

2.4

6 Private leverage,
credit & real estate

2.1
1.0

50

IE 2008-11

2.0

3.0

4.0

Figure 1 (continued)
Evolution of Vulnerabilities
Greece

Euro area 2016

EL 2008-12

EL 2012-16

6 Private leverage,
credit & real estate

2.1
1.0

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017
2.0
3.0
4.0

Figure 1 (continued)
Evolution of Vulnerabilities
Euro area 2016

Greece

EL 2008-12

EL 2012-16

1 Govt. borrowing needs,
conditions and debt structure

1.5

2 Economic strength

2.0

3 Fiscal position

2.4

4 Financial sector and other
contingent liabilities

1.8

5 Institutional parameters

1.3

6 Private leverage,
credit & real estate

2.1
1.0

2.0
Euro area 2016

Spain

3.0
ES 2008-13

4.0
ES 2013-16

1 Govt. borrowing needs,
conditions and debt structure

2.1

2 Economic strength

2.5

3 Fiscal position

1.8

4 Financial sector and other
contingent liabilities

2.3

5 Institutional parameters

2.2

6 Private leverage,
credit & real estate

1.9
1.0

2.0
Euro area 2016

Cyprus

3.0
CY 2008-13

4.0
CY 2013-16

1 Govt. borrowing needs,
conditions and debt structure

2.0

2 Economic strength

2.0

3 Fiscal position

2.6

4 Financial sector and other
contingent liabilities

2.3

5 Institutional parameters

2.2

6 Private leverage,
credit & real estate

2.1
1.0

2.0

3.0

4.0

51

E. Timing of request
Figure 1
CY: Programme approved 8 May 2013
Portfolio Inflows

700
600
500
400
300
200
100
0

4
3
2
1
0
-1
-2
-3

25
20
15
10
5
0
-5
-10
-15
Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Spreads

Spread to Bund (bp)

4Q moving sum (in € billion)

NPLs

Capital
6

18

50

5

15

20

40

4

12

15

30

3

9

10

20

2

6

5

10

1

3

0

0

0

0

NPLs (% Total loans, RHS)

Expert indications for
potential early request

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

60

25

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

30

NPLs (in € billion)

CET1 (in € billion)

12 months before
the formal request date

Sources: Haver, ECB, European Commission, SNL, Bankscope

52

% GDP (RHS)

CET1 RATIO (%, RHS)

Formal request date

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Figure 2
EL: Programme approved 15 March 2012
Spreads

Portfolio Inflows
20
15
10
5
0
-5
-10
-15
-20
-25

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

40
30
20
10
0
-10
-20
-30
-40
-50

3,000
2,500
2,000
1,500
1,000
500
0

Spread to Bund (bp)

4Q moving sum (in € billion)

NPLs

% GDP (RHS)

Capital

100

40

80

32

60

24

40

16

20

8

0

0

35
30
25
20
15
10
5
0

NPLs (in € billion)

21
18
15
12
9
6
3
0
Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

48

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

120

CET1 (in € billion)

NPLs (% Total loans, RHS)

Expert indications for
potential early request

12 months before
the formal request date

CET1 RATIO (%, RHS)

Formal request date

Sources: Haver, ECB, European Commission, SNL, Bankscope

Figure 3
ES: Programme approved 24 July 2012
Spreads

Portfolio Inflows

600
500
400
300
200
100
0

15
10
5
0
-5
-10
-15
-20
Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

150
100
50
0
-50
-100
-150
-200

Spread to Bund (bp)

4Q moving sum (in € billion)

NPLs

% GDP (RHS)

Capital

210

12 250

15

175

12

35

10 200
8
150
6
100
4
50
2

0

0

0

0

105

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

70

NPLs (in € billion)

NPLs (% Total loans, RHS)

Expert indications for
potential early request

9
6
3
Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

140

CET1 (in € billion)

12 months before
the formal request date

Sources: Haver, ECB, European Commission, SNL, Bankscope

CET1 RATIO (%, RHS)

Formal request date
53

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

Figure 4
IE: Programme approved 22 December 2010
Spreads

Portfolio Inflows
150
125
100
75
50
25
0
-25
-50
-75
-100

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

300
250
200
150
100
50
0
-50
-100
-150
-200

1,200
1,000
800
600
400
200
0

Spread to Bund (bp)

4Q moving sum (in € billion)

NPLs

% GDP (RHS)

Capital

80
70
60
50
40
30
20
10
0

40
35
30
25
20
15
10
5
0

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16
NPLs (in € billion)

CET1 (in € billion)

NPLs (% Total loans, RHS)

Expert indications for
potential early request

21
18
15
12
9
6
3
0

35
30
25
20
15
10
5
0

12 months before
the formal request date

CET1 RATIO (%, RHS)

Formal request date

Sources: Haver, ECB, European Commission, SNL, Bankscope

Figure 5
PT: Programme approved 27 May 2011
Spreads

Portfolio Inflows

1,400
1,200
1,000
800
600
400
200
0

25
20
15
10
5
0
-5
-10
-15
-20
-25
-30
Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

50
40
30
20
10
0
-10
-20
-30
-40
-50
-60

Spread to Bund (bp)

4Q moving sum (in € billion)
Capital
21
18
15
12
9
6
3
0

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

35
30
25
20
15
10
5
0

NPLs (in € billion)

NPLs (% Total loans, RHS)

Expert indications for
potential early request

18

25

15

20

12

15

9

10

6

5

3

0

0

CET1 (in € billion)

12 months before
the formal request date

Sources: Haver, ECB, European Commission, SNL, Bankscope
54

30

Jan 08
Jul 08
Jan 09
Jul 09
Jan 10
Jul 10
Jan 11
Jul 11
Jan 12
Jul 12
Jan 13
Jul 13
Jan 14
Jul 14
Jan 15
Jul 15
Jan 16
Jul 16

NPLs

% GDP (RHS)

CET1 RATIO (%, RHS)

Formal request date

A crisis prevention tool to support members by
guaranteeing future access to financial assistance
to prevent difficulties in raising funds in the capital
markets when this appears a likely scenario. It comes
in two versions depending on the soundness of the
member’s policies and fundamentals.

EFSF could use these instruments as a complement
to a regular facility, maximising the efficient use
of funds: (1) To purchase bonds in primary market
auctions to support market access and reduce the risk
of a failed auction. (2) To intervene in the secondary
market to support its functioning and price formation in
exceptional circumstances or to create some liquidity.
Available under exceptional financial market disturbance
and risks to financial stability, e.g. when unusually
volatile or unidirectional movements, as judged by the
ECB.

The aim of this last resort instrument is to preserve
financial stability in the euro area as a whole or its
member states by assisting in specific cases of acute
bank failure endangering fiscal sustainability due to
contagion.

Precautionary
programme

Market support
instruments

Direct
recapitalisation of
financial institutions

Assistance can be provided to the credit institution
for the benefit of the ESM member within or outside
a macroeconomic programme.

Help resolving financial distress originating primarily
in the financial sector, and not directly in fiscal
or structural policies. Participate in the public
recapitalisation of systemically relevant financial
institutions to ensure long-term viability and resilience,
having demonstrated the unavailability of privately
financed solutions.

Recapitalisation
of financial
institutions (loan)

(For ESM since
December 2014)

Objective

Assist a member that has significant financing needs
and has largely lost market access or has access to
financing at rates that would endanger the sustainability
of public finances.

Instrument

Loan facility

EFSF main eligibility criteria

ESM main eligibility criteria

Regulatory burden-sharing arrangements
are insufficient to address the capital
shortfall. Institution- or sector-specific or
macroeconomic conditionality and respect of
state aid provisions required.

A systemically relevant institution supervised
by the ECB or is likely to be in breach of capital
requirements and the relevant ESM member
is unable to provide assistance without very
adverse effects on its financing.

For secondary market support: same
conditionality as under the EFSF.

Sustainable public debt and external
position; Absence of major critical bank
solvency problems; track record of market
access.

N/A

For primary market support: compliance with
policy conditionality under a loan facility or
precautionary facility.

Country subject to enhanced surveillance.

Same as under the EFSF but specifying
sustainable general government debt

The ESM member should demonstrate
an inability to meet capital shortfalls via
private sector solutions and recapitalise the
institutions without adverse effects for its
own financial stability and fiscal sustainability.
Ability to reimburse the loan.

Macroeconomic adjustment programme
carrying conditionality that reflects the
severity of weaknesses. Assessment of
financing needs, debt sustainability and
existence of risks to financial stability of the
euro area as a whole or its member states.

Respect of commitments under
Stability and Growth Pact and Excessive
Imbalances Procedure;

Country subject to enhanced surveillance.

Sustainable public debt and external
position; Absence of major critical bank
solvency problems; track record of market
access.

Respect of commitments under the
Stability and Growth Pact and Excessive
Imbalances Procedure;

Restructuring or resolution of financial
institutions with institution specific
conditions including respect for state aid
requirements, and horizontal conditionality
including reforms in supervisory,
corporate governance and domestic laws.

Difficulties caused by exceptional
circumstances risking the financial
stability of the euro area as a whole and
of its member states. Macroeconomic
adjustment programme under appropriate
conditionality.

Used by

Not used

Not used

Not used

Spain (2012)

Ireland (2010),
Portugal (2011),
Greece (2012),
Cyprus (2013)

F. Choice of instrument

55

G. Crisis timeline

EFSF/ESM focused crisis timeline
SELECTED GENERAL AND PROGRAMME EVENTS
Sep 2008

Lehman Brothers declared bankrupt.
Ireland unveils a two-year blanket guarantee to safeguard the deposits and debts at six
financial institutions.

Jan 2009

Standard & Poor’s (S&P) downgrades Greek, Portuguese, Spanish sovereign debt.

Mar 2009

Anglo-Irish bank is nationalised, public capital injection to two Irish banks.
Ireland loses AAA sovereign credit rating.

Apr 2009

Irish emergency budget.

Oct 2009

Greek government announces fiscal deficit estimate at 12.5% of GDP.

Dec 2009

Moody’s downgrades Greece to A1. Fitch downgrades Greece to BBB+.

Jan 2010

Greek government approves Stability and Growth Plan.

Feb 2010

Spain announces annual GDP reduction of 3.6% for 2009.

Mar 2010

First EU leaders’ emergency summit on Greece.
Irish NAMA acquires assets from banks at 47% average discount, increasing banks’
losses.

Apr 2010

Eurostat announces Greek fiscal deficit for 2009 as 13.6% of GDP.
US President Barack Obama expresses concerns about Greece.
Moody’s downgrades Greece to A3. Fitch downgrades Greece to BBB-.
S&P downgrades Greece to BB+ and Portugal to A-.
Greece requests financial support.

May 2010

ECB announces Securities Market Programme.
Ministers agree on bilateral loans to Greece (GLF).
S&P downgrades Greece from BB+ to B+.
Fitch downgrades Spain from AAA to AA+.

Jun 2010

Ministers establish EFSF and agree on the Articles of Association and the Framework
Agreement.
EFSF BoD approves Framework Agreement.
Moody’s downgrades Greece from A3 to Ba1.

Jul 2010

The results of the first pan-European stress tests of the banking system are published.

Aug 2010

Ireland nationalises Irish Nationwide Building Society (INBS) and merges it with AngloIrish Bank.
S&P downgrades Ireland to AA-.

Sep 2010

Moody’s downgrades Spain from AAA to AA.
European Commission presents a package of six legislative proposals aimed at reforming
economic governance and strengthening the framework for preventing excessive
imbalances and excessive deficits.
EFSF receives top rating by Moody’s and S&P.
Irish government announces an extension of the state blanket guarantee for short-term
bank liabilities to end-December 2010.

Oct 2010

German Chancellor Angela Merkel and French President Nicolas Sarkozy agree in
Deauville to create a permanent crisis resolution mechanism and to require private sector
involvement in future sovereign debt restructuring.
Irish government announces Anglo Irish capital needs (€29-34 billion).
Moody’s changes Irish outlook to OW DG; Aa2. Fitch downgrades Ireland by to A+/ NEG.
Fitch downgrades Ireland by to A+/ NEG.

56

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Nov 2010

Eurogroup takes note of Eurostat validation of ELSTAT data for 2009.
Ireland’s borrowing costs rise to 7.7% as the government announces record budget cuts.
Eurogroup publishes statement on the contours of a permanent crisis mechanism.
Irish formal request for financial support (€90 billion).
Eurogroup and ECOFIN ministers unanimously agree to grant financial assistance to
Ireland.
S&P downgrades Ireland by 2-notches to A/ CW NEG.

Dec 2010

Irish Parliament votes to approve the EU-IMF financial assistance program for Ireland.
Moody’s downgrades Ireland by 5-notches to Baa1/ NEG. Fitch downgrades Ireland by
3-notches to BBB+/ STA.
Fitch downgrades Portugal to A+.

Jan 2011

Fitch publishes EFSF First time rating at AAA Stable.
Fitch downgrades Greece from BBB- to BB+.

Feb 2011

S&P downgrades Ireland to A-/ CW NEG.
Ireland holds general elections.
EFSF disburses €1.7 billion to Ireland.
EFSF disburses €1.9 billion to Ireland.

Mar 2011

Leaders agree to make the EFSF’s €440 billion lending capacity fully effective, and to
allow EFSF and ESM primary market interventions.
Portuguese Prime Minister Jose Socrates resigns. Three rating agencies downgrade
Portugal and maintain negative outlook.
Moody’s downgrades Greece from Ba1 to B1.
Agreement on extended repayment period and lower interest rate under Greek Loan
Facility (GLF) loan.

Apr 2011

Fitch changes Irish outlook to NEG; BBB+.
Moody’s downgrades Ireland by 2-notches to Baa3/ NEG. S&P downgrades Ireland to
BBB+/ STA.
Fitch downgrades Portugal by 3-notches to BBB-/ RW NEG; Moody’s downgrades
Portugal to Baa1/ OW DG.
ESM Programme partners conclude first review mission to Ireland.
Portugal lodges formal request for financial support.

May 2011

Fitch downgrades Greece from BB+ to B.
IMF Managing Director Dominique Strauss-Kahn resigns.
Eurogroup and ECOFIN concur with the European Commission and the ECB that
providing a loan to Portugal is warranted to safeguard financial stability in the euro area
and the EU as a whole.

Jun 2011

Moody’s downgrades Greece to Caa1. S&P downgrades Greece to CCC.
Contagion fears dominate euro area sovereign bond market.
Portuguese general elections.
Eurogroup takes note of Greek Debt Sustainability Analysis (DSA) prepared by the
European Commission and the IMF and agrees that the required additional funding will
be financed through both official and private sources.
European Commission publishes Portuguese Economic Adjustment Programme.
EFSF disburses €2.2 billion to Portugal.

57

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Jul 2011

The European Banking Authority (EBA) publishes results of the second round of panEuropean stress tests.
Moody’s downgrades Ireland to Ba1/ NEG and Greece to Ca. Moody’s downgrades
Portugal by 4-notches to Ba2/ NEG.
Fitch downgrades Greece from B to CCC.
Irish authorities establish Irish Bank Resolution Corporation Limited (IBRC) by merging
Anglo Irish and INBS.
Programme partners conclude second review mission to Ireland.
Euro area leaders and EU institutions decide on a new package of measures to end the
crisis and prevent contagion, including a new program for Greece and voluntary private
sector involvement.

Aug 2011

ECB reactivates secondary market purchases and starts purchasing sovereign bonds.
Bilateral agreement of Greece and Finland over guarantees for the latter’s participation in
the Greek EFSF programme.
Programme partners conclude first review mission to Portugal.

Sep 2011

EU leaders announce plans to leverage the EFSF.
Programme partners interrupt fifth review mission to Greece.

Oct 2011

Amendments to the EFSF Framework Agreement come into force.
IMF Global Financial Stability Report estimates that losses due to exposures to sovereign
bonds of “high-spread” euro area countries could reach €200 billion for EU banks. US
President Barack Obama warns about the spill over effects of Europe’s financial crisis.
Euro area leaders hold emergency meeting in Frankfurt.
Spain downgraded by three main rating agencies.
Council agrees on improvements to European economic governance.
Greek Parliament approves pension reforms, new remuneration scales for public sector
and other measures related to the Mid-Term Strategic Fiscal Programme.

Nov 2011

EFSF approves Funding strategy.
Mario Draghi replaces Jean-Claude Trichet as ECB president.
G20 summit discusses euro crisis.
S&P warns of mass euro area downgrade over debt crisis.
Euro leaders discuss the option of Greece exiting the euro area at the G20 summit in
Cannes.
George Papandreou steps down as Greece’s prime minister.
Six central banks announce coordinated actions to enhance liquidity support to the global
financial system.
Fitch downgrades Portugal to BB+/ NEG.
EFSF disburses €3.0 billion to Ireland in two tranches.

Dec 2011

First 36-month Long-Term Refinancing Operation (LTRO) by the ECB.
Leaders bring forward the entry into force of the ESM Treaty to July 2012; increase the
IMF’s resources by up to €200 billion; and reaffirm the “unique and exceptional” nature
of the decisions concerning Private Sector Involvement (PSI) in Greece.
Leaders adopt the Treaty on Stability, Convergence, and Governance in the Economic
and Monetary Union (the “fiscal compact”).
EFSF approves Guidelines for new instruments, with indirect bank recapitalisation and
precautionary instruments.
S&P revises EFSF outlook to Credit Watch Negative on AAA rating.
Fitch and S&P change Irish outlook to NEG; BBB+.
S&P changes Portuguese outlook to CW/ NEG on BBB- rating.
Ireland announces the personal insolvency reform.
EFSF disburses €1 billion to Ireland.
EFSF disburses €1 billion to Portugal.

58

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Jan 2012

S&P downgrades 9 Euro area countries.
S&P downgrades EFSF to AA+ and revises outlook to Developing.
S&P changes Irish outlook to NEG; BBB+ and downgrades Portugal to BB/ NEG.
EFSF disburses €1.2 billion to Ireland.
EFSF disburses €1.7 billion to Portugal.
EFSF disburses €0.5 billion to Ireland.
EFSF disburses €1 billion to Portugal.

Feb 2012

Moody’s downgrades Portugal to Ba3/ NEG.
S&P revises EFSF outlook to Negative on AA+.
The European debt crisis and the need for the EU to increase the size of its stabilisation
fund dominate the G20 agenda.
China and EU leaders meet to discuss the European debt crisis.
Greek ministers quit over a vote on austerity measures. One coalition partner withdraws
support for the government.
Formal request for the second Greek bailout. A Eurogroup meeting to discuss the second
bailout is postponed and replaced by a conference call. Pressure on Greece mounts.

Mar 2012

Most EU member states sign the fiscal compact.
Euro area finance ministers expand the financial firewall to €700 billion (resources of the
EFSF and ESM).
Fitch upgrades Greece to B-/ STA.
Moody’s downgrades Cyprus to Ba1/ NEG.
Greek Finance Ministry announces an 85.8% participation rate in the PSI operation,
cutting the debt by approximately €105 billion.
Eurogroup approves second adjustment programme for Greece.
EFSF disburses €34.6 billion in securities to Greece.
EFSF disburses €5.9 billion to Greece.

Apr 2012

IMF Financial Sector Assessment Program (FSAP) mission to Spain announces that
stress tests covering more than 90% of the domestic banking sector identify ten banks
as vulnerable.
S&P upgrades Greece to CCC/ STA and downgrades Spain by 2-notch to BBB+/ NEG.
EFSF disburses €2.7 billion to Ireland.
EFSF disburses €3.3 billion to Greece.
EFSF disburses securities for €78 billion and €25 billion to Greece.

May 2012

In Greek legislative elections, the two dominant parties fail to secure enough votes for
a majority in Parliament. A second snap election is called.
Fitch downgrades Greece by 2-notches to CCC.
EFSF disburses €5.2 billion to Portugal.
EFSF disburses €4.2 billion to Greece.

Jun 2012

Leaders endorse the concept of banking union and open the door to possible direct
bank recapitalisations by the ESM once an effective single supervisory mechanism is
established.
Moody’s downgrades Cyprus by 2-notches to Ba3/ OW NEG and Spain by 3-notches to
Baa3/ OW NEG.
Fitch downgrades Cyprus to BB+/ NEG and Spain by 3-notches to BBB/ NEG.
Cyprus and Spain formally request ESM stability support. IMF sends a team to Cyprus to
prepare for discussions on programme.
EFSF disburses €1 billion to Greece.

59

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Jul 2012

ECB President Mario Draghi announces the ECB’s willingness to do 'whatever it takes' to
save the euro.
EFSF disburses €2.6 billion to Portugal.
EFSF disburses €1 billion Portugal.
EFSF disburses €480 million to Ireland.
Ireland returns to international bond markets with the sale of a five-year bond.
Eurogroup unanimously agrees on the details of the financial assistance to Spain.
EFSF approves Spanish Master Facility Agreement.

Aug 2012

ESM approves Pricing Guideline and maximum lending volume.
S&P changes Greek outlook to NEG; CCC, and Cyprus to BB/ CW NEG.
EFSF disburses €1 billion to Ireland and €1.02 billion to Portugal.

Sep 2012

ECB announces an Outright Monetary Transactions (OMT) framework for intervention in
sovereign bond markets of countries accepting EFSF and ESM support.
ESM approves Guidelines on financial assistance instruments and borrowing.
Following a street protest, the Portuguese government abandons proposed fiscal
measures.
EFSF disburses €4.9 billion to Greece.

Oct 2012

The ESM is formally inaugurated in Luxembourg.
Moody’s changes Spanish outlook to NEG from OW NEG and downgrades Cyprus to
B3/ NEG.
S&P downgrades Spain to BBB-/ NEG and Cyprus to B/ CW NEG.
Fitch publishes ESM rating at AAA Stable.

Nov 2012

Fitch downgrades Cyprus by 2-notches to BB-/ NEG.
Moody’s downgrades EFSF to Aa1 Negative.
S&P downgrades EFSF to AA Stable.
ESM approves ESM lending documentation and Diversified Funding Strategy.
Greece and programme partners reach full staff-level agreement on updated programme
conditionality.

Dec 2012

IMF announces EU financial system stability assessment, recommending further steps
towards banking union.
S&P downgrades Greece to SD, then upgrades Greece to B-/ STA following the debt buy
back.
EFSF disburses €0.8 billion to Portugal.
ESM disburses in 5 portions for overall €39.468 billion to Spain and separately €1.865
billion.
Greek government launches a debt buyback scheme seeking to retire about half of the
debt owed to private creditors.
EFSF disburses €11.3 billion to Greece for debt buy back.
EFSF disburses €23 billion to Greece in two tranches.

Jan 2013

Dutch Finance Minister Jeroen Dijsselbloem is appointed Eurogroup president.
Moody’s downgrades Cyprus by 3-notches to Caa3/ NEG.
Fitch downgrades Cyprus by 2-notches to B/ NEG.
Portugal sells €2.5 billion of five-year bonds through banks, the first offering with that
maturity in almost two years.
EFSF disburses €2 billion to Greece.

Feb 2013

Nikos Anastiades wins presidential election in Cyprus.
ESM disburses €1.865 billion to Spain.
EFSF disburses €0.8 billion to Portugal.
EFSF disburses €1.4 billion to Greece.
IBRC is liquidated by the Irish authorities.

60

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Mar 2013

S&P stabilises Portuguese outlook to BB/ STA.
S&P downgrades Cyprus to CCC/ NEG.
Eurogroup has its first exchange with the new Cypriot government.
Ireland sells €5 billion of new benchmark ten-year bonds to meet nearly all its funding
needs through the next year.
Eurogroup agrees to the adjustment of the maturities of the EFSF loans to Ireland and
Portugal to smooth the debt redemption profiles.
Eurogroup and Cyprus reach agreement on a future macroeconomic adjustment
programme.

Apr 2013

Fitch changes Cypriot outlook to RW NEG; B.
S&P changes Cypriot outlook to STA; CCC.
ESM BoG approves an assessment of the public debt sustainability of Cyprus. Cyprus
and programme partners reach staff-level agreement on the policy conditionality
underlying the macroeconomic adjustment programme.
EFSF disburses €2.8 billion to Greece.

May 2013

Fitch upgrades Greece to B-/ STA.
Portugal issues its first new government bonds in two years, in a heavily-oversubscribed
offer of ten-year debt that raises €3 billion.
EFSF disburses €0.8 billion to Ireland.
EFSF disburses €2.8 billion to Greece.
EFSF disburses €4.2 billion to Greece.
EFSF disburses €7.2 billion to Greece.
ESM disburses €2 billion to Cyprus.

Jun 2013

Fitch downgrades Cyprus to B-/ NEG.
S&P places Cypriot rating as Selective Default.
IMF Executive Board reviews Greece misreporting, remedial steps, and discusses ex
post evaluation of 2010 Stand-By Arrangement (SBA).
Beginning of the bank Balance Sheet Assessment process in Ireland.
EFSF disburses €1.6 billion to Ireland.
EFSF disburses €3.3 billion to Greece.
EFSF disburses €1.05 billion to Portugal.

Jul 2013

Euro area Member States sign ESM Treaty. ESM becomes the sole and permanent
mechanism for responding to new requests for financial assistance.
Fitch downgrades EFSF to AA+ Stable.
S&P upgrades Cyprus to CCC+/ STA.
EFSF disburses €2.5 billion to Greece.

Aug 2013

Portugal’s Constitutional Court rejects a bill that would effectively allow the state to fire
civil servants.

Sep 2013

EFSF disburses €1 billion to Ireland.
ESM makes a €1.5 billion disbursement in bonds to Cyprus.

Nov 2013

Fitch changes Spanish outlook to Stable; BBB.
Moody’s upgrades Greece to Caa3/ STA.
S&P upgrades Cyprus to B-/ STA.
Eurogroup congratulates Spain on successful financial sector programme
implementation.
EFSF disburses €3.7 billion to Portugal.
Completion of the Irish Prudential Liquidity Assessment Review (PLAR) process
(deleveraging) for two banks.

61

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Dec 2013

EU finance ministers agree on the design of the Single Resolution Mechanism.
EFSF disburses €2.3 billion to Ireland
EFSF disburses €0.5 billion to Greece.
ESM disburses €100 million to Cyprus.
End of Irish EFSF programme.
End of Spanish ESM programme.

Jan 2014

Greece’s highest administrative court reverses wage cuts that were imposed by the
government.

Mar 2014

ESM approves establishment of Early Warning System.

Apr 2014

EFSF disburses €1.2 billion to Portugal.
Eurogroup calls on Greece to do its utmost to conclude a review rapidly.
EFSF disburses €14.8 billion under Greek PSI.
EFSF disburses €6.3 billion to the Greek segregated account.
Greece returns to global capital markets for the first time since 2010, raising €3 billion in
a five-year bond deal.

May 2014

Portugal’s Constitutional Court strikes down some austerity measures, thus requiring the
government to implement compensatory measures to reach its fiscal targets.
End of Portuguese EFSF programme.
Eurogroup underlines the need for completion of the prior actions for the disbursement
to Greece.

Jun 2014

Spain makes a voluntary early repayment of €1.3 billion to the ESM.
Moody’s revises EFSF and ESM outlook to Stable on Aa1 rating.
ESM Members reach political understanding on the operational framework of the direct
recapitalisation instrument.
European Commission publishes first Irish Post Programme review.
Portuguese government decides to allow the programme to expire at the end of June
without the final review and without receiving the last tranche.

Jul 2014

ESM disburses €600 million to Cyprus.
Spain makes a €308 million repayment of unused funds to the ESM.

Aug 2014

ESM Boards establish the Direct Recapitalisation of Institutions instrument.
Moody’s upgrades Greece to Caa1/ STA.
ESM approves Cyprus fifth review.
EFSF disburses €1 billion to Greece.

Sep 2014

S&P upgrades Greece to B/ STA.
Euro area finance ministers approve Ireland’s plan to repay its IMF loan ahead of
schedule.

Oct 2014

ECB takes over the role of banking supervisor for the Euro area’s 120 largest banks under
the Single Supervisory Mechanism.
IMF publishes Portuguese Fiscal Transparency Evaluation.

Dec 2014

Fitch revises ESM outlook to Stable on AAA rating and downgrades EFSF to AA Stable.
Ireland completes the early repayment of almost half of its loans from the IMF.
ESM disburses €350 million to Cyprus.
European Commission publishes Portuguese and Spanish Post Programme Monitoring
report (Oct 2014).
Greece and EFSF execute second Amendment Agreement.

Jan 2015

European Commission publishes Irish Post Programme Monitoring report.

Feb 2015

Ireland starts early repayments to the IMF.
Eurogroup receives request for extension of the Greek programme.
EFSF receives €10.9 billion in redelivery of bonds not used by the Greek HFSF.

62

Mar 2015

Spain makes a €1.5 billion voluntary early repayment to the ESM.

Apr 2015

European Commission publishes Spanish Post Programme Monitoring report
(Spring 2015).

Evaluation of EFSF and ESM financial assistance: Technical Appendices | June 2017

SELECTED GENERAL AND PROGRAMME EVENTS
Jun 2015

Eurogroup welcomes Cypriot programme being brought back on track.
Expiry of Greek EFSF programme.

Jul 2015

EFSF approves Reservation of Rights relating to non-payment to the IMF.
Spain makes a €2.5 billion voluntary early repayment to the ESM.
ESM disburses €100 million to Cyprus.
ESM approves Decision in principle to grant financial assistance to Greece and first
disbursement is made.

Oct 2015

ESM approves Maximum Authorised Maturity for ESM borrowing operations.
ESM disburses €500 million to Cyprus.

Jan 2016

European Commission publishes Irish Post Programme Monitoring report Autumn 2015.

Mar 2016

IMF staff completes 2016 FSAP mission in Ireland.
End of Cypriot ESM programme.
Eurogroup supports the Cypriot government’s decision to exit its macroeconomic
adjustment programme without a successor arrangement.

Apr 2016

European Commission publishes Portuguese Post Programme Monitoring report.

Jun 2016

European Commission publishes Spanish Post Programme Monitoring report.
ESM to conduct first evaluation of EFSF and ESM activities in the context of financial
assistance programmes.

Sources: The time line draws on EFSF and ESM documents; Eurogroup statements; Greek Crisis Timeline: a chronology of key events for the Greek
debt crisis, a database of key official documents & pronouncements (Athens Centre for International Political Economy (ACIPE)); An IMF-centric
timeline of key events during the Euro area crisis, September 2008 - December 2014 (IMF Independent Evaluation Office (2014)); Ireland Ex Post
Evaluation of Exceptional Cases under the 2010 Extended Arrangement (IMF IEO 2015); Fitch, Moody's, Standard & Poor's

63