Evaluation PSOM/PSI
Report number. 1377
2016

1999-2014

Evaluation PSOM/PSI
1999-2014

Evaluation team
Phil Compernolle (APE Public Economics)
Philip de Jong (APE Public Economics)
Bert van Manen (Timpoc)
Klaas Molenaar (Timpoc)
Karen Reijnen (MDF)
Johan Vonk (APE)
Mike Zuyderduyn (MDF)
Local experts
Gonzalo Manrique, Ronald Bourgeois Carpio (Peru)
Ahmed Monem (Egypt)
Darijo Lazic (Bosnia-Herzegovina)
Samson Tholley (Sierra Leone)
Joyanta Roy (Bangladesh)
Samual Mtebi (Uganda)
With contributions from:
Leonie Gercama (APE), Selma van der Haar (APE)
Evaluation for the Dutch Ministry of Foreign Affairs, Department Sustainable Economic Development (DDE)
The Hague, 2016

Report number. 1377
© APE Public Economics
Website: www.ape.nl/en

Table of contents
List of Abbreviations

7

Executive summary

i

1

Evaluation PSOM/PSI
1.1
Introduction
1.2
Evaluation approach
1.3
Overview evaluation methodology
1.4
Data sources
1.5
Synthesis of results
1.6
Structure of the report

1
1
2
12
15
18
18

2

Overview of PSOM/PSI
2.1
Introduction
2.2
PSOM/PSI in retrospect
2.3
Governance
2.4
PSOM/PSI portfolio

21
21
21
22
27

3

Relevance
3.1
Introduction
3.2
Description of selection process
3.3
Relevance: PSOM/PSI objectives
3.4
Relevance: Country objectives
3.5
Ex-ante additionality
3.6
Conclusions on relevance

31
31
31
41
51
52
59

4

Efficiency
4.1
Introduction
4.2
Programme expenditure and operating costs
4.3
Portfolio, staff and target countries
4.4
Productivity
4.5
Cost drivers
4.6
Conclusions

62
62
62
65
67
68
71

5

Effectiveness
5.1
Introduction
5.2
Goal achievement
5.3
Contribution PSI/PSOM
5.4
Effectiveness: combining success and contribution

72
72
73
91
97

6

Impact
6.1
CSR, including working conditions
6.2
Impact on market and sector development
6.3
Impact on applicants
6.4
Sustainability
6.5
Conclusions

101
101
106
107
108
110

7

Lessons learned
7.1
International partnerships
7.2
Adaptability
7.3
Gender
7.4
Trade
7.5
Boosting an enterprising culture
7.6
Monitoring and Evaluation

111
111
112
112
113
114
114

Annex I Interviews

117

Annex II: Terms of Reference

119

Annex III: Evaluation team and reference committee

141

Annex IV: Case study interview guide

142

Annex V: Employee survey guidelines

150

Annex VI: Overview PSOM/PSI contribution

157

Annex VII: Quantitative methodology and analysis
Monitoring data
Multivariate analyses

159
159
160

Annex VIII: Survey questionnaire

180

Annex IX: Case studies (overview visited projects)

183

Annex X: Evaluation matrix

187

List of Abbreviations
APSOM/APSI
BAS
CSR
DDE
DGGF
IOB
FMO
LDC
LIC
LMIC
MoFa
MoV
PSD
PSO
PSOM
PSI
RVO
SME
ToR
UMIC

Independent advisory body for PSOM/PSI
The monitoring system of RVO
Corporate social responsibility
Sustainable Economic Development Department
Dutch Good Growth Fund
Policy and Operations Evaluation Department of the Dutch Ministry for Foreign affairs
Entrepreneurial Development Bank (Dutch development bank)
Least developed country
Low income country
Lower middle income country
Ministry of Foreign Affairs
Means of verification
Private Sector Development
Programme for Co-operation with Countries in Eastern Europe
Programme for Co-operation with Emerging Markets
Private Sector Investment programme
Netherlands Enterprise Agency
Small and medium sized enterprise
Terms of reference
Upper middle income country

Evaluation PSOM/PSI | 7

Executive summary
The Programme for Co-operation with Emerging Markets (PSOM) and its successor, the Private Sector Investment programme (PSI), were established by the
Dutch Ministry of Foreign Affairs (MFA) to foster innovative private sector investments by Dutch (and later non-Dutch) companies in cooperation with local business partners in selected project countries. Between 1998-2014 both programmes
awarded subsidies (or grants) to 1107 investment projects in 59 countries. These
grants covered 50%, or 60% in the case of frail states, of the project budget, with
a cap of 750,000 euros (and 900,000 for frail states). The final tender for PSI was
in 2014. The last projects are expected to be completed in 2020. The programme
is administered by the Netherlands Enterprise Agency (RVO).
In short, the aim of PSOM and PSI was to trigger investment projects that would
otherwise not have been realized because of high (perceived) product risk or market risks, lack of access to finance or lack of access to technology. These projects
could entail existing or new (greenfield) businesses (but no start-ups). After the
subsidy period, follow-up investments were expected to lead to further growth of
sales, profit and employment of the business. This should lead to sustainable economic development in the PSOM/PSI country by increasing employment,
knowledge transfers and income.
Intervention logic
Through the selection and subsidisation of projects PSOM/PSI aims to attain three
main outcomes at the level of the supported businesses: (1) the establishment of
innovative businesses in developing countries and countries with emerging markets, (2) job creation and knowledge transfer and (3) the implementation of policies concerning corporate social responsibility, gender equality and environmental
sustainability.
In the intermediate and long term, it is expected that the individual projects lead
to the following development impacts:
1. Improved job opportunities (for local population, suppliers and outgrowers);
2. Expansion and catalysing effect of businesses impact (improvement local
economy, sustainable growth);
3. Increased local attention for CSR, gender and environmental business
policies.
These impacts are considered to be beyond the direct control of PSOM/PSI, occurring within the wider environment of the supported businesses.

Evaluation PSOM/PSI | i

Evaluation questions
Topics of this evaluation are the relevance, efficiency, effectiveness and impact of
the programme. The relevance of PSOM/PSI is assessed by answering the question whether the activity would not have been realised without the subsidy. Efficiency poses the question the cost-effectiveness of the implementation of the
programme. Effectiveness regards goal achievement and the contribution of
PSOM/PSI to projects being successful. While effectiveness deals with direct results (employment, knowledge transfer and sales), impact goes beyond that to
sector, market and corporate social responsibility (CSR). Moreover, impact is concerned with the longer-term effects, or sustainability, of PSOM/PSI after the grant
period has expired.
Methodology and data sources
Given the scope of this evaluation and the limitations of available data at RVO, it
was not possible to rigorously establish the impact of the programme using quantitative methods. Moreover, the nature of the programme limits the options for
such an analysis:
 Since the projects are managed by an international partnership, it is very
hard to define a valid control group.
 The programme subsidizes new activities, so pre-intervention (baseline) data cannot be collected (on the project level).
The evaluation team used instead a mix of qualitative and quantitative methods
to analyse the evaluation questions. 49 qualitative case studies (field visits) were
complemented by the quantitative analysis conducted in the Netherlands on the
basis of monitoring information from RVO. The quantitative and qualitative studies fed into each other by answering the same evaluation questions using different data sources and methods for analysis.
49 case studies were done across 6 countries with projects starting in 2000 and
some running until 2017.1 They were based on data collected through semistructured interviews (by phone and on location) and observations of the companies involved with PSOM/PSI projects. Furthermore, in order to validate findings
and give due consideration to country-specific factors determining the success of
PSOM/PSI projects, the country studies also included interviews with a select
group of local stakeholders (e.g. to determine whether findings are samplespecific or can be assumed to be representative of a specific sector/country RVO
programme managers responsible for the selected countries were interviewed as

1

ii |

In Bangladesh, Egypt, Uganda, Sierra Leone, and Bosnia-Herzegovina 8 case studies were
undertaken and in Peru 9.

2016

well as part of the preparations for the country studies (together with the review
of project files).
Moreover, desk research and interviews were combined with an analysis of project files and electronic monitoring data. These provided a base for quantitative
analyses. A survey amongst 91 rejected applications was done to see whether
they had been able to realise the project without PSOM/PSI subsidy.
Relevance
The evaluation of the relevance of the PSOM/PSI programme focused on:
1. Selection and objectives: Do the selection criteria and process ensure the
selection of projects in line with the objectives of PSOM/PSI?
2. Country priorities: Are the supported investments in line with the objectives and priorities of the PSOM/PSI country?
3. Ex-ante additionality:2 Would the projects have been realized on the same
scale and at the same pace without the financial contribution of PSOM or
PSI?
Projects were selected on the strength of the partnership between the applicant
and the local partner, the quality of the project plan and the development impact.
The project had to be innovative and should not distort the market. The project
plan should contain clear, logical and measurable results on production outcomes,
knowledge transfer (training), spill-overs for key parties in the production chain
(including CSR outcomes) and a viable financial plan. The impact criterion focussed on the spin-off of the project and its development effects. Applicants had
to argue what the potential spin-off of the project would be, in terms of follow-up
investments and additional turnover, in case of the long-term establishment of a
commercially viable company.
Conclusions on relevance
The PSOM/PSI programme is considered relevant with regard to the selection of
projects in light of the overall objectives of the programme:
 The strength of the partnerships varied between countries and projects, but
in 20 out of 30 finalised projects visited for the case study the relationship is
still ongoing and positively appreciated. Among the successful projects that
were surveyed by RVO for their spin-off analysis, 80% were still run by the
same partnership.
 The majority of projects are new for the company, sector and country (31/49),
which indicates that PSOM/PSI indeed targeted projects with elements of innovation or at least ‘newness’. Newness should also be suitable for the context, which was not always the case. A project can be new to the company,
2

Additionality ex ante according to DCED (Heinrich, 2014)

Evaluation PSOM/PSI | iii





sector, country or even the world. Some projects are truly innovative, others
show that this claim is exaggerated to meet the PSOM/PSI requirements. The
innovation criterion also ensured that there was little unwanted market distortion.
Potentially, most projects have an impact on the value chain (sector) or wider
market (country), beyond the company and those directly involved, except for
those projects that were set up solely to serve the applicant (e.g. production
units).
Selected projects all have the potential to be commercially and financially
successful. The financial profile of selected projects differs from rejected projects on a number of indicators. The largest differences are found in the financial profile of the applicant company. For instance, the average profits reported by selected applicants were € 8.7 mln, whereas the average profits
reported by rejected applicants were € 6.2 mln. This is confirmed by multivariate analysis, which shows that the financial ratio is the only project characteristic that significantly affects the selection probability. Local partner financial
indicators and the division of shares do not seem to differ significantly between selected and rejected projects.

Additionality
For the majority of projects (30/49) the PSOM/PSI grant is considered additional
as source of funding (or for the reduction of risks). This is, however, strongly dependent on the country’s context, whereby in middle income countries most
PSOM/PSI projects might have happened, regardless of the subsidy. For 7/49 projects additionality is doubtful, with investments being larger than necessary due
to the ‘perverse’ availability of the grant funding and the tendency of applicants
to ask the maximum. The remainder 12/49 of the projects would have happened
without PSOM/PSI support (not additional). All in all, most projects would not
have been realised on the same scale and at the same pace without the financial
contribution of PSOM or PSI.
Country needs
PSOM/PSI is, however, considered less relevant with regard to country-specific
needs. As both PSOM and PSI were set up as thematic project-support rather than
country programmes, there is by nature little alignment with country-specific priorities of national governments or even programmes of Dutch embassies. Even
though RVO programme managers do consider the country context, PSOM/PSI
consists of one programme design (e.g. selection process, indicators, grant modality) for all countries (apart from PSI Plus countries), ranging from middle income
countries like Peru to low income countries like Uganda. A more country-specific
approach could have improved the relevance of the portfolio (e.g. in certain countries loans might be more appropriate than grants).

iv |

2016

Moreover, lack of alignment with other private sector development programmes
(from the Netherlands or elsewhere) is a missed opportunity, in particular with
regard to the possible synergies between PSOM/PSI and other Dutch programmes
for private sector development and economic diplomacy. PSOM/PSI could also
have used its presence in a country as a provider of significant grants to the private sector to leverage general improvements in business environment, including
addressing entrepreneurial capacity.
Efficiency
Programme expenditure (€ 2.1 million in 2000 - € 56.8 million in 2014) and operating costs (€ 0.6 million in 2000 - € 6.0 million in 2014) increased considerably. So
did the portfolio (workload: 90 projects in 2000 – 435,5 projects in 2014), staff
(fte: 19 in 2004 – 38 in 2014) and number of (untied) target countries (21 in 2004
– 60 in 2014). In the PSI period between 2009 and 2014 we observe a steady increase in productivity. Operating costs and fte per programme expenditure and
management and overhead costs per project all decreased during PSI. This is attributable to the increase in workload (scale economies), the increased standardization of and familiarity with the PSI selection procedure, stabilization of the
number of countries (less marketing efforts), the decrease in communication efforts (with the planned termination of the programme in 2014) and possibly other
efficiency gains. The only exception is the development costs per project. These
increased from €10,000 per selected project in 2005 to €35,000 in 2014. This is
due to the high number of applications in the last few tenders. Hence, the number
of applications is an important cost driver in the PSI period.
Effectiveness
Effectiveness is assessed by considering:
 Goal achievement: the extent to which the project targets have been met in
terms of jobs and knowledge transfer, sales and investments, and engaging
outgrowers.
 Attribution: the extent to which PSOM/PSI contributed to the observed
changes (among other factors contributing to success or failure of projects).
Overall, two thirds or more of the finalized projects achieved their targets on employment, knowledge transfer, or outgrowers. Finalized projects have a lower goal
achievement rate on sales and follow-up investments. In short, the projects are
more successful in reaching development goals than in achieving business targets.
Statistical analysis was used to assess the contributory effect of PSOM/PSI, which
could be determined for employment and sales, though not for training. To further evaluate the effectiveness of PSOM/PSI, this evaluation relied on the case

Evaluation PSOM/PSI | v

studies to identify the way in which the programme contributed to the achievements of the projects.
From the case studies we learn that:
1. Firstly, 33 of 49 projects visited were operational businesses. In five
cases it was too early to tell. However, this 75% (33/44) cannot be
compared with RVO monitoring information on goal achievement because of a bias in the sample as operational businesses can be visited,
while non-performing projects are by nature less accessible.
2. Secondly, in 18 out of all the 443 projects (41%) evaluators concluded
based on interviews with applicants, project partners, file review and
company visits that the PSOM/PSI programme played a crucial role in
the resulting company (i.e. decisive contribution without which the
outcome of the project might well have been different). In the remainder of the projects visited, PSOM/PSI was seen to have been only
one of many factors contributing (at least by providing a grant), but
this support was considered to not have been decisive for its
achievements.4
3. Finally, if we then combine this information, out of the 49 projects visited, 17 projects that were operational businesses, received a contribution from the PSOM/PSI programme that was considered to be decisive - 5 projects started too recent to assess (39% or 17/44).5
Business status

PSOM/PSI
contribution

Operational

3

Total

Not-operational

Decisive

17

1

Not decisive

16

10

33

11

44

For ease of reading we exclude the 5 cases in which it was too early to tell whether a
company was operational or not, though that does not necessarily exclude an assessment of the contribution (which can be made at very early stages).
4
See box 3 for the assessment of contribution as either decisive (main factor, beyond the
financial contribution) or minimal (one of many factors, mere financial contribution)
for the achievement of results.
5
If you would subtract from the 49 project those considered not to be ex ante additional
(see chapter 3 on relevance), 14 projects would be left that are operational and to
which PSOM/PSI made a significant contribution (32% or 14/44).

vi |

2016

Predictors of success
Quantitative analysis of ex-ante financial indicators finds that only the applicant’s
balance (total of assets) and equity ratio (this ratio measures the proportion of
the balance financed by the applicant) differ significantly between stopped and finalized projects, i.e., lower for stopped projects than for finalized projects.
Analysis of the scores of the ranking system shows that there is no relationship
between ex-ante rankings and project success. This might not be surprising in light
of the fact that only positively appraised projects were ranked, i.e. the projects
fulfilled the requirements for a PSOM/PSI programme. However it shows that project success is difficult to predict. Rankings rely on ex-ante information, and success is also influenced by many external factors.
Contribution of PSMO PSI (quantitatve analysis)
The contribution of the PSOM/PSI subsidy significantly contributes to the
achievement of the employment target. The outcome implies that if PSOM/PSI
committed subsidy is raised by ten percent (say from € 550.000 to € 605.000), the
number of jobs created at the end of the project is increased by almost four,
keeping other things equal. A higher subsidy also translates into more sales, albeit
at a lower level of confidence: 10% more subsidy means 5.3% more sales.
Impact
To assess development impact we used the case studies to investigate the extent
to which the projects contributed to:
1. Improved job opportunities (for local population, suppliers and outgrowers);
2. Expansion and catalysing effect of businesses impact (improvement local
economy, sustainable growth);
3. Increased local attention for CSR, gender and environmental business
policies.
From these we conclude that:
 27 out of 49 cases were found to have an effect on the broader sector
(e.g. on outgrowers and local suppliers) or market (e.g. import substitution) beyond the effects on the joint venture itself;
 The role of PSOM/PSI projects in promoting CSR is limited as CSR is mostly
market driven. Nevertheless, in half of the cases studied there was some
(19 cases) or strong (4 cases) influence of PSOM/PSI on CSR; e.g. through
dialogue between applicant and local partner or explicit financial support
through PSOM/PSI for certification;
 Certification and chain responsibility are promoted by PSOM/PSI. Working
conditions are mostly above the country’s average.

Evaluation PSOM/PSI | vii



PSI focuses on female employment in numbers, which might have a small
temporary effect in some of the projects, but much less on policies and
practices required to employ and retain women and improve decent
working conditions in the longer term;

Ultimately, whether a business venture is commercially sustainable depends on
the entrepreneurial skills of those who become the owners of the project, once
the grant period is concluded. The evaluation found many instances where
PSOM/PSI contributions have helped to launch a project that proved successful
through the efforts of the entrepreneurs involved.

viii |

2016

1
1.1

Evaluation PSOM/PSI
Introduction
The Programme for Co-operation with Emerging Markets (PSOM) and its successor, the Private Sector Investment programme (PSI), were established by the
Dutch Ministry of Foreign Affairs (MFA) to foster innovative private sector investments by Dutch (and later non-Dutch) companies in cooperation with local business partners in selected project countries. Between 1998-2014 the programmes
awarded subsidies to 1107 investment projects in 59 countries6. The final tender
for PSI was in 2014. The last projects are expected to be completed in 2020.7 The
programme is administered by the Netherlands Enterprise Agency (RVO).
In short, the aim of PSOM and PSI was to trigger investment projects that would
otherwise not have been realized because of high (perceived) product risk or market risks, lack of access to finance or lack of access to technology. These projects
could entail existing or new (greenfield) businesses (but no start-ups). After the
subsidy period, follow-up investments were expected to lead to further growth of
sales, profit and employment of the business. This should lead to sustainable economic development in the PSOM/PSI country by increasing employment,
knowledge transfers and income.
MFA evaluates the results of PSD programmes, including PSOM/PSI, every five
years. Previous PSOM/PSI evaluations were conducted by Ecorys (2005) and Triodos/Facet (2009). In 2016, the Sustainable Economic Development Department
(DDE) commissioned the consortium of APE, MDF and Timpoc consultants to conduct an evaluation for the period 1999 to 2014. This draft report presents the
findings of this evaluation.

6

Technically PSOM was not a subsidy programme. In this report we will use the term ‘subsidy’ to denote the PSOM/PSI support.
7
In line with PSOM/PSI documentation, this evaluation report uses the term ‘projects’.
However, it is worthwhile noting that RVO provided grants to enterprises, which is actually different from being projects (or even implementing or funding projects).

Evaluation PSOM/PSI | 1

1.2

Evaluation approach
1.2.1 Intervention logic
Figure 1-1 shows the intervention logic of PSOM/PSI, as defined by RVO in 20128.
The intervention logic forms the basis for the evaluation. The chain starts with the
inputs with which RVO runs the programme: the annual grant budget, the programme budget (operations), the number of full-time employees involved in
PSOM/PSI and the knowledge and expertise available among PSOM/PSI staff. The
main programme activities carried out with these inputs are: PSOM/PSI promotion, the selection and approval of projects, the monitoring and evaluation of projects and programme revision. These activities result in a selection of projects
which are awarded a subsidy: the programme outputs.
The efforts of applicants and local partners (‘project partners’ in Figure 1-1) also
an important input for the overall programme output. Partners contribute both
financially (own contribution) and in terms of labour, knowledge, local networks
and contacts. As PSOM/PSI required a considerable co-financing by the applicant
and partners (see Table 3-6 for the percentages of the various sub-programmes),
the applicant and partners had to make a strong financial commitment, which was
checked during application stage and in the financial reports as well.
Through the selection and subsidisation of projects PSOM/PSI aims to attain three
main outcomes at the level of the supported businesses: (1) the establishment of
innovative businesses in developing countries and countries with emerging markets, (2) job creation and knowledge transfer and (3) the implementation of policies concerning corporate social responsibility, gender equality and environmental
sustainability.
In the intermediate and long term, it is expected that the individual projects lead
to the following development impacts:
1. Improved job opportunities (for local population, suppliers and outgrowers);
2. Expansion and catalysing effect of businesses impact (improvement local
economy, sustainable growth);
3. Increased local attention for CSR, gender and environmental business
policies.
These impacts are considered to be beyond the direct control of PSOM/PSI, occurring within the wider environment of the supported businesses.

8

2 |

The result chain, was established in 2012 and slightly adjusted in 2015, to match
OECD/DAC guidelines (ToR).

2016

Figure 1-1 Result chain

Source: Terms of reference9
1.2.2 Evaluation questions
The Terms of Reference (ToR) for this evaluation prescribed the evaluation questions. These have been adapted in the Inception Report, and accepted after discussion with the Reference Committee for this evaluation January 19, 2016. For
the purpose of this evaluation, some slight changes have been made to the questions, though the intent remains the same.
Table 1-1 contains the evaluation questions (including their place in the report). In
the following sections we will describe the evaluation questions in more detail.
Table 1-1: Evaluation questions
Criteria
Relevance
and ex-ante
additionality

9

Chapter in

Evaluation questions

report
Do the selection criteria and process ensure that the selection 3.2 - 3.4
of projects is in line with the objectives of PSOM/PSI?
Would the projects have been realized on the same scale, in
3.5
the same pace and with the same impact without the financial
contribution of PSOM or PSI (additionality ex ante according to

Ministry of Foreign Affairs (2015) Terms of reference evaluation PSOM & PSI

Evaluation PSOM/PSI | 3

DCED)?
To what extent have the targets of the projects been achieved
(goal achievement)?
What are the changes in the effect variables in comparison to
Effectiveness
the situation at the start (baseline)?
Which is the attribution of the observed changes to the intervention?
How is the efficiency of the implementation of the PSI programme by RVO affected by:
Efficiency
a.
PSI country list (the number of countries on the list)?
b.
applicants from third countries?
What is the impact of projects on the implementation of CSR
principles? How has this affected employees/environmental
issues/other local enterprises, etc.? How have the projects influenced gender relations?
What is the impact of the projects on the development of the
market in the country of implementation (horizontal linkages)?
Which other (non-intended) effects can be attributed to the
Impact and
projects?
sustainability What is the impact on the employees of the PSI supported
company? (when possible disaggregation between
male/female employees and special attention to impact on
female employment and on decent working conditions for
women)
Given the information available (gathered by RVO.nl through
monitoring and the spin-off survey) on stopped, completed
and ongoing projects; how sustainable are the PSOM/PSI investments?

5.2 - 5.3
5.3 – 5.4
5.4
4.1 – 4.6

6.1

6.2

6.1 – 6.5
6.1

6.4

Source: Terms of reference evaluation PSOM/PSI

1.2.3 Relevance
Relevance focuses on several questions:
 In the first place whether supported projects are in line with PSOM/PSI objectives. This requires an evaluation of the selection process to assess
whether the right projects were chosen to meet the PSOM/PSI objectives
(partnerships, innovation, development impact and commercial viability).
 Secondly, relevance also includes the assessment of PSOM/PSI’s ex-ante additionality, which refers to the additionality of the available funding, before
the funding decision was made. For example, selected projects should ultimately be commercially viable, but the entrepreneurs should not have been
able to finance the projects themselves, or to obtain commercially available
funding for the project. The subsidy is additional to the market if the activity

4 |

2016

would not have been realized (or not within the same time-frame and at the
same scale) without the subsidy.10
DCED (2014) sets out eight criteria for the assessment of ex-ante additionality
(See Box 1). According to the ToR (p. 16) the evaluation should focus on three criteria specifically:
1. Financing: Is the applicant unable to self-finance the project (within a reasonable time frame) through own funds or third parties (e.g. commercial
providers, other donors)?11
2. Risk: Is the applicant unwilling to implement the project because the
partners perceive the risks as too high?
3. Market distortion: Does the project supported by PSOM/PSI risk setting
other companies already operating or ready to enter the same market at
a disadvantage?
Box 1: DCED criteria for ex ante additionality
DCED criteria for assessing ex-ante additionality (DCED 2014)
 The company has insufficient funds to self-finance the project.
 The company lacks the knowledge or competencies to design and/or implement a business model in a way that maximizes poverty-reducing or
other (economic) development impacts.
 Without the subsidy, the company would be unwilling to implement the
proposed business model and/or changes in the operational standards
because of a perceived negative balance of costs/risks and benefits.
 The company cannot access commercially available funding (or technical
support).
 The subsidized activity does not displace other companies operating in
the market, or that are ready to undertake the same project without
public support.
 The cost-shared contribution does not duplicate other donor-funded
support
 Public support leverages investments from other entities that would otherwise not be forthcoming
 Conditions attached to support, or agency activities complementing the
cost-sharing collaboration, are expected to have a positive influence on
wider business operations, operations by other businesses, or the business environment.

10
11

DCED 2014
See section 3.6.4. on another side of additionality, best described as overambitious projects due to the availability of grant financing rather than business case.

Evaluation PSOM/PSI | 5

The ex-ante additionality of PSOM/PSI funding was evaluated through an assessment of the selection process itself, and the outcome of the selection process.
This assessment consists of:
 Selection process. Is the selection process designed in such a way that projects in line with PSOM/PSI objectives are selected? The analysis of the selection process is based on desk research of RVO documentation, interviews
with RVO staff, and relevant literature.
 Outcome of the selection process based on the 49 case studies. In the case
studies additionality (for financing and risk) was assessed per project (including assessment of innovativeness and market disturbance of projects).
 Quantitative analysis of rejected and selected projects. Using a database of
all selected PSI projects and a random sample of 200 rejected PSI projects,
we have quantitatively analysed which characteristics of the proposals, applicants, and local partners increase (or decrease) the probability of successful application. Do rejected and selected projects have a dissimilar profile?
How do they differ? Based on this dataset we also looked at the reasons why
projects were rejected, and whether these reasons include lack of ex ante
additionality.
 Survey amongst rejected applicants. This survey included questions on the
status of the rejected project (was the project implemented without
PSOM/PSI funding? At the same scale?), reasons for either stopping the project or continuing it without PSOM/PSI funding, experience with similar projects, and whether the applicants found other funding sources. All these
questions aim to inform the evaluation about the additional role of
PSOM/PSI funding.

6 |

2016

Box 2: Assessment of ex-ante additionality in the case studies
Ex-ante additionality, whether the project would have been realised at a similar scale and pace without PSOM/PSI, is hard to determine once the project
has started or ended. Nevertheless, interviews with applicants and local partners did give useful insight into the extent to which the PSOM/PSI support
was considered additional. The focus was on additionality regarding financing
(e.g. availability of other sources of financing or own financing), but the evaluators also considered the extent to which PSOM/PSI support was required to
reduce investment risks.
Additionality was assessed through a combination of the following questions
(as well as document review from RVO files and context analysis):
 What was the motivation to join the PSOM/PSI programme?
 What other (public and private) parties were involved with the project
(or would have been if PSOM/PSI did not support the project), and their
role?
 Would the project have happened at the same scale and pace without
PSOM/PSI?
 Would you have joined the programme if it provided a loan rather than a
grant?
 General assessment of the financial strength and size of the applicant
and local partner,
 Country risk profile.
Additionality is important to avoid market distortion, therefore the case studies also considered:
 How innovative is the project?
 Are there comparable projects/products in the country?
 What were the main competitors at the start of the project, and now?

1.2.4 Efficiency
The ToR did not define efficiency, nor does RVO report on specific value for money indicators to MFA. Therefore, the evaluation team devised a number of indicators to evaluate whether the PSOM/PSI was implemented in an efficient manner.
These indicators were extracted from financial data from annual reports and annual plans from RVO. Such indicators are, among others:
 Total costs;
 Costs per selected project;
 Extra costs for applicants from third countries;
 Cost per application.

Evaluation PSOM/PSI | 7



Cost drivers

Ideally, the evaluation of efficiency would include a benchmark analysis, comparing the efficiency of the PSOM/PSI programme operation with efficiency of operation of other similar programmes. However, there is no such data about similar
programmes readily available12. As an alternative, we therefore evaluated the development of the operation cost of PSOM/PSI using a number of efficiency indicators, for the period 2000-2015.13 Disaggregation for country or nationality of applicant was not possible as data was not available at this level.
1.2.5 Effectiveness
Following the terms of reference for this evaluation, effectiveness assesses:
 Goal achievement: the extent to which the project targets have been met in
terms of jobs and training, sales and investments, engaging outgrowers14
 Contribution of PSOM/PSI to the observed changes (among the other factors
contributing to success or failure of projects)15
However, it is important to note that goal achievement and operational success of
PSOM/PSI projects is only the first step. For PSOM/PSI to be truly effective a significant and additional contribution of PSOM/PSI has to be established. Therefore,
this evaluation takes the following reasoning:
 PSOM/PSI is effective if
A. Set project targets are achieved (goal achievement as monitored by
RVO);
B. Evaluation through the case studies provides evidence that
o projects are operational / commercially viable,16 and
o PSOM/PSI support made a difference (contribution is considered
decisive for the success of the project, see box 3).
12

There are similar programmes, such as AECF.
The analyses in this chapter are based on information from the annual reports 20002015. The annual report of 1999 was not available. For some of the indicators information was only available for the years 2004-2015 or 2004-2014.
14
The extent to which PSOM/PSI projects are innovative, also part of this evaluation question, has been described extensively in chapter 3.4.2. as part of the evaluation of relevance.
15
Note that the inception report refers to the attribution of the observed changes to the
intervention (‘What are the changes in the effect variables in comparison to the situation at the start (baseline)?’). As will be discussed here below, given the evaluation
methodology (and lack of counterfactual), it is more appropriate to refer to the contribution of PSOM/PSI rather than attribution.
16
A project was considered operational if there was active production upon visiting the
site (e.g. machines in use, staff on site, evidence of clients). This is closely linked to
goal achievement in the area of sales and employment.
13

8 |

2016

Figure 2. Evaluation of PSOM/PSI effectiveness

Operational

PSOM/PSI
Contribution

Goal
achievement

Evaluation PSOM/PSI | 9

Box 3: Assessment of PSOM/PSI contribution in the case studies
The case studies, which include document review, interviews with applicants and local
partners, and company observations, were used to reconstruct the PSOM/PSI contribution for a representative sample of 49 projects. For the synthesis of these case studies a distinction was made between:
a) minimal contribution, whereby PSOM/PSI is one of many factors contributing to
the success of the project (merely through the financial contribution); or
b) decisive contribution, whereby PSOM/PSI is considered by the project partners to
be one of the main factors, going beyond the mere financial support, including
less tangible contributions such as risk-reduction, catalyst role and positive effects of the international partnering that are all non-financial components of the
programme.
The contribution of PSOM/PSI was assessed in the case studies through the following
questions (combined with document review from RVO files):
 What is the experience with RVO
 Critical success factors (and barriers) for the project
 Extent to which PSOM/PSI contributed to achievements (how and if not, why
not) through
o Investment in hardware
o Knowledge transfer, training and technical assistance
o Direct employment creation
o Other
 Has the project affected CSR practices?
 Has the project affected other parts of your company?
 What would you recommend for future investment programmes?
In the case study interview report there were two occasions in which evaluators
would rank PSOM/PSI contribution as no causal link to be established / one of many
factors / main factor / crucial link (see reporting format in annex IV). However, during
the synthesis-process, during workshops and review of the interview reports, it
turned out that the use of two distinct categories (one of many factors and significant) was more appropriate for an assessment of the PSOM/PSI programme.
a) It was concluded that at the minimum PSOM/PSI contributes financial support for a newly-established joint venture, so that in practice the programme
will always contribute to the project (at least as one of many factors), just as
any other investors and the entrepreneurs themselves (who obviously invest
much more financially and in-kind). Therefore, the category (no or negligible
link) was merged into the category ‘one of many factors’ or minimal contribution.
b) This category is countered by the category ‘decisive contribution’, which
combines the projects for which project partners agreed that PSOM/PSI was
one of the main or the main factor contributing to the success. Such a decisive contribution required going beyond the free money, with a contribution
that no other factor could have made, and not just through providing financial support (but also, e.g. knowledge transfer or access to other resources).
Chapter 5.3.3. describes how PSOM/PSI made such contributions.

10 |

2016

Several information sources were used to evaluate the outcomes:
 Monitoring data from RVO. RVO has provided the evaluators with a large
dataset containing digitally available monitoring data for both PSOM and PSI
projects. This dataset includes indicators such as: proposed and realised
turnover, proposed and realized employment, and the number of employees trained. This data was used to provide an overview of PSOM/PSI’s portfolio.
 Quantitative analysis of success rate. The RVO monitoring data was also
used for statistical analysis of the factors that increase the probability of a
project being successful. This includes factors about the project (country,
sector, tender, subsidy), as well as data about the applicant, local partner,
and the selection process (rankings).
 Qualitative country studies. The case studies of 49 PSOM/PSI projects in six
countries serve to qualitatively evaluate the way in which outcomes have
(not) been achieved, reaching a better understanding of the factors that
contributed to success (see chapter 2 for more information on methodology). Moreover, the country studies offer an opportunity to validate the
available monitoring data.
1.2.6 Impact and sustainability
According to the ToR for this evaluation, impact relates to the effects of PSOM/PSI
further down the results chain. While effectiveness deals with direct results (employment, training and sales), impact goes beyond that to sector, market and corporate social responsibility (CSR).17 Moreover, impact is concerned with the longer
term effects, or sustainability, of PSOM/PSI after the grant period has expired.
Given the scope of this evaluation and the limitations of available data at RVO, it
is not possible to rigorously establish the impact of the programme using quantitative methods. Moreover, the nature of the programme limits the options for
such an analysis:
 Given that the projects are managed by an international partnership, it is
very hard to define a valid control group. For subsidy programmes rejected
proposals can often be used as a control group. This requires rejected proposals which are similar to the accepted ones (i.e. were rejected because of
lack of funds, as opposed to lack of quality). However, this approach is not
viable for PSI/PSOM, where proposals were rarely rejected because of a lack
of funds.

17

Though arguably CSR could be considered part of effectiveness.

Evaluation PSOM/PSI | 11



The programme subsidizes new activities, so pre-intervention (baseline) data cannot be collected (on the project level).18

Impact will therefore primarily be assessed on the basis of the 49 case studies in 6
countries, where interviews with project management, employees, and stakeholders in the country address:
 (Potential)19 impact of the programme on the sector and market in which
the project operates (e.g. spin-off to others in the value chain)
 Impact on working conditions of employees in participating companies (and
perhaps broader within the sector)
Moreover, RVO has also recently completed a spin-off survey, which assesses results in the two years after the closing of a project. The spin-off survey focusses on
all PSOM/PSI projects that were successfully finalized before December 2013. The
survey seems to have some response bias towards successful businesses and the
findings can’t be fully attributed to PSOM/PSI20.But it does provide useful information to complement the findings of this evaluation.

1.3

Overview evaluation methodology
1.3.1 Evaluation matrix
During the inception phase of the evaluation the ‘evaluation matrix’ was finalized.
The full evaluation matrix is included in annex VIII and describes the way in which
each evaluation question will be answered using different data sources (triangulation).
1.3.2 Mixed-method
The evaluation team used a mix of qualitative and quantitative methods to analyse the evaluation questions. The 49 qualitative case studies were complemented
by the quantitative analysis conducted in the Netherlands on the basis of monitor18

R. Oostendorp, J.W. Gunning (2012) Advisory Report Using baselines and control groups
in monitoring and evaluation for Private Sector Investment Programme (PSI). Commissioned by AgentschapNL.
19
Depending on how long the project has been ongoing. There is a bias in the case study
methodology towards more recent projects as there was less commitment from older
projects to share information (in particularly from projects that have since closed or
experienced difficulties, thus with limited impact). Some of the older PSOM projects
declined to collaborate with this evaluation “because they had already been so often
reviewed”.
20
Moreover, the spin-off survey was held among company owners/managers, their views
on employment and CSR issues were not cross-checked with staff members.

12 |

2016

ing information from RVO. The quantitative and qualitative studies feed into each
other by answering the same evaluation questions using different data sources
and methods for analysis. Throughout the evaluation, findings were discussed in
team workshops in order to ensure both evaluation methods strengthened each
other (e.g. provide background information for data analysis, validation of findings).
1.3.3 Case studies (field visits)
The country studies were of a qualitative nature and were based on data collected
through semi-structured interviews (by phone and on location) and observations
of the companies involved with PSOM/PSI projects. Furthermore, in order to validate findings and give due consideration to country-specific factors determining
the success of PSOM/PSI projects, the country studies also included interviews
with a select group of local stakeholders (e.g. to determine whether findings are
sample-specific or can be assumed to be representative of a specific sector/country RVO programme managers responsible for the selected countries will
be interviewed as well as part of the preparations for the country studies (together with the review of project files).
Case studies consisted of the following elements:
 Review of documentation from project files of each company selected into
the sample (49);
 Interviews with RVO programme managers of each country to be visited;
 Telephone interviews with Dutch partners of the sample of 48 PSOM/PSI
projects (applicants).
 Interviews and observations on the projects in the sample of 48 PSOM/PSI
projects (i.e. interviews with the local partners and/or managers of the projects), including group interviews with employees;
 Interviews with about 3 stakeholders/experts in each country, focusing on
the relevance of PSOM/PSI within the country context and used to validate
company interviews (e.g. identification of external factors that affected the
outcomes of PSOM/PSI).
Case studies were carried out together with local consultants.
Country selection
The case study countries were chosen based on criteria in the Terms of Reference,
and was adapted in accordance with the preferences of the reference committee.
The final country selection is shown in Table 1-2. Counties were selected on the
basis of:
 Geographical coverage (different regions, one of which Arab);
 Country income (LIC and MIC);

Evaluation PSOM/PSI | 13





At least one fragile state (PSI+);
Sector coverage;
Coverage in previous evaluations21.

Table 1-2: countries selected for case studies
Country

Location

Development PSI+

PSI Arab

Projects in sample

stage
Bosnia

Eastern Europe

UMIC

NO

NO

8

Bangladesh

Asia

LIC/LDC

NO

NO

8

Peru

Latin-America

UMIC

NO

NO

9

Egypte

Africa

LMIC

NO

YES

8

Sierra Leone

Africa

LIC/LDC

YES

NO

8

Uganda

Africa

LIC/LDC

NO

NO

8

Interview guidelines and reporting formats
In order to ensure comparability between case studies standardized interview
guidelines and reporting formats were designed in the inception phase. These reporting formats were tested during a test run in Peru, and were later refined during feedback sessions. The guidelines and reporting formats are included in annex
IV.
1.3.4 Quantitative analysis
The monitoring data collected by RVO was analysed using quantitative research
techniques. Next to the digitally available monitoring data, the evaluation team
also collected additional financial data for all accepted PSI projects and 200 rejected PSI projects. Using this dataset, the following analyses were possible:
 An analysis of the selection process (profiles of selected and rejected companies, characteristics which increase the probability of selections)
 Analysis project outcomes (are there any indicators which increase the
probability of success?)
 Analysis of the contribution of the subsidy (how much does one additional
euro of subsidy affect the outcome variables?)
 Analysis ranking system (does a higher ranking score mean projects are
more successful?)
The quantitative methodology and results are presented in annex V.

21

14 |

In consultation with the reference committee the current evaluation did not include countries included in previous evaluations. For this reason, Tanzania was substituted for Uganda. Secondly
Pakistan was substituted for Sierra Leone, as the latter can more clearly be categorized as a
fragile state.

2016

1.4

Data sources
1.4.1 Desk research and interviews
The background and implementation of PSOM/PSI was reconstructed by means of
desk research of relevant documentation and interviews with RVO staff and a
member of the APSI committee. The evaluation team interviewed RVO staff
members in the positions: project advisor, financial advisor, team manager and
staff at the directory level. An overview of the interviewees can be found in Annex
I.
1.4.2 RVO monitoring data
PSOM/PSI recipients (i.e. applicants) reported on the progress of their project towards pre-defined result indicators throughout their grant period. Up to 2011 this
was required after reaching a result indicator. As of 2011, with a change in government wide subsidy guidelines22, a maximum of one report per year was set.
These result indicators fall into broad categories defined by RVO but are specified
by applicants (in consultation with RVO). Examples of such targets (and examples
of accompanying means of verification) are:
 Registration of new company and/or joint venture (e.g. shareholders’
agreement)
 Formulation of CSR policy (e.g. copy of CSR policy)
 Building (e.g. local inspection and photos)
 Hardware installed (e.g. local inspection, protocol of operations)
 Staff trained (e.g. amounts of employees contracted and trained, with specific % female, training logbook)
 Company operational (e.g. sales records, copy of receipts)
In the final report, RVO asked about impact based on the following indicators, for
which applicants report on realised and expected (in two years) figures:
 Turnover
 Follow up investments
 Number of jobs in the project
 Number of jobs occupied by females in the project
 Number of people trained during the project
 Number of outgrowers engaged during the project
RVO asked the following additional indicators under PSI:

22

Number of jobs created at low/medium and high level

The reporting requirements for PSOM/PSI changed with the introduction of the government wide uniform subsidy framework (USK/RUSK). These standards were introduced to reduce the administrative burden associated with subsidies.

Evaluation PSOM/PSI | 15

 Number of people trained of low/medium and high level and outgrowers
 Number of subcontractors
Furthermore, the recipients were asked to report on working conditions, environmental impact and chain effects, alongside narrative information on external
factors, investment climate and economic sustainability.
The exact indicators which were collected evolved over time. Monitoring was
more extensive for PSI projects then for PSOM projects. For instance, both PSI and
PSOM used an indicator for the number of jobs created, but for PSI jobs were also
disaggregated into high level- and low level jobs. With PSI the spin-off indicators
were also introduced. The idea was that projects provide indicators 2 years after
the end of the subsidy period, in order to monitor impact and sustainability. Monitoring also partially reflects the specific focus of the sub-programmes. For PSI Arab RVO also collects data on young entrepreneurs and young employees.
The monitoring of the projects by RVO was largely based on the assessment of the
incoming project reports by the RVO project managers.23 Since the introduction of
new standardized subsidy guidelines (RUS/USK)22, formal, paper-based reporting
requirements were reduced and monitoring of progress became more dependent
on informal monitoring by the project managers personally (by phone and mail).
Thereto project advisors visited projects in the regions they managed. Up to 2015,
each country was visited twice a year, both to monitor existing projects as well as
to assess new applications. Since 2015, once PSI had stopped, project managers
tried to visit projects at least once a year for monitoring purposes (e.g. to inspect
building and training of outgrowers) and to discuss changes to the project plans
(e.g. delays, changes in certification). At the end of the grant period, project managers usually pay an extra visit (e.g. to check whether the hardware financed is in
use).24




Monitoring data are stored in the BAS-system. This system has a number of
drawbacks, as it is not designed as a monitoring system, but as a project
management system. In the first place it is nog easy to generate tables for
useful indicators. Secondly the system overwrites previous values for the
collected indicators. This means project managers can see the latest monitoring information, but not the development over time.
The monitoring data was not complete. Data were more consistently collected (and registered in the system) for indicators used for the subsidy pro-

23

Project managers were responsible for all projects in a certain region. During the evaluation period, each advisor managed between 10-30 projects (see chapter 4 for a more
in-depth description of the workload).
24
Interviews RVO

16 |

2016



gress (sales, jobs, trainings), but were less complete for indicators such as
the type of jobs, follow-up investments, or number of out growers. Even for
a core indicator such as jobs RVO data were registered in BAS for just 251
projects (59% of finalized projects).
Especially spin-off data was sparsely collected. There was no incentive for
businesses to send information two years after the subsidy relationship had
ended. In 2015/2016 RVO carried out a survey in order to spin-off data from
completed projects.

1.4.3 Validation of monitoring data
Initially, the country case studies undertaken for this evaluation were considered
an opportunity to validate the RVO monitoring data. However, the monitoring of
key indicators by RVO, as well as the validation thereof, are both snapshots at a
particular moment in time. For ongoing projects, the values of the key indicators
will vary, among others along with the economic tide in the individual countries.25
For finalized projects, it is difficult to check the data retrospectively. In any case,
validation of indicators such as employment and sales proved hard given the fluidity of the indicators and lack of standard definitions. For example, different definitions of sales are used, and employment consists of different categories that are
not always easily distinguished (e.g. fixed contracts versus longer term employees). Several companies did not maintain separate accounting for the PSOM/PSI
project, so it as hard to distinguish between employment and sales generated by
PSOM/PSI projects or by the company of the local partner. Moreover, often managers were not eager to share detailed company information with the evaluators
(for various reasons).
1.4.4 Survey amongst rejected projects
The purpose of the survey among rejected applicants was to address questions on
additionality and selection (i.e. relevance). A survey was held among 555 applicants whose applications were rejected in the period 2010-2014. Out of the 555
invitations, 450 were delivered and 105 bounced. 91 respondents completed the
survey giving a response rate of 20 %. Of these 91 respondents, 28 received a
grant for the revised project proposal in a later tender round26. Most of the re-

25

Only for finalised projects, would there be a fixed value at end-point. This information is
used, e.g. in the efficiency section (e.g. PSOM/PSI cost per job generated). However,
for the case studies most of the projects are visited when ongoing or having been finalized for some time, i.e. not at the exact time of RVO monitoring.
26
Often rejected projects were given pointers in order to bring the project (and the application) more in line with PSOM/PSI requirements and goals. Although re-application
wat by no means a guarantee for success.

Evaluation PSOM/PSI | 17

spondents were either micro (25%) or small (46%) enterprises and were active in
agriculture (41%) or industry (37%).

1.5

Synthesis of results
As described above, this evaluation covers several components and different data
analysis methods for answering the evaluation questions. This is illustrated in the
evaluation matrix in annex VIII, which indicates what data and analysis were used
for the respective questions. Care was taken to ensure a coherent synthesis of
findings from the different evaluation methods.
In an ideal world, qualitative studies are used to inform the data and theory behind the quantitative studies (e.g. hypotheses). However, given the scope of this
evaluation, the quantitative study depends on secondary data from RVO monitoring systems, which limits the options for analysis (and thus the questions that can
be answered with that data). However, the case studies strengthen the quantitative studies, as they a) validate the underlying data on site; b) evaluate the same
evaluation criteria through a different method, c) provide information about the
way in which the identified changes and relations of the quantitative studies
might have come about. And vice versa, the quantitative study, conducted in early
phases of the evaluation, might well provide useful pointers for the qualitative
evaluation at country level. Ultimately, these methods should be considered as
two, interlinked, sources of information with which the evaluation criteria will be
answered.
Different methods were used to ensure the different data sources and analysis
were combined most effectively:
 Midway the evaluation, a team workshop was organised to discuss preliminary findings from the quantitative study and the first of the qualitative
country studies. This enhanced the consistency of the different studies.
 At the end of the evaluation phase, a final team workshop was organised to
discuss findings. Moreover, the full team commented on multiple drafts of
the final report.
 The final report answers the evaluation questions, mixing the different data
sources and analysis methods, in order to make sure these strengthen each
other towards clear conclusions.

1.6

Structure of the report
The rest of this report is structured as follows:
 Chapter 2 provides an overview of the PSOM/PSI programme, the main developments, key figures, and a timeline.

18 |

2016



Chapter 3 answers the evaluation questions with regards to relevance and
ex-ante additionality. It looks at the selection process, PSOM/PSI objectives,
country relevance and ex-ante additionality.
 Chapter 4 analyses the efficiency of PSOM/PSI.
 Chapter 5 discusses the effectiveness of the programmes. It looks a goal
achievement, stopped and finalized projects, and the contribution of
PSOM/PSI
 Chapter 6 discusses impact and sustainability.
 Finally, Chapter 7 presents recommendations.
The Chapters 3-6 end with conclusions per chapter.

Evaluation PSOM/PSI | 19

20 |

2016

2
2.1

Overview of PSOM/PSI
Introduction
This chapter provides a descriptive overview of development of PSI and PSOM
over time, as well a summary of the subsidy spent, and the PSOM/PSI portfolio.
This chapter also describes the programme management and governance.

2.2

PSOM/PSI in retrospect
2.2.1 PSOM 1998-2008
PSOM started as a pilot programme in 1998. The programme was inspired by the
Programme for Co-operation with Countries in Eastern Europe (PSO), a programme aimed at positioning Dutch companies on the Eastern European markets
after the fall of the Berlin wall. That programme was deemed a success and the
possibilities for starting a similar programme in developing countries were explored. Additional requirements regarding development impact and own contributions were added, and PSOM’s first tender was launched in 1999. In the period
1998-2001, eight target countries were eligible.
After 2001 Dutch development policy became more focussed on incorporating
private sector actors. Additional target countries were added to the programme
and the programme budget increased (€ 129 million across 2002-2004). In 2003
the programme was partially untied, i.e. non-Dutch applicants could apply for a
number of project countries27. In 2004, PSOM and PSO were merged.
2.2.2 PSI 2009-2014
PSOM was stopped in 2008 as the result of legal wrangling concerning the nature
of the financial support28. For this reason, PSI was introduced in 2009 as a new
subsidy programme with similar goals and character. In order to facilitate the increased number of countries and project applications in PSI, the selection procedure was professionalised (see chapter 3).

27
28

The only restriction is that applicants should not originate from the project country.
Hanzeland casus

Evaluation PSOM/PSI | 21

PSI plus
With the introduction of PSI, an additional sub-programme was introduced, focusing on fragile states: PSI Plus. The requirements for this sub-programme were
more flexible than for the regular PSI programme; the maximum subsidy was
higher and an additional insurance was offered (and financed) to cover investment risks in fragile states (MIGA insurance).
PSI Arab
In 2012, following the Arab spring, a second sub programme was introduced specifically for projects in the Middle East and North Africa (MENA) region29: PSI Arab.
PSI Arab focussed on young people and women both as owners and as employees.
Decommissioning
The Ministry of Foreign Affairs decided to close PSI from 2015 onwards.30 The last
PSI tender was held in March 2014. The last projects are expected to finish in
2020.
Figure 2-1: timeline of PSOM and PSI

2.3

Governance
The PSOM/PSI programme is implemented and managed by RVO, an agency of
the Ministry of Economic Affairs. Both programme costs and operating costs31 are
financed by the Ministry of Foreign Affairs. Annually, RVO had to provide a proposal for both budgets, based on estimates of the number of projects and the
number of days that are required to select and manage each project. Project se29

Algeria, Egypt, Iraq, Yemen, Jordan, Morocco, the Palestinian Territories and Tunisia
(Besluit van de Staatssecretaris van Buitenlandse Zaken van 13 December 2011, nr.
DJZ/BR/1506-11, Stcrt. 2011, 23127).
30
PSI was replaced by DGGF.
31
These cover the costs of managing the programme: holding tenders, selecting new projects and managing existing projects. See chapter 4 for more information on the development of these costs.

22 |

2016

lection is carried out by a selection team consisting of a team leader, a project advisor and a financial advisor. The management of the projects is divided geographically between country managers. Each country manager manages the projects in
several countries in the same region. Before 2015, project advisors visited the
country twice a year, both to monitor existing projects as well as to assess new
applications. Since 2015, project advisors try to visit a project at least once a year.
These visits are used for monitoring purposes - for instance to check whether local
outgrowers are included in the project - and to discuss certain changes to the
plans -for instance delays due to unforeseen circumstances or changes in certification. Additionally, at the end of the projects they pay an extra visit to check
whether the hardware financed under the programme is in fact installed and in
use. When projects face large problems/difficulties they can be put on hold by the
project advisor.
Table 2-1 shows the development of PSOM and PSI in terms of the available subsidy and the number of target countries. Over its lifetime the programme awarded up to € 820 million in subsidies. The number of countries invested in rose from
8 in the first phase (1998-2001), to 59 at the time of the last tender (2014).
Table 2-1: Overview PSOM and PSI
PSOM/PSI period Period

Total budget available Countries

PSOM 1st phase

1998-2001

€ 37.9 million

8

PSOM 2nd phase 2002-2004

€ 129 million

2002: 11
2003: 17
End of 2003: 21

PSOM 3rd phase

2004-2006

€ 45.9 million

42

PSOM 4th phase

2007-2010

€ 235 million

53

PSI 1st Phase

2009-2010

€ 140 million

51

PSI 2nd phase

2011-2014

€ 232 million

59

Total

1998-2014

€ 820 million

-

Source: ToR (p. 5).
2.3.1 Applications and acceptance rate
In total there were 2,912 applications for PSOM and PSI between 1998 and 2014.
Of these applications 1,350 (46%) were for PSOM and 1,562 (54%) for PSI. Figure
2-2 shows the development of the number of granted and rejected applications
for PSOM/PSI per year. Each year on average 38% of the applications was granted
and the remaining 62% were rejected32. The number of applications per tender increased between 1999 and 2014, reflecting the increase in the available subsidy

32

The rejection rate is roughly similar for PSOM (36%) and PSI (39%).

Evaluation PSOM/PSI | 23

budget, the increase in the number of eligible countries, and the increasing number of firms interested to do business in developing countries.33
Figure 2-2 Number of applications(rejected and granted)for PSOM and PSI
400
Number of applications

350
300
250
200
150
100

50
0

Rejected

Granted

Source: RVO monitoring data, analysis: APE
PSI plus and PSI Arab
Almost 75% of the total number of applications between 2009 and 2014 fall under
PSI Regular 20% under PSI Plus and 7% under PSI Arab. Figure 2-3 shows the share
of each sub-programme in the total number of PSI applications. The number of
tenders differed per year. Applications for PSI Regular and PSI Arab were combined in a single tender.34 For PSI Plus, there were initially separate tenders.35 This
proved to be too labour-intensive and the PSI Plus tenders were then combined
with the tenders for PSI Regular and Arab from 2011 onwards. There was only one
tender in March 2014. This being the last one, the number of applications almost
doubled in this last tender with a total of 215 for PSI Regular combined with PSI
Arab and 49 for PSI Plus, compared to the 343 applications in total for the two
tenders in 2013.

33

Interviews RVO staff
Although, a separate budget is available and a separate ranking is done for Regular and
Arab projects.
35
th
Government Gazette No. 231, Published on 27 of November 2008 and Government
st
Gazette No. 18299, Published on 1 of December 2009
34

24 |

2016

Figure 2-3: Number of applications for PSI per subprogramme 2009-2014
400
Number of applications

350
74

300
250
200

88
18

40
21

39

50

49
31

150
100

195

168

207

179

219

184

50
0
2009

2010

2011

PSI Regular

PSI Arab

2012

2013

2014

PSI Plus

Source: RVO monitoring data, analysis: APE

2.3.2 Subsidies committed
Both the total number of applications and the number of granted projects increased over the years. This was in line with developments in the programme
budget. As Figure 2-4 demonstrates, the annual budget originally committed to
the granted PSOM/PSI projects36 increased from around €10 million per year in
1999-2001 to almost €90 million at its peak in 2013. A break in this trend can be
observed in 2008 when there was only one tender due to the transition from
PSOM to PSI. Likewise, in 2014 there was only one tender for PSI Regular and one
tender for PSI Plus, as the PSI programme ended that year.

36

The originally committed budget is the sum of all grants that are committed at the moment of acceptance of the projects. The final budget can differ from this amount as
not all grants are (fully) disbursed, e.g. when projects are stopped mid-way.

Evaluation PSOM/PSI | 25

Programme budget (€ Millions)

Figure 2-4 Originally committed PSOM/PSI budget 1999-2014
100
90
80
70

60
50
40
30
20
10
0

Source: RVO monitoring data, analysis: APE

Figure 2-5 shows the average committed subsidies per project. The amount fluctuates between € 300,000 and € 500,000 between 1998-2008. The average committed amount per project has increased since the start of PSI to almost € 600,000
per project by 2013.37
Figure 2-5 Average committed subsidy per project38

Source RVO monitoring data, analysis: APE
37

26 |

After 2005, projects are still ongoing so the subsidy committed is not necessarily the
same as the subsidy disbursed.

2016

Figure 2-6 shows the originally committed, committed and disbursed subsidy for
all accepted projects under PSOM/PSI. The originally committed amount is determined in the selection phase. The disbursed subsidy amount is determined after
the project is completed. The difference between originally committed and committed amount is mainly explained by prematurely stopped projects, and ongoing
projects (i.e. not as committed, nor yet disbursed).

Millions of euro's

Figure 2-6 Originally committed, committed and disbursed amount per tender
100
90
80
70
60
50
40
30
20
10
0

Originally committed

Committed

Disbursed

Source RVO monitoring data, analysis: APE

2.4

PSOM/PSI portfolio
2.4.1

Target countries

Income levels
In total 43% of selected projects were based in LIC’s. 34% of selected projects
took place in LMIC’s. The remainder (23%) were based in UMIC’s.
Region
In total 43% of projects are located in Africa (Figure 2-7). The regions with the
lowest percentage of projects were Central and Eastern Europe, and the MENA
region. Both represent only 8% of the total project portfolio. The disbursements
show the same regional distribution.

Evaluation PSOM/PSI | 27

Figure 2-7: Percentage of granted projects per region

8%

Africa
Asia

15%
43%

Central and Eastern
Europe

8%

Latin America

25%

Middle East and North
Africa

N = 1107; all granted projects
Source: RVO monitoring data, analysis: APE

Did the introduction of PSI Arab have an effect on the share of subsidies going to
the Mena39 region? Figure 2-8 shows the development of total subsidies and
Mena subsidies over time. The share of Mena country subsidies increases at a
faster rate after the introduction of the PSI Arab program.

39

28 |

Countries in the Mena region are: Algeria, Egypt, Iraq, Jordan, Libya, Morocco, Palestine Territories, Tunis and Yemen. Only projects from these countries are eligible for
PSI Arab.

2016

Figure 2-8: Mena country subsidy versus total subsidy

N = 91 granted projects in MENA countries and 1107 granted projects in total
Source: RVO monitoring data, analysis: APE

Approximately 10% of the granted projects under PSI and PSOM were situated in
so-called fragile states. If we look only at the granted projects under PSI, then 22%
of the projects occur in fragile states. This is approximately the same percentage
as the 20% share of PSI Plus, which focused on fragile states, in the total of accepted PSI projects.
53% of the projects operated in the agricultural sector, including agro-processing.
Approximately 30% of the projects were industrial ones, including energy projects, and the remaining 15% focused on the service sector.
In the first years of PSOM country specific MOUs were signed with the target
countries, detailing the sectors and amounts which were to be invested in the
country. However later the distribution of the portfolio among countries or sectors was purely demand driven, based on the incoming applications and their
strength. No targets were used for these categories during the selection process.

Evaluation PSOM/PSI | 29

30 |

2016

3
3.1

Relevance
Introduction
As described in chapter 1, the evaluation of the relevance of the PSOM/PSI programme focuses on:40
1. Selection and objectives: Do the selection criteria and process ensure the
selection of projects in line with the objectives of PSOM/PSI?
2. Country priorities: Are the supported investments in line with the objectives and priorities of the PSOM/PSI country?
3. Ex-ante additionality:41 Would the projects have been realized on the
same scale, at the same pace and with the same impact without the financial contribution of PSOM or PSI?
This chapter starts with a description of the selection process for PSOM/PSI. As
the differences in the selection process of PSOM and PSI are minimal, they are described together with special references to changes in the process of time (3.2).
Thereafter, the relevance with regard to the objectives of PSOM/PSI (3.3), country
priorities (3.4) and additionality (3.5) is discussed one by one.

3.2

Description of selection process
The information presented in this chapter is based on project documents of the
PSI programme and interviews with current PSI officers at RVO. Overall, the programme design has not changed much over the years. The selection, management
and financing procedures described in this chapter roughly applied to the PSOM
programme as well (although the processes became more elaborate over time).
For instance, the selection procedure was steadily professionalized and standardized (e.g. introduction of a more comprehensive ranking system).
Project selection followed the procedures for public tenders, which starts with an
announcement in the Government Gazette. The selection process was carried out
by a selection team consisting of a team leader, a project advisor and a financial

40
41

APE, Inception Report Evaluation PSOM/PSI 1998-2014, final version, January 2016
Additionality ex ante according to DCED (Heinrich, 2014)

Evaluation PSOM/PSI | 31

advisor. The first was responsible for the assessment of the content of the application, the latter for the financial assessment.
Figure 3-1 depicts the selection procedure for PSOM/PSI.42 Before applicants applied for the programme through the tender, they had the opportunity to participate in an intake interview with RVO staff to discuss their business case and verify
eligibility. RVO encouraged applicants to make use of this intake interview. Thereafter, the assessment of the applications followed six steps as described here below.
Figure 3-1:

Selection procedure PSOM/PSI and means of verification (on the
right)

3.2.1 Formal requirements
For each application, the first step was to check adherence to the formal submission requirements. These concerned procedural requirements such as complete
and correct applications and the absence of unethical practices43. In case of incomplete applications applicants had one opportunity to add any missing docu42

The information presented in this paragraph is based on programme documents on the
PSOM/PSI programme and interviews with current PSI officers at RVO.
43
As of 2013 these are defined as practices that are on the FMO exclusion list of unethical
practices – e.g. child labour and other illegal and/or unethical activities such as projects in the tobacco- and liquor industry: https://www.fmo.nl/exclusion-list

32 |

2016

ments within one week. Projects that did not comply with the formal requirements would not be assessed any further.
3.2.2 Admission criteria
The second step was a check of the admission criteria. Just like the formal requirements the admission criteria were minimum conditions, but these concerned
the focus, size and form of the project and partnership. As will be discussed later
on in this chapter, some of these criteria have been interpreted loosely in practice.
Most criteria remained unchanged over the PSOM and the PSI period. For instance, the projects had to take place in one of the PSOM/PSI focus countries. Applicants and local partners had to be small or medium enterprises (SMEs) or they
had to demonstrate that their project would benefit SMEs. Both partners had to
be private companies and had to be officially registered at the local Chamber of
Commerce. The applicant had to be exist for at least two years. The local partner
had to be established in the project country. And lastly, in line with the objectives
of the PSI programme, projects had to stimulate local employment. Projects solely
aimed at the export of hardware to project countries were not eligible.
Other admission criteria changed over time. Table 3-1 lists the main differences in
admission criteria between PSOM and PSI:44
 The project duration (non-agricultural), the maximum project budget (€
1,500,000 for all projects) and the maximum grant (€ 900,000 for PSI Plus
countries) were increased under PSI.
 In the first few years of PSOM (1998-2004) the programme offered a reimbursement for proposals that were positively assessed but could not be selected because of budget constraints. This reimbursement covered some of
the costs of writing the proposals.
 Within the PSI Plus programme, an additional subsidy was offered for a MIGA insurance45 to cover the higher risks in these fragile states.
 As for co-financing, under PSOM co-financing by other Dutch government
agencies was allowed, as long as total external financing (including PSOM)
would not exceed 80% of total project budget. Under PSI, however, cofinancing by other Dutch government agencies was not permitted.
44

Other criteria or their interpretation may have also changed somewhat within the PSOM
or PSI period. The table includes those that have done so explicitly.
45
MIGA is the Multilateral Investment Guarantee Agency of the World Bank Group. This
insurance covers up to 90% of equity investments and 95% of debt investments (including the PSI-grant). Because of the cumbersome procedure to obtain the insurance
but few companies obtained MIGA (Source: Annual reports 2011-2013).

Evaluation PSOM/PSI | 33









46
47

34 |

With PSOM/PSI the Dutch government aimed to provide ‘one-off financial
support’ for investment in high risk innovative projects in developing countries46. Therefore, the PSI tender documents mentioned that the partnership
should have no previous projects under PSOM or PSI and that the applicant
should have no more than one running project under PSOM/PSI. These conditions were not included in the tender documents of the PSOM-tenders.
However, the tender documents for PSOM do mention that preference was
given to consortium members that had not previously benefitted from the
programme. This was weighted as one of the factors in the ranking phase.
Local partners were allowed to be a local sister or daughter company of the
applicant or an existing joint venture, though local ownership of the company was preferred.
During the whole evaluation period, projects had to take place in PSOM/PSI
countries. The number of eligible countries increased from 8 in 1999 to 60 in
2014. Between 1999 and 2003 separate tenders were set out for each country. Under PSOM, priority sectors, themes and groups were defined in the
Memorandum of Understanding between RVO and governments of recipient countries47. This did, however, not exclude projects that did not fit within these priorities. Under PSI, no country priorities were defined anymore.
Lastly, at the start of the PSOM programme (1999-2002) projects were required to buy at least 60% of hardware in the Netherlands, but this rule was
abolished soon after.

PSI spin-off analysis, RVO 2016
These are for example the agriculture and agro business sector in Sri Lanka, non-urban
regions in Egypt and previously disadvantaged groups in South Africa. (Tender document I, 2004)

2016

Table 3-1 Main admission criteria for PSOM and PSI
Admission criteria

PSOM (1999-2008)

PSI (2009-2015)

Project duration (months)


general

12-24

30



agriculture

36

36

€ 680,670 - € 1,500,000 (50-67%)

€ 1,500,000 (50%)

Max. project budget (% subsidy)

PSI Plus (60%)
Other reimbursements
Co-financing by other Dutch gov-

48

(1999-2004) ± € 7000 for non-

€ 20,000- € 40,000 MIGA insurance

selected qualified proposals

for PSI-PLUS

Allowed up to 80% total (incl. PSOM)

Not allowed

Preference for none for appli-

None for partnership, max. 1 run-

cant/local partner

ning project for applicant

Priority sectors, themes and groups

No priorities defined

49

ernment agencies
Previous PSOM/PSI funding
Country objectives

defined in MoU
Origins hardware

(1999-2002) 60% Dutch content

No requirements

Source: Tender documents 1999-2008 and Government Gazette 2009-2014

3.2.3 Assessment criteria
When an application was thought to meet all of the above mentioned admission
criteria, the next step was to assess the content and financial feasibility of the
proposed project plans. This assessment was made on the basis of application
forms and interviews with applicants and local partners during company visits to
the applicants and the local partners.50 Moreover, RVOs selection team consulted
local embassies and external experts, e.g. to assess the innovativeness and value
of the proposed technology. This external advice was not decisive but taken into
account by RVO and APSI.51 Those closely involved commented that these procedures not only expanded over time, but professionalised as well.52
Broadly speaking, the assessment criteria covered three main topics: (1) partnership, (2) project and (3) development impact.53

48

The size of the reimbursement varied between NLG 15,000 in 1999 to € 7,500 in 2004.
Source: interviews programme officers.
50
Sometimes applicants visited The Hague, local partners were always visited on site.
source: interviews RVO officers
51
There are examples when forewarnings were ignored (e.g. Uganda).
52
Interviews RVO
53
This classification is not used as such in the PSOM tenders, but it largely covers the assessment criteria used there.
49

Evaluation PSOM/PSI | 35

Partners
To assess the strength of the partnership, two main aspects were considered.
1. The ability of both partners to operate the project. This concerned their
financial and technical capacities. The project had to be in line with the
core business of both partners (so both partners would have a direct interest in a successful project) and each partners’ contribution had to be
balanced and match their financial capabilities.
2. The long term commitment of the partnership. PSOM/PSI officers assessed whether the partners had a trade relationship prior to the proposal. Central to this commitment was the (intention for) the establishment of a joint venture. Though the tender documents mention the
option of other forms of integration, in most cases a joint venture was established. Moreover, this form seems to have been actively encouraged
by RVO (e.g. through the defined project results).
The strength of the partnership is considered as one of the most crucial factors for
the success of PSOM/PSI projects. Over the years this criterion has been assessed
more strictly as many projects were halted in the implementation phase due to
partnership disputes54.
Projects
The second criterion was the project plan (including the business case with market analysis and cash flow analysis of the new activity). This was expected to contain clear, logical and measurable results on production outcomes, training, and
results for key parties in the production chain (including CSR outcomes) and a viable financial plan.
The project plans had to be relevant to RVO objectives. For example, to assess additionality, the application form included questions on whether the project would
be realised if PSOM /PSI funding were not available; whether the project could be
financed by the project partners; whether there would be no commercial funding
available; and whether the project would be eligible for funding under the Funding for Emerging Markets (FOM) of the Dutch Entrepreneurial Development Bank
(FMO). Moreover, the projects were expected to be innovative (and thus limit
market disruption). The answers to these issues would be cross-checked during
project and country visits (e.g. with local embassies and country experts).
Development impact

54

36 |

Source: Interviews RVO officers.

2016

The impact criterion focussed on the spin-off of the project and its development
effects. Applicants had to argue what the potential spin-off of the project would
be, in terms of follow-up investments and additional turnover, in case of the longterm establishment of a commercially viable company.
The operationalisation of the second impact criterion, development effects,
changed over time. During PSOM, the assessment of development effects focussed mainly on employment creation, knowledge transfer and no harm to the
environment. Furthermore, applicants would have to comply to CSR standards.
This was solely assessed by a check-box stating that the partners would follow the
OECD guidelines for multinational corporations and the ILO principles and rights.
With the introduction of PSI, partners would have to prove they had a CRS policy
in place or declare the intention to write one immediately after the start of the
project (which then became a project result). Moreover, more impact criteria
were added over time, e.g. on the environment, the position of women, local
communities and local and national authorities. In PSI Arab, specific attention was
given to the engagement with women and young people. After the programme
evaluation by Triodos Facet in 2010, RVO broadened the scope of the assessment
criteria from impact on the micro-level – e.g. within the company itself – to impact on the production chain, sector and society. As of 2012, PSI applications had
to contain a plan for responsible supply chain management following the OECD
2011 Guidelines for Multinational Enterprises.
3.2.4 Ranking
The assessment criteria formed the basis for the subsequent ranking of the applications. For all positively appraised proposals the selection team filled a score
card with scores for each of the assessment criteria. Subsequently, these proposals were ranked based on their relative scores.55 As the assessment criterion
changed, so did the ranking system. The ranking system became more comprehensive leading and based on an increasingly standardized selection procedure56.
In case the number of positive appraisals exceeded the available programme
budget, the final selection was made on the basis of the ranking. However, in
most tenders the available budget exceeded the budget required for the positively appraised projects (exceptions were 2011 and 2014). In other years, a ranking
was made, but did not have to be applied as the positively appraised proposals fitted within budget.

55

A separate ranking was made for PSI Arab projects (as this programme had a separate
budget).
56
Source: interviews RVO officers

Evaluation PSOM/PSI | 37

3.2.5 External advisory committee
All appraised projects (and some for which there remained a few doubts) were
subsequently evaluated by an external Advisory committee (APSOM/APSI). The
APSOM/APSI consisted of four people; one chairman and three members with expertise in finance, agriculture and production, and experience in developing countries. They were appointed by the Ministry of Foreign Affairs. In several sessions,
the commission discussed the projects based on the assessment prepared by the
RVO selection team. The APSOM/APSI provided a recommendation to RVO on
which projects to select (or not) 57. In general, this recommendation was followed
by RVO58.
The APSI could request additional information from the applicants or other parties, if the information provided was deemed insufficient for a well-founded
judgement (e.g. lack of information about the local context). In case this additional information was still considered to be lacking, the project was rejected.
Box 4. Role of consultants during application process
Of the 49 PSOM/PSI projects studied in depth, 32 used the services of consultants to
write the application. Similarly, the survey among applicants whose projects were rejected in the period 2010-2014 showed that 57% of them contracted an external consultancy agency for their application.
RVO has encouraged applicants to write the application themselves by organising
workshops during which the whole application format was explained in detail. On the
other hand, according to several of those interviewed, RVO has also at times facilitated
the use of consultants, e.g. by providing potential applicants with a longlist of possible
consultants. In some of the case studies, applicants had been approached by consultants providing information about PSOM/PSI or offering assistance with the application.
59
The Dutch consultancy firm, Advance Consulting, was used most often. Other were
Agriplan Subsidy Factory and Berenschot. Consultants received a success fee on a ’no
60
cure no pay basis’ . The case studies provide evidence that these success fees could be
as high as 15% of the PSOM/PSI grant, although the usual range lies between 4%-10%,
61
depending on complexity, risk and other factors.

57

The APSI received a yearly report on project progress and financial performance of all
projects. This report served as feedback on selection.
58
Source: interview APSI member and RVO officers.
59
According to their managing director Advance facilitated over 250 projects of which 2/3
from non-Dutch applicants. (Source: interview managing director Advance Consulting)
60
In recent years they additionally charged a commitment fee in case the partners quit in
th
the first phase after selection (Around 1/4 of the projects do not get finished).
(Source: interview managing director Advance Consulting)
61
Source: interview with Advance Consultancy.

38 |

2016

The consultants helped improve the project plans and write the application. During
project implementation they often continued to be involved, providing services such as
progress, final and financial reporting and sometimes mediation between the project
partners and/or RVO in case of disputes. According to the applicants of the case studies, the main reason for using consultants was the complexity of the application and
reporting formats, which are thought to be very distant from usual practice in business.
Among the rejected PSI-projects, 40% of the survey respondents experienced the application procedure as not transparent.
Based on the case studies, it is not possible to make a firm link between the use of con62
sultants and the success of the projects. However, the practice of using consultants
does raise questions about possible interference with the purpose of the PSOM/PSI
63
programme. For example, it might be more difficult to assess the strength of the
partners due to the consultants acting as intermediaries and the arrangements made
with consultants might well affect the projects (e.g. payment upon selection
with/without management fees thereafter might make a difference in projects supported by consultants). Moreover, the costs of using consultants risk being subtracted
64
from the grant amount available for the project even though officially, the costs for
drafting the application were not eligible for PSOM/PSI finance.

3.2.6 Approval
The final approval was granted by the team manager of RVO, generally in line with
the APSOM/APSI advice. RVO formally informed the Ministry of Foreign Affairs
(Directorate Economic Development, DDE) about their final selection.
In case of rejection, applicants received a formal statement to which appeal was
possible. If RVO considered the proposal to have potential, the letter included
recommendations for future tenders and hence acted as an active encouragement for resubmission, though without guarantee of being selected the second
time round.65 Obviously, applications which were rated negatively and had no
hope for improvement) were discouraged to resubmit.

62

On first sight there are among the 49 case studies less failed projects among the projects that used consultants (6/32 rather than 5/17 that did not). However, this sample
is not large enough to draw any conclusions on the effectiveness or selection of consultants
63
The extent to which these risks materialize could not be assessed in this evaluation due
to lack of information on the finances of the projects during implementation. The examples were provided in different interviews.
64
Sources: PSOM tender documents 1999-2008 and PSI Government Gazette 2009-2014
65
From the case studies emerged examples of applicants that were convinced they would
be granted PSOM/PSI the second time round as long as they precisely addressed the
concerns voiced in the rejection letter.

Evaluation PSOM/PSI | 39

Box 5: Reasons for rejection
The monitoring systems of RVO included only data on approved projects. Therefore, a survey was conducted among unsuccessful applicants whose application was not approved in
the period 2010-2014. More information on the survey is provided in annex VI.
The unsuccessful applicants who responded to our survey (63) identified several criteria as
the main reasons for rejection:
 Financing 21%
 Business plan 19%
 Development impact 16%
 Partnership 10%
 Unknown 14%
Lack of financial resources of the project partners was considered the most important reason for rejection, followed by the strength of the project plan (e.g. the lack of a local market).
In 2013 RVO also conducted a survey among rejected projects in the period 2005-2012.
They found that lack of financial resources (40%) and lack of a local market (30%) were the
most important reasons for rejection.
Interestingly, only 10 % of the respondents mention the strength of the partnership as the
main reason the proposal was rejected, while according to RVO programme officers the
strength of the partnership is one of the most important reason for failure. It is possible,
however, that this criterion is considered too sensitive (ambiguous) to be reported back to
applicants.
Additional reasons for rejection that were mentioned are changes in political context, lack
of PSI budget and sufficient own resources to finance the project without PSI (low ex-ante
additionality). In three instances the applicant or local partner themselves decided not to
proceed with the project.

40 |

2016

3.3

Relevance: PSOM/PSI objectives
Most projects visited complied with the main admission criteria set by RVO. However, the case studies do provide indications that in practice there have been projects that challenge some of the requirements. For example, in the case studies
sample, there are projects that are not in line with the core business of partners
(as required by PSOM/PSI), however, this does not automatically lead to a less
successful project (e.g. building company to be involved with agriculture). There
are also cases in which other government bodies (e.g. from Germany or Belgium)
provided financing for the same projects (which was allowed for PSOM, but not so
for PSI).66
Furthermore, the requirement that applicants should have no more than one ongoing project under PSOM/PSI, nor have had previous projects under PSOM/PSI,
was not always respected.67 Local partners were allowed to access multiple
PSOM/PSI grants. However, the way in which in the random sample of projects
visited in Sierra Leone, three out of the eight projects benefited the same local
partner, all rewarded in 2009 to different local companies but with the same
owner, seems not to match PSOM/PSI intentions.68
With regard to the specific PSOM/PSI objectives (rather than admission criteria),
all case study projects were considered potentially relevant (though to a varying
extent, depending on the fulfilment of each different criterion). To be considered
relevant, the selected projects had to fit the objectives of PSOM/PSI:69
 Based on strong partnerships
 Innovative
 Potential development impact
 Potentially commercially and financially feasible

66

In 4 cases funding from other governmental donors was mentioned as a source of investment (grant or loan). Two others mentioned other donors as a potential source.
PSI Government Gazette 2009-2014: ‘Cofinanciering van het project door andere programma’s van de Nederlandse overheid of van andere overheden is niet toegestaan’.
67
This requirement was initiated officially in 2011. Before that, PSOM/PSI voiced a ‘preference’ for applicants that had not yet made use of the programme (‘de voorkeur gaat
uit naar aanvragers die nog niet eerder een PSOM of PSI-project hebben uitgevoerd’
Source:PSOM tender documents 1999-2008 and PSI Government Gazette 2009-2014).
Local partners were allowed to benefit more than once from PSOM/PSI.
68
Similarly, in Peru one entrepreneur managed three consecutive grants for interlinked
PSOM/PSI projects focused on the same produce. There were indeed three different
applicants, but these did not play a decisive role in the project.
69
Sources: PSOM tender documents 1999-2008 and PSI Government Gazette 2009-2014.

Evaluation PSOM/PSI | 41

Because the relevance is assessed based on current situation of projects, there is
an overlap with the evaluation questions on effectiveness (extent to which the
projects were innovative) and impact (extent to which the projects had a development impact).
3.3.1 Partnerships
Of the 49 projects visited, 36 originated from existing partnerships between the
applicants and the local partners. For example, often PSOM/PSI was used to expand existing trade relations or buyer-producer relationships (the international
applicant being either a supplier or a buyer). As discussed, RVO country managers
are convinced, based on their experience, that a pre-PSOM/PSI relationship is crucial for the success of projects. In the sample, an equal proportion of projects
failed whether they were based on existing partnerships or not (22%-23%). However, we need to keep in mind that the sample does not include partnerships that
stopped during or after the PSOM/PSI period.
In most cases visited, the relationship between the applicant and the local partner remained good after the grant period (20 out of the 30 finalized projects). The
strength of these partnerships is difficult to measure. However, the case studies
do provide insight into the different types of partnerships linked to PSOM/PSI.70
At times, these relationships were actually very personal and affectionate, comparable to longstanding friendships and bonds between families that withstand tests
due to often difficult circumstances (e.g. delays, financial crisis etc.). In other cases, however, the partnerships seemed to be different than intended, for example
(see box 4 for illustrations)71
 Unequal partnerships, e.g. when the local partner is no more than a production unit of the applicant or when the applicant is the sole client of the local
partner, which poses questions about the sustainability of the PSOM/PSI
project.
 Artificial partnerships, which according to project partners interviewed were
set up with the sole purpose of meeting the PSOM/PSI requirements and are
expected to be quickly dismantled as soon as the PSOM/PSI grant period
ends.
 A sub-category of these artificial partnerships (‘one-sided partnerships’) are
those in which the local partner dominates, while the different international
applicants serve primarily to qualify for PSOM/PSI and play a minor role in
the project (even though they are formally responsible for the PSOM/PSI
70

The strength of these partnerships was not a topic of the evaluation according to the
terms of reference, hence no amounts can be provided. However, from interviews
emerges clearly that there are variations between projects, as discussed here below.
71
The amount of cases to which this applies cannot be provided as this was not the focus
of the evaluation but an unexpected result.

42 |

2016

grant). This is not necessarily a reason for project failure, but does limit potential benefits of international collaboration (e.g. knowledge transfer) that
is part of the design of the PSOM/PSI programme.
Though officially the PSOM/PSI requirement was a ‘durable long-term collaboration’,72 without specifying the form this should take, most PSOM/PSI collaboration
was implemented through a joint venture structure. The case studies indicate that
for some projects other types of collaboration (e.g. strengthened trade relations)
might have been more suitable for the intended cooperation and the project implementation during the project period and thereafter.73 The findings from the
spin-off survey that RVO held in 2016 underline this finding. According to respondents, a strong partnership is crucial for the success of the businesses, however, the form of the partnership should be adjustable to the specific situation of
the partners themselves.74
There are mixed perspectives on how flexible RVO was to allow projects to select
the format that best fitted their business (depending on applicants and programme officers). From the case studies emerges evidence that the ‘joint venture’
has at times been interpreted loosely to match the purpose of the project partners, e.g. the applicant and local partner actually being the same entity, or the local partner being in fact a fully-Dutch owned company which – together with the
Dutch applicant – buys all products made. However, others interviewed stated
that the joint venture structure seemed to have been obligatory.75
Box 6. Examples of partnerships
Stronger together
Both the applicant and the local partners are family companies. Apart from
shared values, collaboration is very close due to clear mutual interests. The
Egyptian side aims to gain to access technology and serve the local market in a
timely and cost-effective manner, while the Dutch partner is now able to concentrate on the more complicated engineering works. This way the mutual clients in Egypt and beyond are optimally served by both parties in conjunction.

72

‘De partners moeten een samenwerkingsverband aangaan voor de lange termijn. Dit betreft veelal een joint venture.’ Staatscourant 2014
73
The suitability of the joint venture structure has not been assessed in this evaluation,
however, this issue came up in interviews with applicants and project partners and is
confirmed through the spin-off survey.
74
Spin-off report 2016, p. 37
75
Case study interviews. This question was not assessed in all interviews (hence not possible to provide exact numbers) as it was not part of the terms of reference but
emerged as an unexpected result.

Evaluation PSOM/PSI | 43

Unequal partnership
The joint venture in Bosnia-Herzegovina actually operates as the production
unit of the Dutch applicant (its only customer), and cannot even sell its product
on the local market without approval from the Netherlands. The local management also describes itself in terms of total dependency and its role as to
produce what the applicant orders. While the local partner is accepting of this
role, it does not seem to be in the spirit of the PSOM/PSI programme. In another case in Uganda (horticulture), the applicant was the client of the local partner, which did at time pose challenges to the partnership due to competing interests (e.g. with regard to prices).
Artificial partnership
To cultivate grapes in a desert area in Peru, two financially very strong partners
prepared a project proposal and accepted to establish a Joint Venture as they
were of the opinion this was an admission requirement. However, from the onset of the project assets acquired with RVO grant were considered property of
the local partner, who also takes all decisions related to expansion of area under cultivation. Formally the Joint venture still exists, but it is seen as a cost
centre by the local partner, while the Dutch partner declared that it ‘is not interested in its formal share neither dividend as long as my Peruvian partner
supplies us with fruits’.

Non-Dutch applicants
The number of Dutch and non-Dutch applicants is depicted in Figure 3-2. Overall,
approximately 25% of the granted projects have non-Dutch applicants and the
remaining 75% of projects have a Dutch applicant. Of the 49 projects visited, 38
had Dutch applicants and 11 had applicants from other nationalities.
The share of projects with a non-Dutch applicant has increased since 2009. Increased promotional activities by RVO could explain the increased share of nonDutch applicant from 2009 onwards.

44 |

2016

Figure 3-2: Number of applicants, Dutch and non-Dutch
140

Number of applicants

120

100
80
Non-Dutch
60

Dutch

40
20

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

0

N=1107, all selected projects
Source: RVO monitoring data, analysis: APE

Table 3-2: Applicant (n = 2912 PSOM/PSI applications)

Dutch applicant
Non-Dutch applicant
Total

Total
PSOM/PSI Selected
sample
2225
76%
856
77%
687
24%
251
23%
2912
100%
1107
100%

Rejected
1369
436
1805

76%
24%
100%

Source: BAS monitoring data

In the overall sample, the majority of non-Dutch applicants that are selected for a
grant come from countries other than the PSOM/PSI focus countries (61%), most
often the United States.76 In the sample of 49 projects visited, non-Dutch applicants originated from China, Guinee, Ghana, Switzerland, Italy, Egypt, Kenya, Zimbabwe, and the United Kingdom.

76

In total 22 companies from the US were selected (out of 68 applications).

Evaluation PSOM/PSI | 45

Table 3-3: Non-Dutch applicant from a PSOM/PSI country (n = 303 PSOM/PSI
applications)
Total

Selected

Rejected

Africa

163

54%

49

50%

114

56%

Asia

60

20%

19

19%

41

20%

Central and Eastern Europe

0

0%

0

0%

0

0%

Latin-America

34

11%

16

16%

18

9%

Mena

46

15%

14

14%

32

16%

Total

303

100%

98

100%

205

100%

Source: BAS monitoring data

The case study shows no relationship between the nationality of the applicant and
the subsequent success of the projects (2/11 projects with non-Dutch applicants
failed commercially as did 9/38 of the projects with Dutch applicants). From interviews emerges that there is less interest from Dutch embassies for the PSOM/PSI
projects without a Dutch connection (though there are exceptions, e.g. the support provided by the Ugandan embassy to a project with an Egyptian applicant
and Kenyan-Egyptian local partner).77
Financial capacity local partners
With regard to the local partners, the different case studies unearthed a particularly interesting issue. The evaluators considered it striking that in Uganda the local partners were often not Ugandan entrepreneurs but rather foreigner residents in Uganda. The context analysis and interviews with local partners implied
that this happened because the size of the PSOM/PSI grant, and the matching
fund requirement, is too large given the limited financial capacity of Ugandan entrepreneurs. This capacity is said to have been constraint due to the relatively recent development of an entrepreneurial culture in Uganda, where the higher educated and more capital endowed population groups used to prefer formal
employment rather than starting businesses (considered for the poor). While this
could in principle also apply in Sierra Leone (given the post-conflict context that is
not inductive for entrepreneurism), the limited financial capacity of local partners
was tackled by loans provided by the applicant to local partners, allowing them to
participate despite lack of financial resources. It remains unclear why the Sierra
Leone entrepreneurs did take this risk and Ugandan entrepreneurs did not.78 In
middle income countries as Peru, Bosnia Herzegovina and Egypt, the financial capacity of local entrepreneurs is generally stronger (hence mainly partnerships with
nationals).
77
78

46 |

Interviews – The local partner was registered in Uganda.
e.g. capacity of local entrepreneurs, advice by RVO.

2016

3.3.2 Innovativeness
The case studies confirmed that ‘innovative’ in the PSOM/PSI refers to being ‘new
within context’79 rather than being ‘advanced or original’, which is the standard
interpretation of innovative. Most projects were new for the local partner (company), new for the sector and new for the country (Table 3-4). For example, a PSI
grant was provided for the new cargo hall at the main airport of Sierra Leone. The
cargo hall in itself was not particularly innovative (e.g. with regard to building, energy usage, storage facilities process management), however, it was the first cargo hall in the country that met the EU requirements to handle exports from Sierra
Leone to the European Union.
Table 3-4: Innovative projects
Innovation
BD
Not new
New for the company
1
New for company and sector
2
New for company, sector and 5
country

BiH
1
7

P (9) SL
3
1
2
2
3
1
5

EG

UG
1

2
5

7

Total
5
8
5
31

Source: Case studies

One of the cases illustrates nicely why newness might not necessarily be what fits
PSOM/PSI objectives. In this case, the production methods or equipment were
admittedly new for the company, sector and country, but actually not well suited
for that context. Agricultural equipment imported to Africa from Europe turned
out to be unsuitable for the local circumstances (e.g. heat, soil conditions, technical capacity). Operators had to remain in daily contact with the supplier and the
applicant in the Netherlands, but nevertheless the equipment could not be used
to full capacity. As such, their inappropriateness for the context might have explained why they were not used before and thus these projects should actually
not be considered as particularly innovative even though new. It might have been
more appropriate to assess more critically the technical requirements taking local
conditions into account before a final investment decision was taken.
3.3.3 Potential development impact
PSOM/PSI aimed to select projects with a potential development impact and
based on the applications forms of the 49 case studies, almost all of the projects
visited seemed to have this potential, at least at the sector level. Beyond direct
79

According to the regulations published in the Staatscourant, 2014, ‘Het project is significant vernieuwend voor het betreffende land. Het innovatieve karakter dient tenminste
het type product of dienst, de productiemethode of de dienstverleningswijze te betreffen.’

Evaluation PSOM/PSI | 47

employment creation, this refers to effects within the chain (suppliers, outgrowers, consumers) as well as broader societal effects (e.g. import substitution,
changes in business climate).80 In the case studies the potential for development
impacts was assessed during the field visit and interviews (chapter 6).
39 out of 49 projects visited provided evidence of potential effects within the sector. In particularly through:
 Outgrower schemes
 Local suppliers
 Traineeships
 Increased opportunities for customers (e.g. transport)
19 out of 49 projects visited provided evidence of potential effects beyond the
sector and within the broader market or country. This would occur for instance
through:
 Improved food security
 Import substitution
 Country reputation (and business climate)
7 projects had less potential for impact beyond the companies directly involved.
This was the case, for example, when the local company risked being no more
than a production unit of the applicant, which was the sole client.
Box 7. Examples of impact
Sector impact
The investment made in a pilot plot to cultivate Pomegranates has attracted interest from
neighbouring farmers. In two years the number of outgrowers has increased from the originally planned 5 to 9 and one more showing interest. The outgrowers have indicated, however,
that the organic way of cultivating this fruit has to be abandoned because of financial considerations.
Market impact
The peanut shelling project in Egypt will not just provide business opportunities to scores of
local farmers (outgrowers). There is a substantial peanut processing industry in Egypt which
will now be able to source peanut kernels locally, hence substitute imports hence save forex.
3.3.4 Commercial and financial viability
With regard to the commercial and financial viability of the subsidised projects, all
of the case study projects seemed potentially relevant based on the descriptions
80

48 |

Sources: PSOM tender documents 1999-2008 and PSI Government Gazette 2009-2014

2016

of projects in the application forms. However, document review of the proposals
reveals that many proposals are overambitious, e.g. with overly optimistic IRR and
sales projections, rather than solid business cases more in tune with the size of
operations and capacity of applicants. The focus of the assessment process by
RVO was primarily on financial ratio analysis and assessment of track records,
with regard to the applicant and (sometimes) the local partner. From the documents review we note less consideration for the true business case (and related
sales and financial plans) of the proposed new joint venture/project. Box 8 describes which financial indicators predict a successful application.

Evaluation PSOM/PSI | 49

Box 8. Profile selected and rejected partnerships: financial characteristics
The analysis of the selection process identifies a clear distinction between the financial characteristics of
rejected and selected partnerships. Table 3-5 shows the mean values for a number of financial indicators
for rejected and selected partnerships. The table only shows indicators for which the differences between the two groups were statistically significant (based on bivariate analysis). Most significant variables concern financial indicators of the applicant. Applicants in selected partnerships are more profitable,
have a higher turnover, larger cash flow, and more favourable equity and financing ratios than their rejected counterparts. For instance, the average profits reported by selected applicants was € 8.7 mln,
whereas the average profits reported by rejected applicants were € 6.2 mln. The only indicator for which
the difference between rejected and selected local partners is significantly different, is the equity of the
local partner. Selected local partners reported equity of on average € 1.1 mln, and rejected local partners
reported on average € 0.6 mln.
Interestingly, the expected internal rate of return (IRR) is significantly higher in rejected projects. This indicates projects were rejected because they were deemed too optimistic about future revenues.
The full list of indicators is shown in table A. 5-2 in annex VII. This table shows that the there are no statistically significant differences between the average division of shares between rejected and selected
projects, and no statistically significant differences between the means of most local partner financial indicators. Due to multicollinearity only the financing ratio of the applicant remains statistically significant
when the indicators are included in a multivariate analysis (see table A. 5-3).
Table 3-5: Financial profile selected and rejected projects
Selected
Variable

Rejected

N

Mean

N

Mean

499

€ 1.090.939

115

€

515

8,47

122

5,74

Financing ratio applicant 514

4,92

123

0,94

Equity local partner

499

€ 1.090.939

115

€

640.245

Turnover applicant

510

€ 8.748.207

123

€

6.210.224

Net profit applicant

505

€

413.235

125

€

148.566

Cash flow applicant

513

€

614.258

125

€

237.650

Own contribution

510

€

295,084

126

€

268,129

516

15,98%

155

18,16%

Local partner
Equity local partner

640.245

Applicant
Equity ratio applicant

Project characteristics
Internal rate of return

Financial indicators predict the chance of successful application
The importance of the financial position of the application is also apparent when we analyse the determinants for a successful application. Using logit type regression analysis, the effect of proposal characteristics on the probability of being selected can be estimated (see Annex V for the methodology). The regression includes, among others, financial ratios, project characteristics (IRR, division of shares), and
region. Of these variables, what mattered in the selection was the applicant’s net profit, cash flow, financing ratio and solvency ratio (strong correlation with selection), in particular the financing ratio. Project specific variables (such as the division of shares or the IRR) were not significant. Projects from Latin
America had a relatively higher probability of being selected.

50 |

2016

3.4

Relevance: Country objectives
The terms of reference for this evaluation included the evaluation question of
whether the supported investments were in line with the objectives and priorities
of the PSOM/PSI country.81 This is a valid question to be posed for each development intervention funded by one government on the territory of another state.82
However, PSOM and PSI were set up and implemented as project-focused and
demand-led programmes rather than country programmes, and as a result paid
little attention to specific priorities in PSOM/PSI countries83. In the countries visited for this evaluation there has rarely been any contact between RVO and national governments or other relevant bodies (e.g. Department of Trade and Industry,
Chamber of Commerce).84 Indeed, local government was mostly unaware of the
PSOM/PSI work.
On the one hand, the country studies conducted for this evaluation do indicate
that the portfolio of PSOM/PSI is diverse enough to fit within country priorities related to PSD, without explicitly doing so. On the other hand, the evaluation does
raise questions about having one programme design to fit some very heterogeneous country contexts. The support required by companies in Peru is very different
from the needs of companies in Sierra Leone, for example, whereas the selection
criteria and process, as well as the management of the programme, are in principle identical.
Moreover, not only was PSOM/PSI not strategically aligned with host country objectives, it was also not explicitly aligned to the objectives of the Dutch government in those countries (e.g. selection of sectors). As a result, cooperation and
support between embassies and RVO varied a great deal over time and between
countries.85 This was a missed opportunity, e.g. to improve monitoring of projects
and to support projects by linking them to other Dutch initiatives.

81

Ministry of Foreign Affairs, June 2015, Terms of Reference Evaluation PSOM & PSI, version #5
82
OECD DAC guidelines define relevance as the extent to which the objectives of a development intervention are consistent with beneficiaries’ requirement, country needs,
global priorities and partners’ and donors’ policies. OECD, 2002, Glossary of Key Terms
in Evaluation and Results Based Management.
83
At the start of the programme MOUs were signed with target countries. However, this
was dropped in favour of the more demand driven approach.
84
Assessed through interviews in country (e.g. Chamber of Commerce, Ministry of Trade
and Industry…) and RVO
85
Interviews and case studies

Evaluation PSOM/PSI | 51

3.5

Ex-ante additionality
A project can be evaluated as ex ante additional if it would not have been realized
on the same scale, at the same pace and with the same effects and impact without the financial contribution of PSOM/PSI. Only when PSOM/PSI support is additional to what the private sector has to offer, can the public support be seen to
add value and can PSOM/PSI support be potentially beneficial (if not, PSOM/PSI
would merely replace private funding or in the worst-case crowd out private initiatives). 86
According to the terms of reference for this evaluation, the following aspects of
additionality were to be considered: 87
 Financing: Is the applicant unable to finance the project (within a reasonable
time frame) with own funds or with funding from third parties (e.g. commercial parties such as banks, or even other donors)?
 Risk: Is the applicant unwilling to implement the project because partners
perceive the risks as too high?
 Market distortion: Does the project supported by PSOM/PSI not risk disadvantaging other companies already operating or ready to enter the same
market?
Box 2 describes the way in which the evaluators assessed ex-ante additionality in
the case studies (chapter 1.2.2). To this information, the results of the survey with
rejected applicants is added to evaluate the additionality of PSOM/PSI.
3.5.1 Financing
RVO paid a lot of attention to the financial strength of the applicants and to some
extent also to the financial strength of the local partners.88 The aim of the programme was to target companies that would be strong enough to sustain the project in the longer run, match the PSOM/PSI grant with own investments during
the grant period and build on it thereafter (e.g. through follow-up investments).
From the country case studies follows that PSOM/PSI applicants and local partners mostly relied on own resources to finance the proposed projects. Table 3-6
shows the financing of PSI projects. In about 19% of selected PSI proposals third

86

Heinrich, 2014
Based on the terms of reference for this evaluation. There are more possible components of ex ante additionality as described in Heinrich, 2014.
88
Interviews and document reviews.
87

52 |

2016

party involvement89 was proposed with an average investment of about € 175
thousand (14%), significantly less than own or RVO contributions.90
Table 3-6. Financing PSI projects
Average contribution

PSI

N

Contribution RVO

€ 676,507

617

Contribution applicant

€ 290,969

617

Contribution local partner

€ 286,677

617

Contribution third parties

€ 175,535

109

€ 1,298,360

617

Total

91

Source: BAS monitoring data

Other sources of finance
Most partnerships reported difficulties accessing commercial financing. Moreover,
for PSOM/PSI as a whole, follow-up investments were much lower than targeted.
The innovative (or even greenfield) nature of some of the PSOM/PSI projects as
well as their location, results in limited access to commercial financing (project
and country risk). In some countries local loans were considered too expensive
anyway. For example, in countries like Sierra Leone and Uganda the extremely
high interest rates (around 25%) and collateral requirements of commercial banks
were said to make commercial financing inaccessible.
However, several of the applicants from high income countries as well as local
partners in the middle income country Peru, stated that they would have made
the same investment if funding from RVO had been a form of soft loan instead of
a grant. Companies would have accepted subordinated loans, financing constructions based on performance in which loans could be converted into grants in case
of failure, or shares or short term loans (based on predefined criteria and valuation calculations).
Those interviewed for the case studies were asked about alternative sources of
support for the PSOM/PSI project (potential and actual). These included, among
others, support from the Belgian Investment Company for Developing countries
(BIO)92 and KfW Development Bank from Germany. Table 3-7 provides an overview of those alternative sources of funding and technical assistance mentioned
by applicants and local partners.

89

109/586 selected projects
Source: APE review of BAS monitoring data
91
The averages do not add up exactly due to missing data for some projects
92
http://www.bio-invest.be/
90

Evaluation PSOM/PSI | 53

Table 3-7. Overview of actual and potential alternative sources of support
comparable to PSOM/PSI as mentioned by applicants and local
partners in case study countries
Equity

Bank loans

Soft loans

Technical Assistance

Private investors
Africa Century
West Africa Venture Fund
Aavishkaar Investment Fund
NGO investors, e.g. First Step (SL)
Ugandan Development Bank
Ugandan Agriculture Credit Facility
Rabobank
Triodos bank
Local banks
Stichting DOEN
FMO
Belgian government BIO
KfW Development Bank / German Investment Cooperation
French PROPARCO
SNV
CBI (RVO)
PUM NL Senior Experts
GTZ
EU Centre for the Development of Entreprise

Out of the 63 survey respondents whose application was rejected and not accepted in a latter round, 16 (25%) managed to carry out the project without PSI support. Most of them executed the project at a smaller scale or slower pace (10) and
some of them worked together with a different local partner (3). As for the financing, 9 projects were fully financed by the partnership itself. The others used either
a subsidy/soft loan from organisations / bodies such as the German Government,
the German Investment Cooperation (2), or a privately financed loan (3). Interestingly, 6 projects that were rejected based on their financial status did manage to
finance the project themselves (4) or obtain a private loan (2).93 47 projects (75%)
were not undertaken. Not surprisingly, lack of financing was most often cited as
the main reason (27). Second was the partnership (8) - most often the local partner did not want to proceed without PSI funding.

93

54 |

be it at a smaller scale (3), slower pace (3) or with a different local partner (1). In two of
those cases the respondent indicated that the project was rejected because RVO did
not consider additional funding needed

2016

3.5.2 Risk
PSOM/PSI support may also be considered additional if it allows the applicant
(and local partner) to overcome impeding risks because of country context or unknowns related to the project, as is often the case with innovation. In the survey
of rejected applicants, the third reason for not pursuing the project was the perceived local political risk. This does, however, not imply that all grant support to
fragile countries, for example Sierra Leone, is automatically additional. Entrepreneurs may have access to other means to reduce risks (e.g. building up experience
in the country, guarantee schemes, subordinated loans).
PSOM/PSI has provided grants to companies in about 60 countries. About 10% of
the granted projects were in so-called fragile states. 94 In such countries countryspecific risks are relatively high and companies might be hesitant to invest if they
have not been active in those countries before. In those cases, the involvement of
PSOM/PSI can be seen as a means to reduce risk and stimulate additional investments.
However, PSOM/PSI also provided grants for projects in less risky countries,
among which Peru and other non-conflict, middle income countries. In these
countries, additionality due to country-risk reduction is unlikely. As Table 3-8 illustrates, a country like Peru has a relatively high ‘ease of doing business’, including
access to financial resources at lower costs (from commercial banks, investors,
and own sources) and highly educated and experienced entrepreneurs. A country
like Sierra Leone battles with low ease of doing business and very high risk premiums on lending, thus a context where it is more likely that PSOM/PSI is additional
through its risk reducing capacity.

94

If we look only at the approved projects under PSI, then 22% of the projects occur in
fragile states. This is approximately the same percentage as the 20% share of PSI Plus
(focused on fragile states) in the total of accepted PSI projects.

Evaluation PSOM/PSI | 55

Table 3-8. PSOM/PSI countries country risk

GDP/capita (US $)

Bangla-

Bosnia

desh

Herz.

Egypt

Peru

Sierra
Leone

Uganda

1,086.8

4,851.7

3,365.7

6,549.4

792.6

714.6

a

172

82

126

45

147

135

Risk premium on lending (%)

5.8

-

0.1

-

17.0

11.3

Ease of doing business index
b

Fragile state

a the higher the index the more difficult doing business
b lending rate minus treasury bill rate
Source: World Bank
3.5.3 Market distortion
To ensure additionality an intervention should not distort the market by giving
grantees unfair advantage over other companies active or potentially active in the
same area. The extent to which there were comparable projects that were disadvantaged because of the support from PSOM/PSI was difficult to assess within the
scope of this evaluation.
One way in which PSOM/PSI sought to avoid market distortion was by supporting
innovative projects. As discussed in section 3.3.2, the ‘newness’ of projects varied,
but the majority of projects supported were new within the PSOM/PSI country
context (i.e. new for the sector and/or country). In the cases in which the project
supported was not ‘new’ (5/49), there was in principle a risk of market distortion
by providing one company with an advantage over its competitors. Where the
project supported was ‘new for the company’ (8/49), there was also a risk of distorting the market as those projects allowed one company to gain an advantage
over others. Country studies identified a few cases in which PSOM/PSI supported
projects, while there were existing local producers doing the same thing and applying the same technology.95
However, even in the cases for which the project was new only for the applicant,
in principle all potential competitors could have successfully applied for PSOM/PSI
as well. The fact that PSOM/PSI was open for any partnership meeting the entry
requirements, reduces the risk of unfair competition (compared to cherry picking
of specific companies). It was up to companies to find out about the programme 96

95

It was difficult for those interviewed to recall whether there had been competition at
the time of the PSOM/PSI application rather than currently. However, in seven cases
those interviewed stated that there had been competitors.
96
At the moment of introduction in a (new) country PSOM/PSI was promoted through
presentations and seminars at embassies, chambers of commerce, business associa-

56 |

2016

and successfully apply, everyone had an equal chance if they fulfilled the selection
criteria (e.g. find an international partner to apply). Since 2003, applicants could
originate a list of other countries than the Netherlands (so-called ‘untied) and no
link with the Netherlands was even required, which resulted in several truly international PSOM/PSI projects, with even less risk of market distortion.
Country case studies show several cases in which PSOM/PSI funding had an effect
on competition, mainly by crowding out of smaller scale production (e.g. less mechanic, smallholder farming). However large the impact is thereof, this is not necessarily market distortion in the longer term.
3.5.4 Conclusions on additionality
Additionality is hard to prove or measure. There is an incentive for the applicants
to claim additionality since it is a requirement for funding, but it remains difficult
to (dis)prove or (in)validate such. This would require, for example, proof of having
tried to access other sources of funding. Moreover, the extent to which PSOM/PSI
funding is additional might change over time (e.g. once accorded, more alternative options become available).97
During the application process additionality was assessed by RVO on the basis of
self-assessment by the applicants and additional advice from local embassies. In
the case studies, the extent to which projects funded were indeed additional was
validated through desk review, interviews and company visits. Project applicants,
local partners and managers were asked about other sources of funding used and
alternative sources, and explicitly about what they would have done without
PSOM/PSI. A reference question was whether the project partners would have accepted a soft loan rather than a grant. Where appropriate and relevant, the interviews with external stakeholders and document review were used to assess the
plausibility of answers.
As shown in Table 3-9, in the majority of projects, 30/49, the PSOM/PSI contribution is considered additional by addressing financing needs and investment risks in
specific countries.
Table 3-9. Additionality
B

BiH

P (9)

SL

E

U

Total

Additional

6

5

3

3

7

5

30

Doubts about additionality

2

3

6

5

1

3

19

tions, through advertisements and through promotion in the existing company network of RVO, both in the Netherlands and abroad (Interviews).
97
Hence the focus on ex ante additionality

Evaluation PSOM/PSI | 57

a. Source: Case studies

For the remainder of projects (19/49) there is doubt about whether PSOM/PSI
was additional. This is mainly because there are indications that the applicant
and/or local partner would have had access to other sources of financing (e.g.
companies well provided with capital, limited country risks, and evidence of bank
loans). Another reason to doubt the additionality of PSOM/PSI support is projects
supported were set up overly ambitious (e.g. with regard to buildings, training
costs, employment) in response to the available funding from PSOM/PSI. This is
the case for 6 projects (3 in Bosnia Herzegovina, 2 in Sierra Leone and one in
Egypt).

58 |

2016

Box 9. Examples of additionality
Additional contribution PSOM/PSI
A project in Uganda required PSOM/PSI support to establish a completely new
undertaking with large start-up costs (e.g. fish ponds) and significant lag time
before income could be generated (fish production). At the start of the projects, no other banks or investors were interested given high perceived risks.
However, once established and profitable thanks to early support with
PSOM/PSI grants, investors have been found to scale up the business.
Doubt about additionality PSOM/PSI
All but two of the local companies visited in Peru proved to be financially
strong and could have raised the necessary funds themselves either from own
resources or from local financiers. The PSOM/PSI grants funds speeded up decisions to start new ventures in three cases, but all local companies could have
repaid the funds eventually. The two flower growing companies supported are
illustrative. One receiving a grant in the early years under PSOM belonged to
an international company and the second one (PSI) was managed by a strong
(family owned) local company.
Too additional
In Bosnia Herzegovina there were several projects where major constructions
works were carried out simply because the grant money was there and because according to the local partner “RVO liked it”. The fact is that in this
country there is ample industrial infrastructure available, remaining from the
socialist past, which might have been converted and used (as other PSOM/PSI
projects did). In one of the cases, the company should clearly have rented
amply available space rather than construct a new one, which would also have
reduced the start-up problems experienced due to incompetent builders.

3.6

Conclusions on relevance
The PSOM/PSI programme is considered relevant with regard to the selection of
projects in light of the overall objectives of the programme:
 The strength of the partnerships varied between countries and projects, but
in 20 out of 30 finalised projects visited for the case study the relationship is
still ongoing and positively appreciated. Among the successful projects that
were surveyed by RVO for their spin-off analysis, 80% were still run by the
same partnership.
 The majority of projects are new for the company, sector and country (31/49),
which indicates that PSOM/PSI indeed targeted projects with elements of innovation or at least ‘newness’. Newness should also be suitable for the context, which was not always the case. A project can be new to the company,

Evaluation PSOM/PSI | 59





sector, country or even the world. Some projects are truly innovative, others
show that this claim is exaggerated and cooked up to meet the PSOM/PSI requirements. The innovativeness criterion was also important to prevent unwanted market distortion. The majority of projects supported were new within the PSOM/PSI country context. In the cases in which the project supported
was not ‘new’ (5/49), there was an ex-ante risk of market distortion by providing one company with an advantage over its competitors. Where the project
supported was ‘new for the company’ (8/49), there was also an ex-ante risk of
distorting the market as those projects allowed one company to gain an advantage over others.
Potentially, most projects have an impact on the value chain (sector) or wider
market (country), beyond the company and those directly involved, except for
those projects that were set up solely to serve the applicant (e.g. production
units).
Selected projects all have the potential to be commercially and financially
successful. However, in the selection process there is a tendency to pay specific attention to the financial strength of the applicant. The financial profile of
selected projects differs from rejected projects on a number of indicators. The
largest differences are found in the financial profile of the applicant company.
This is confirmed by multivariate analysis, which shows that the financial ratio
of the applicant is the only characteristic that significantly affects the selection
probability. Local partner financial indicators and the division of shares do not
differ significantly between selected and rejected projects. Monitoring reports
also deal more with project plans and predefined activities financed with the
grant funds rather than overall business operations, its evolution (partly determined by external factors as well as by the entrepreneurial decisions taken
over time in reaction to such developments) and expected business results of
the projects (sales are discussed only in the final phases of the projects).

For most projects (30/49) the PSOM/PSI grant is considered additional as source
of funding (or for the reduction of risks). This is, however, strongly dependent on
the country’s context, whereby in middle income countries most PSOM/PSI projects are likely to have happened anyway, regardless of the subsidy. For 7/49 projects additionality is doubtful, with investments being larger than necessary due
to the ‘perverse’ availability of the grant funding and the tendency of applicants
to ask the maximum. All in all, most projects would not have been realised on the
same scale without the financial contribution of PSOM or PSI.
PSOM/PSI is, however, less relevant with regard to country-specific needs. As
both PSOM and PSI were set up as project-support rather than country programmes, there is by nature little alignment with country-specific priorities of national governments or even programmes of Dutch embassies. On the contrary,

60 |

2016

one programme design is used for all countries (apart from PSI Plus countries),
ranging from middle income countries like Peru to fragile low income countries
like Sierra Leone. A more country-specific approach could have improved the relevance of the portfolio (e.g. innovativeness, potential development impact, additionality and market distortion).
Moreover, lack of alignment with other private sector development programmes
(from the Netherlands or elsewhere) is a missed opportunity, in particular with
regard to the possible synergies between PSOM/PSI and other Dutch programmes
for private sector development and economic diplomacy. PSOM/PSI could also
have used its presence in a country as a provider of significant grants to the private sector to leverage general improvements in business environment, including
addressing entrepreneurial capacity.

Evaluation PSOM/PSI | 61

4
4.1

Efficiency
Introduction
In this chapter we analyse the efficiency of the implementation of the PSOM/PSI
programme by RVO. We start by looking at programme expenditure and operating costs. Second, we present the developments in the PSOM/PSI project portfolio, the number of staff needed to manage the portfolio and the number of countries in which the programme is operated. Third, we present the developments in
several productivity indicators both on programme management as well as on
project management. Lastly, we analyse the main cost drivers that influence operational efficiency. 98

4.2

Programme expenditure and operating costs
The PSOM-PSI programme has expanded considerably since its introduction in
1999. Figure 4-1 depicts the trend in programme expenditure. The programme
expenditure is the amount of subsidy disbursed annually by the PSOM/PSI programme. During the PSOM period, annual programme expenditure increased
from € 2.1 million in 2000 to 24.5 million in 2008, with a dip in 2004. This was the
year in which PSOM was integrated with PSO99 and EVD merged with Senter International (this resulted in fewer tenders being set out) 100. During the PSI period
programme expenditures more than doubled from € 24.4 million in 2009 to € 56.8
million in 2014.
Figure 4-1 also shows ratio of operating costs and programme expenditure. The
latter shows how many cents were needed to manage 1 euro of PSOM/PSI subsidy. The ratio declined from 0.28 in 2000 to 0.09 in 2015, which means that in in

98

The analyses in this chapter are based on information from the annual reports 20002015. The annual report of 1999 was not available. For some of the indicators information was only available for the years 2004-2015 or 2004-2014.
99
The Programme for cooperation with Eastern Europe
100
The EVD (Economische Voorlichtingsdienst, predecessor of RVO) was merged with the
st
International programmes of Senter Internationaal on April 1 2004, in order to mainstream the agencies of the Ministry of Economic Affairs.

62 |

2016

2000 it took 28 cents to manage each 1 euro of subsidy and in 2015 it took 9
cents.

€ 60,0

,1,0

,0,9
€ 50,0

,0,8
,0,7

€ 40,0

,0,6
€ 30,0

,0,5

,0,4
€ 20,0

,0,3
,0,2

€ 10,0

,0,1

Programme expenditure

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

,0,0
2000

€ ,0

Operating costs/programme expenditure
(ratio)

Programme expenditure (€, millions)

Figure 4-1 Programme expenditure (€, millions) and ratio operating
costs/programme expenditure 2000-2015

Operating costs/ programma expenditure

Source: Annual reports RVO

4.2.1

How are operating costs divided?

RVO divides total operating costs in three main components: development, management and overhead costs:
1. Development costs encompass all labour costs associated with the selection of new projects for subsidies: the launch of the tenders, and the scoring, ranking and selection of the applications. This also includes the costs
of appeal procedures;
2. Management costs are all labour costs associated with management,
monitoring of projects in de PSOM/PSI portfolio and payment of the subsidy;
3. Overhead costs are labour costs as well as out-of-pocket costs.
a. These labour costs cover all non-project-related work: general expenditures for, for instance, the development of annual reports
and communication efforts101.
101

Other overhead costs concern non-project related coordination with the Ministry of
Foreign Affairs, foreign embassies, local governments, NGO’s and other RVO divisions;

Evaluation PSOM/PSI | 63

b. Out-of-pocket costs are defined by RVO as all non-labour operating costs such as costs of plane tickets for project visits, costs of
communication tools and costs of external advice. The latter includes fees for APSI appraisal of applications but also for market
conformity checks of local hardware prices and other external advice102.
Figure 4-2 below shows the trend in the shares of each component for the period
2004-2015103. While total operating costs increased, the share of each component
remained stable around 30-35%. The main exceptions are the relatively low development costs and high overhead costs in 2008 and the relatively high development costs in 2013. The former can be explained by the transition from the
PSOM to the PSI programme in 2008: thus few new projects and a lot of organisational costs. The latter can be explained by the large number of applications for
the last PSI tender in 2013. In 2015 there were no new tenders and consequently
no development costs.

Operating costs (€, millions)

Figure 4-2. Operating costs: development, management and overhead 2004-2015
€ ,08
€ ,07
€ ,06

(29%)

€ ,05

€ ,04

(30%)

(41%)

€ ,03
€ ,02

(36%)

€ ,01

(41%)

(23%)

Development

Management

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

€ ,00
Overhead

Source: Annual reports RVO
development of MoU’s with foreign governments; presentation of PSOM/PSI in the
Netherlands (chamber of commerce, export seminars, etc); publicity in professional
magazines and foreign media; development of policy documents; training of policy officers; internal policy management; general legal costs and general administrative
support.
102
Some of the out of pocket costs are actually related to development (e.g. APSI fees,
plane tickets and external advice) or management (e.g. plane tickets and external advice). Nonetheless, the information in the annual reports is not detailed enough to allocate these cost to development or management.
103
These figures were not included in the annual reports of 1999-2003.

64 |

2016

4.3

Portfolio, staff and target countries
Of course, total operating costs do not paint the entire picture. They are dependent on a number of factors such as the development of the project portfolio, the
size and composition of the staff and the number of target countries in which the
programme is operated.
4.3.1 Portfolio
Table 4-1 provides an overview of the development of the PSOM/PSI project portfolio between 2004 and 2015.104 It shows the number of project applications, the
number of projects that were selected, and the average workload (number of projects under management) in a year105.
Table 4-1 PSOM/PSI portfolio (# projects), staff and target countries 2004-2015
Year

Applications

Average
workload
90

Staff (fte)

243

Selected projects
68

18.8

Target countries
21

2004
2005

127

99

144

23.0

40

2006

158

83

203

25.2

41

2007

208

86

239

33.5

53

2008

95

50

248

36.7

51

2009

213

101

257

39.6

54

2010

207

103

304

42.1

57

2011

300

106

349

47.2

57

2012

240

120

394

46.1

58

2013

343

125

408

47.3

60

2014

263

58

436

37.9

60

2015

-

-

426

28.5

-

Source: Annual reports RVO

The number of PSOM/PSI applications RVO received fluctuated considerably over
the years. RVO did not have any explicit annual targets. The number of applications depended amongst others on the number of tenders per year, communication efforts by RVO, promotion by local embassies, acquisition effort of consultants and (local) economic factors such as the negative effects of the financial
104

Some of these figures were not included in the annual reports of 1999-2003. Therefore,
the analysis starts from 2004.
105
The total number of project under management vary throughout the year due to the
selection of new projects and the completion of running projects. In line with the RVO
st
definition annual workload is calculated by: (the number of projects at Jan. 1 + the
st
number of projects at Dec 31 ) / 2.

Evaluation PSOM/PSI | 65

crisis106. Up to 2004 separate tenders were set out per country or group of countries. After that, there were generally two tenders per year, with the exception of
2008 when 1 tender was cancelled (due to the transition from PSOM to PSI). This
is reflected in the relatively low number of applications in 2008. Similarly, in 2014
one tender was held as well, but for this tender RVO received a record number of
263 applications.
Not all applications were selected for participation in the PSOM/PSI programme
(granted a subsidy). The number of selected project per year ranged between 50
(2008 and 2013) and 120 (2012). On average, around 39% of applications was selected annually. However, in 2014 this percentage was much lower (22 %), due to
the high number of applications after the announcement of the decommissioning
of the programme.
As a result of the steady flow of project applications and selections the average
workload showed a continuous increase from 90 projects in 2000 to 436 projects
in 2014. After 2014 average workload decreased as no new project were selected.
4.3.2 Staff
Table 4-1 also shows the annual staff-size in full-time equivalents (fte). In their
annual proposal for funding from the Ministry of Foreign Affairs RVO provides an
estimate of the total number of employees needed to manage the programme for
the coming year. This estimate is based on the estimated workload and norms for
the number of days needed to manage a project. Separate estimates were made
for different levels of employees (team leaders, project advisors, support staff,
etc.). The total annual number of fte employed by RVO to operate the PSOM/PSI
programme increased from 19 in 2004 up to 47 at its peak in 2013.
4.3.3 Target countries
Table 4-1 also shows the number of PSOM/PSI target countries. These are the
countries that are eligible for new PSOM/PSI projects. Projects were not selected
in all eligible countries each year. During the PSOM-period, the number of target
countries more than doubled from 21 in 2004 to 51 in 2008. During the PSI period
the number increased at a slower pace with about 10 more to a total number of
60 by 2014. As of 2003 the programme was open for non-Dutch applicants for socalled untied countries (see section 2.2). The number of project from untied countries increased as well. Between 2003 and 2008 around 6 % of all selected projects
were in untied countries. With the introduction of PSI this increased considerably
from 25 % in 2009 to 56 % in 2013, as the number of untied countries increased.
106

66 |

Source: interviews with RVO policy officers

2016

Initially, for the assessment of applications from untied countries, and the management of these projects, a higher number of days was budgeted. This was
dropped in 2012107 after it became clear that untied projects did, in practice, not
require much more assessment/management time.

Productivity
4.4.1

Programme management

Production per staff-member
Figure 4-3 shows the average programme expenditure per fte: the amount of subsidy managed by one RVO staff-member. During the PSOM period there was an
initial increase in the average programme expenditure managed by one fte, but
between 2006 and 2009 the graph shows a decrease of over € 0.75 million.
In this period, RVO staff increased from 25 to almost 40 fte, while total programme expenditure stayed around 24,000,000. RVO programme officers explained that in this period extra employees were needed for the expansion of the
programme and the preparation of the transition to PSI108. During the PSI period
the average programme expenditure per fte increased from € 0.6 million in 2009
to € 1.7 million in 2015 as productivity increased.
Figure 4-3 Average programme expenditure managed per staff-member (fte)109

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

€ ,02
€ ,02
€ ,01
€ ,01
€ ,01
€ ,01
€ ,01
€ ,00
€ ,00
€ ,00
2004

Programme expenditure (€,
Millions)

4.4

programma expenditure/fte
Source: Annual reports RVO
107

Source: Interviews with RVO programme officers
Source: interviews RVO programme officers
109
These figures do not include external hires. There costs are included in out-of-pocket
expenditures.
108

Evaluation PSOM/PSI | 67

4.4.2

Project management

Total operating costs per project
Figure 4-4 shows the trends in total realized operating costs and average annual
workload. Both increased steadily over the development of the programme. After
a short stagnation in growth with the introduction of PSI, between 2007 and 2009,
annual workload increased at a faster rate than total operating costs. As a result
of the termination of the programme in 2014, workload and operating costs decreased. In 2004 the average workload was 90, and operating costs were € 2.0
million. In 2014 the average workload was 436, and operating costs were € 6.0
million.

€ 8,00

500

€ 7,00

450
400

€ 6,00

350

€ 5,00

300

€ 4,00

250

€ 3,00

200

Workload (# projecten)

Operating costs (€, millions)

Figure 4-4 Realised Operating costs and workload (number of projects/year)110
2000-2015

150

€ 2,00

100

Operating costs

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

0
2002

€ ,00
2001

50
2000

€ 1,00

Workload

Source: Annual reports RVO

4.5

Cost drivers
The previous paragraphs show that productivity increased during the PSI period,
as the programme expanded in terms of programme expenditure, portfolio,
110

68 |

We were only able to reconstruct the workload per year as of 2004.

2016

eligible countries and operating costs. In this paragraph we extend the costdrivers analysis that was done for the PSOM/PSI period 2006-2009111 to the PSI
period 2009-2012. For the cost driver analysis 2009 was selected as this was the
first year the PSI programme was fully operated and 2012 as the last ‘typical’ PSI
year, e.g. the year before the sharp increase in number of applications enacted by
the decommissioning of the programme in 2013 and 2014. Table 4-2 provides an
overview of the changes in operating costs and portfolio between these years.
Table 4-2 Developments in operating costs and portfolio 2009-2012
2009

2012

% change

5,021,723

6,471,542

29%

Workload

256,5

393,5

53%

Applications

213

240

13%

Selected projects

101

120

19%

Target countries

54

58

7%

Development costs

1,796,073

2,265,408

26%

Management costs

1,479,857

2,059,664

39%

1,757,113

2,146,470

22%

General

627,952

915,089

46%

OOP

709,399

847,643

19%

Communication

266,395

254,574

-4%

Legal

153,367

129,164

-16%

Total operating cost (€)
Portfolio (# projects/countries)

Variable costs (€)

Overhead (€)
Total

Source: Annual reports RVO
Between 2009 and 2012, total operating costs per year increased by 29%, while
the average workload per year increased by 53%. The largest share of the
operating costs are management costs (39%) followed by development costs
(26%) and overheads (22%).
Table 4-3 shows a breakdown of the main cost drivers. First, it shows the increase
in total realised operating costs between 2009 and 2012 (1). This is € 1.449,819.
Second, it shows the management and selection costs (2) and the overhead costs
(3) that would occur in 2012 based on the efficiency level (operating costs per
project) of 2009 and the change in the number of project applications and selections between 2009 and 2012. Based on the efficiency levels of 2009, manage111

Ibid.

Evaluation PSOM/PSI | 69

ment and selection costs would be € 1,018,082 in 2012 and overhead costs would
be € 938,497. This is higher than the actual increase in operating costs. The calculated management and selection costs (70 %) and the overhead costs (65 %) together are 35 % higher than the total increase in operating costs. Hence, total operating costs are 35 % lower than would be expected based on the 2009 efficiency
level. This indicates higher productivity in terms of lower selection and management costs per project or lower overhead costs. In paragraph 4.4.2 we saw that
management costs and overhead costs per project indeed decreased in this period while development costs per selected project stayed the same despite of the
increase in the number of applications.
Table 4-3 Cost drivers of annual realised operating costs, between 2009 and
2012112
Cost drivers (€)
Total actual increase in operating costs (1)

2009-2012
€ 1.449,819 (100%)

Increase in costs keeping efficiency constant
Management. & selection costs (2)

€ 1.018,082 (70%)

Overhead costs (3)

€ 938,497 (65%)

Other increase in costs (i.e. efficiency) (1-2-3)

-€ 506,760 (-35%)

Source: Annual reports RVO with own calculations
If we compare this to the cost driver analysis in the Triodos evaluation we find
some interesting differences. Between 2006 and 2009 the increase in operating
costs was 17.6 %-points higher than expected based on 2006 efficiency level. This
translates to an annual additional cost increase of around 6 %. According Triodos
the change could be partially explained by inflation and increases in wages and
partially by changes in overhead. Conversely, based on our analysis, between
2009 and 2012 the increase in operating costs was 35 %-points lower than expected based on 2009 efficiency levels. This translates to an annual cost decrease
of 10%. Which may be explained by an increase in workload (scale economies), increased standardization of and familiarity with the PSI selection procedure, stabilization of the number of countries (less marketing efforts), a decrease in communication efforts (with the planned decommissioning of the programme in 2014)
and possibly efficiency gains. A large part of the increase in workload consists of
projects in untied countries (Paragraph 4.3.3). On the basis of our analyses we
find no indication that projects in untied countries lead to higher operating costs.

112

70 |

As the composition of out of pocket costs differ per year, the analysis is done while
leaving out out-of-pocket costs except for communication and legal costs.

2016

4.6

Conclusions
Programme expenditure (€ 2.1 million in 2000 - € 56.8 million in 2014) and operating costs (€ 0.6 million in 2000 - € 6.0 million in 2014113) increased during
PSOM/PSI, as did the portfolio (workload: 90 projects in 2000 – 435,5 projects in
2014), staff (fte: 19 in 2004 – 38 in 2014114) and number of (untied) target countries (21 in 2004 – 60 in 2014). Overall, we see an increase in productivity
throughout the programme in terms of operating costs and fte per programme
expenditure and operating costs and fte per project. We distinguish three different trends over the course of the programme:
 First, between 2000 and 2005 we observe an increase in productivity, with
the exception of 2004, the year in which the PSOM and PSO programme
were integrated and EVD merged with Senter International.
 Second, between 2006 and 2009 there was a decrease in productivity. While
total operating costs increased from € 3.0 million to € 5.0 million, total programme expenditure stayed stable around € 24,000,000. The main costdrivers in this period were wages, interest rates and overhead costs (legal
costs and implementation of PSI).
 Lastly, in the PSI period between 2009 and 2014 we observe a steady increase in productivity. Operating costs and fte per programme expenditure
and management and overhead costs per project all decreased during PSI.
The only exception is the development costs per project. This Increased due
to the high number of applications in the last few tenders. Hence, the number of applications is an important cost driver in the PSI period. Other cost
drivers that partially explain the increase in efficiency are: the increase in
workload (scale economies), the increased standardization of and familiarity
with the PSI selection procedure, stabilization of the number of countries
(less marketing efforts), the decrease in communication efforts (with the
planned termination of the programme in 2014) and possibly other efficiency gains.

113
114

Almost € 7.0 million in 2013.
47 in 2013.

Evaluation PSOM/PSI | 71

5
5.1

Effectiveness
Introduction
Following the evaluation matrix described in the inception report,115 effectiveness
is assessed in the following ways:
 Goal achievement: the extent to which the project targets have been met in
terms of jobs and training, sales and investments, and engaging outgrowers.116
 Attribution: the extent to which PSOM/PSI contributed to the observed
changes (among other factors contributing to success or failure of projects).117
As such, PSOM/PSI effectiveness is a combination of goal achievement and operational success of PSOM/PSI projects, requiring moreover a significant contribution
of PSOM/PSI to these achievements. Additionality is also an important feature to
consider the success of the programme (as discussed in chapter 3). In the longer
run, impact and sustainability are further requirements for the programme being
successful. Goal achievement can be answered using RVO monitoring data, the
other aspects of PSOM/PSI effectiveness require evaluation to assess the contribution of the programme to goal achievement and commercial success.

115

APE Public Economics (2016). Evaluation PSOM/PSI (1998-2014): inception report.
The extent to which PSOM/PSI projects are innovative, also part of this evaluation
question, has been described extensively in chapter 3.4.2. as part of the evaluation of
relevance.
117
Note that the inception report refers to the attribution of the observed changes to the
intervention (‘What are the changes in the effect variables in comparison to the situation at the start (baseline)?’). As will be discussed here below, given the evaluation
methodology (and lack of counterfactual), it is more appropriate to refer to the contribution of PSOM/PSI rather than attribution.
116

72 |

2016

Figure 5. Evaluation of PSOM/PSI effectiveness

Goal
achievement

Operational
success

PSOM/PSI
Contribution

This chapter starts off with the evaluation of goal achievement based on a descriptive analysis of RVO monitoring data (as described in chapter 1) and validation of this data based on the country case studies (5.2.1). The different definitions of success are explored in 5.2.2, while 5.2.3 provides illustrations from the
case studies. RVO monitoring data is used for statistical analysis, assessing whether indicators used during the application process are possible determinants of later goal achievement (5.2.4). Subsequently, this chapter reports on the evaluative
findings on PSOM/PSI contribution to the achievements of projects, including the
ways through which this contribution is made (5.4). Section 5.5. combines both
goal achievement and attribution to determine the effectiveness of the PSOM/PSI
programme.

5.2

Goal achievement
5.2.1 Targets
Based on the above described RVO monitoring data, Table 5-1 provides information on the key indicators and targets for all currently finalized PSOM/PSI projects (428 out of 721 projects). The table includes separate averages for PSI and
PSOM, totals for PSOM and PSI together, for the case study countries, and the
sample of case study projects. The totals in the table are lower because monitoring data are incomplete: for instance, regarding employment we have data on
proposed versus realised jobs for 330 (out of 428) projects. This illustrates the extent to which digitalised RVO monitoring data are incomplete.

Evaluation PSOM/PSI | 73

Taking PSOM and PSI together proposed and realised figures are in line as regard
employment and sales. On average, more people are trained than proposed. The
percentage of projects that achieved its goals is 66% for employment and training,
and 54% for sales. In relation to the planned numbers and amounts the PSI projects perform better than PSOM projects.
In the following the achievement of key targets is discussed on the basis of the
RVO monitoring data for finalized and stopped projects (with additional remarks
based on the case study findings):118
Employment

74 |



About two third of the projects for which data was reported (221/330) did
achieve the targets with regard to the amount of jobs created. 109/330 projects did not achieve this target, on a further 98 projects the BAS system does
not contain data on employment.

118

Unfortunately, the case studies include too many projects with incomplete reports so
that their targets and realizations are omitted from the tables here below. Among
others, due to the often long recall period and lack of available information, verification during company visits was not possible.

2016

Table 5-1 Key results indicators PSOM/PSI (based on 428 finalized projects)
Employment (# jobs)

Knowledge transfer (# persons)

Sales

N completed

Proposed
(average)

Realized
(average)

N completed

Proposed
(average)

Realized
(average)

N completed

Proposed
(average)

Realized
(average)

PSOM

239

85

77

225

246

291

174

€ 983.202

€ 834.511

PSI

91

49

71

89

300

411

84

€ 996.127

€ 1.318.990

Total

330

75

76

314

261

325

258

€ 987.411

€ 992.248

€ 168.300
€ 577.333
€ 857.691
€ 593.667
€ 1.018.830
€ 769.532
€ 834.600
€ 859.500
€ 634.444
€ 275.000
€ 831.430
€ 1.161.200

€ 916.300
€ 1.178.833
€ 931.884
€ 535.905
€ 945.909
€1.066.289
€ 560.560
€ 642.489
€ 657.693
€1.955.604
€ 1.621.153
€ 2.645.131

Achieved goal
(%)
Bangladesh
Sample
Bosnia
Sample
Egypt
Sample
Sierra Leone
Sample
Uganda
Sample
Peru
119
Sample

221 (67%)
8
6
8
4
11
5
5
4
11
3
11
6

208 (66%)
72
118
16
12
33
123
57
58
67
128
74
92

158
175
18
15
38
80
47
52
65
68
96
85

8
6
8
4
11
5
5
4
11
3
10
5

139 (54%)
71
75
16
12
53
100
66
59
507
128
103
128

73
77
18
15
65
66
53
54
518
68
232
186

8
6
7
3
11
5
5
4
9
2
10
5

Source: Monitoring data, calculations APE

119

The evaluation team visited 11 projects in Peru. Because three projects had the same local partner and actually consisted of one prolonged project, these
are reported as one case study. The total number of Peruvian case study projects reported in this report is therefore 9 projects.

Evaluation PSOM/PSI | 75



On average, PSOM/PSI projects employed 76 new people, ranging from 23 in
Eastern Europe to 80 in Africa and Latin America (see Table 5-2 ). As can be
expected, the average increase in employment is highest in the agricultural
sector (80). Compared to services and industry, agriculture is relatively labour
intensive. The regions with the highest average employment increase are Latin America and Africa (80).

Based on the country case studies, the following caveats are to be noted:
 Employment reported to RVO does not always consist of ‘extra’ employment
(solely for the PSOM/PSI project), as there are cases in which the project is
integrated in a larger enterprise and people were shifted between operations.
 Employment figures do not always distinguish between contract and seasonal employment.
Table 5-2 Employment targets: average number of jobs (based on 428 finalized
projects)
N

jobs proposed

PSI

91

49

Jobs created
at completion
71

Committed
subsidy per job

PSOM

239

85

77

Total

330

75

76

Agriculture

206

58

80

€ 17.000

Industry

100

45

56

€ 21.000

Services

36

27

30

€ 20.000

Africa

136

54

80

€ 16.000

Asia
Central and Eastern
Europe
Latin America

116

53

59

€ 17.000

22

23

27

€ 33.000

47

55

80

€ 22.000

MENA

21

41

46

€ 20.000

Sector

Region

Source: Monitoring data, calculations APE

The spin-off study by RVO confirms these findings. According to the spin-off report the PSOM/PSI programme is associated120 with employment growth. Espe120

The survey solely provides descriptives on certain output, outcome and impact criteria.
It does not provide proof on causality. The findings therefore cannot be fully attributed to the PSOM/PSI programme.

Evaluation PSOM/PSI | 77

cially in the agriculture the number of jobs increased significantly after the grant
period. A large part of this increase consists of seasonal jobs. In general, most created jobs are at a basic skill level. The service sector provides most high level jobs.
All wages are stated to be on or above the legal minimum wage121. The employment of women varies considerably among the projects. In industry and services
the vast majority of employees are men while in agriculture it is the other way
around.
Figure 5-6 shows the distribution of employment gains per project, sorted from
the project with the lowest gain to that with the highest gain. The figure gives insight into the distribution of the employment gains. Even though average job increase per project is 76, some 73% of projects had a lower increase in employment. 14% of projects generated 50% of gained employment.

1000

120%

Employment realized

900
100%

800
700

80%

600
500

60%

400
40%

300
average
employment=76

200
100

20%

0

0%
0

25

50

75

100 125 150 175 200 225 250 275 300 325

Project rank on employment (lowest to highest)
Employment realized

Cumulative % of total employment realized

Source: Monitoring data, calculations APE

121

78

This information was obtained by surveys with company directors/managers and not
cross-checked with employees.

|

2016

Cumulative percetage of employment realizef

Figure 5-6: Distribution of employment gains

Knowledge transfer




Two thirds of the projects (208/314) achieved its targets with regard to the
number of people trained. 106/314 projects did not reach their targets, while
114 projects did not report on training.
On average, PSOM/PSI projects trained 325 people, ranging from an average
of 43 in Eastern Europe to 500 in Africa (see Table 5-3). The sector in which
most employees were trained was agriculture.

The country case studies provide the following insights with regard to this target:
 Most training consists of instructions provided by suppliers of hardware procured with the help of PSOM/PSI funds. Such instruction is a natural part of
learning to work with new hardware.
 In a smaller number of projects, additional training offered comprised field
trips of applicant’s staff and/or local partner, use of international experts,
training in areas less related to production (e.g. AIDS training, computer and
language skills). Within the scope of this evaluation, it was not possible to assess whether this extra training was just offered because of availability of
funding, and whether it had an additional impact on the companies’ effectiveness.
Table 5-3: Number of employees trained (target and realized), total 428 finalized
PSOM/PSI projects
N

# trained pro- # trained at completion
posed

PSI

90

230

411

PSOM

225

246

291

Total

314

261

325

Agriculture

201

339

471

Industry

97

77

132

Services

36

46

65

Africa

136

307

500

Asia

112

217

248

Central and Eastern Europe

21

43

46

Latin America

44

196

247

MENA

21

86

106

Sector

Region

Source: monitoring data, calculations APE

Evaluation PSOM/PSI | 79

Female employment
For a more limited number of projects (89/428), information was collected on female employment: RVO reports targets and realizations on female employment
for PSOM and PSI projects. However, proposed targets are only known for PSI projects.
Almost three quarters (65/89) of the finalized PSOM/PSI projects did achieve its
targets on female employment. 24/89 did not achieve their targets. Section 6.1.3
discusses the findings of the case studies in this area, which indicate that despite
these targets, the programme has limited effect on female employment.

Sales


139/258 of PSOM/PSI projects did reach their targets with regard to sales according to RVO data. The complement (119/258 projects) did not realised
their sales targets, while 170 did not report on sales.
On average, sales vary between € 264.298 per annum in the service sector
and to € 1.190.872 in the industry sector (see Table 5-4). The region with the
highest sales level was Asia (€ 1.084.573).
However, document review and company observations during the case studies did point out that a lot of applications projected rates of return and sales
targets that were too optimistic given the limited time frame of PSOM/PSI
support. Moreover, the definition of sales changed over the years: under
PSOM it was cumulative sales over the duration of the project; under PSI it is
the most recent report on annual sales.





Table 5-4 shows the sales per job created. This ratio varies considerably between
sectors and between regions, being highest in Central and Eastern European projects (€ 41.445 per employee), and lowest in African projects (€ 16.970 per employee). It differs little between sectors agriculture and industry, but is much lower in services.
Table 5-4: Average sales –target and realized, total 428 finalized projects
N Sales proposed

Sales realized

Sales per job

PSI

84

€ 1.318.990

€ 29.172

PSOM

174

€ 983.202

€ 834.511

€ 20.499

Total

258

€ 987.411

€ 992.248

€ 23.367

156

€ 956.496

€ 1.017.219

€ 24.416

€ 996.127

Sector
Agriculture

80

|

2016

Industry

73

€ 1.186.429

€ 1.190.872

€ 26.423

29

122

€ 652.734

€ 357.943

€ 10.317

Africa

115

€ 865.148

€ 870.754

€ 16.970

Asia

69

€ 1.212.646

€ 1.084.573

€ 23.593

Central and Eastern Europe

18

€ 1.145.405

€ 981.278

€ 41.445

Latin America

36

€ 814.595

€ 1.220.226

€ 37.079

MENA

21

€ 1.082.230

€ 971.837

€ 18.467

Services
Region

Source: monitoring data, calculations APE

Figure 5-7 shows the distribution of realized sales over the participating projects,
and appears to be in line with the previous graph on employment. Projects are
ranked from lowest sales to highest sales (bars, left axis). The line shows the cumulative percentage of total sales. Average sales were € 992.248, but 74% of projects realized lower sales. The top 12% of projects generated 50% of total sales.

€ ,14

120%

€ ,12

100%

€ ,10

80%

€ ,8
60%

€ ,6
40%

€ ,4
Average
sales=992,000

€ ,2

20%

€ ,0

0%
0

25

50

75

100

125

150

175

200

225

250

Project rank by sales (lowest to highest)
Sales

Cumulative % of total sales realized

Source: monitoring data, calculations APE

122

One finalized project is in the sector transport. We included this project into the sector
services.

Evaluation PSOM/PSI | 81

Cumulative sales per project

Sales realized per project
Millions

Figure 5-7: distribution of sales realized

Outgrowers
The number of outgrowers123 realized is monitored under PSOM and PSI. Outgrowers proposed are only monitored under PSI, and not consistently monitored
(or registered) in the BAS system.


93% of projects for which outgrower data is monitored (31/33) achieved the
targets with regard to the number of outgrowers covered in the project.
On average, the number of outgrowers realized varies between 84 per project in Eastern Europe to 1.227 in Africa (see Table 5-5).



Although this target is only relevant for agricultural projects RVO can classify projects with an agricultural character in another sector. For instance, a food processing project is classified as industrial while it can have outgrowers.
Table 5-5: Outgrowers – average targets and realized, total 428 finalized projects
N

Outgrowers pro-

Outgrowers rea-

posed

lized

PSI

28

593

756

PSOM

9

385

747

Total

37

543

754

Agriculture

33

567

803

Industry

4

338

350

Services

-

-

-

Africa

18

827

1.227

Asia

8

228

325

Central and Eastern Europe

2

108

84

Latin America

5

377

396

MENA
4
Source: monitoring data, calculations APE

215

266

Sector

Region

Subcontractors
RVO monitors the number of subcontractors124 since the introduction of PSI.
Therefore, there are only a few projects, 31 out of 100 finalized PSI projects, with
information on both the target proposed and the target realized.
123

Outgrowers are defined as farmers who deliver their products to a project. These farmers are not contracted employees of the project.
124
Subcontractors are defined as companies or organizations that deliver goods and services towards a PSI project. Subcontractors can occur in all three sectors.

82

|

2016

On average, 8 subcontractors were proposed to engage in PSI projects. Eventually,
14 subcontractors were engaged in PSI projects. Herewith, 28/31 of the PSI projects did achieve their target regarding subcontractors.
(Follow-up) investments


In all projects, the partners themselves did invest a lot, both financially and inkind. The average share of own contributions is € 577,646). In general, the
applicants’ investments are in cash and those of local partners in kind (land,
management time, buildings).125
Follow up investments during the project period vary. According to RVO data
(table 5-6), follow up investments fall short of targets, in all sectors and regions, except for Latin America.
Among the respondents of the spin-off survey 75% of the companies did
follow-up investments, with an average of € 1,000,000 per company. Most of
these investments were financed by the companies’ own resources.





Table 5-6: Follow-investments – average targets and realized, total 428 finalized
projects
N

Follow-up investments pro-

Follow-up investments

posed

Realized

PSI

35

€ 1.233.629

€ 979.631

PSOM

148

€ 1.911.488

€ 770.682

Total

183

€ 1.781.842

€ 810.645

Agriculture

123

€ 1.909.175

€ 817.322

Industry

44

€ 1.703.329

€ 769.824

Services

16

€ 1.018.885

€ 871.571

Africa

84

€ 1.543.543

€ 640.742

Asia

54

€ 2.361.355

€ 690.862

Central and Eastern Eu-

7

€ 742.143

€ 548.049

29

€ 1.475.271

€ 1.460.941

MENA
9
€ 2.333.170
Source: monitoring data, calculations APE

€ 1.223.956

Sector

Region

rope
Latin America

125

Case study interviews. Too limited information on follow-up investments was provided
to the evaluators (for various reasons including this being strategic business information no longer under obligation to share with RVO).

Evaluation PSOM/PSI | 83

5.2.2 Stopped and finalized projects
Subsidies for projects that are unsuccessful can be halted prematurely. The applicant, RVO or these two together can decide to stop the subsidy. According to RVO
(based on an analysis of 262 unsuccessful projects), 60% of the projects are terminated by the applicant, 35% by RVO and 5% are a joint decision or the reason is
unclear.
The majority of unsuccessful projects face problems during the initial phase of the
project. Reasons to terminate a project are: problems concerning partners, market conditions, unachieved results, financial problems, lack of confidence in successful completion of the project, failure to meet the obligations or changes in the
political situation. Often, more than one of those reasons may lead to project failure.126
Figure 5-8 shows the number of stopped, ongoing and finalized projects per tender year. Across 1999-2006, approximately 30% of the granted projects per tender are deemed unsuccessful and are stopped. The percentage of unsuccessful
projects is higher for tenders 2007 and 2008. The percentage is around 40% for
these two tenders. The percentage of unsuccessful projects declines from 2009
onwards. However, a lot of projects from these tenders are still ongoing and so
success is still uncertain. In total, 293 projects are stopped until October 2015,
which is the moment of data collection at RVO.
Figure 5-8: The number of ongoing, finalized and stopped projects per tender year

Number of projects

140
120
100
80
60
40
20
0

Year
Ongoing projects
N = 1107; all granted projects
126

84

Finalized projects

Stopped projects

Source: RVO monitoring data, analysis: APE

Wortelboer, J. (2015) PSI & PSOM overview 1998-2015, Den Haag: RVO

|

2016

5.2.3 Disclaimer on goal achievement
PSOM/PSI intends to support innovative projects in often difficult country contexts (with the exception of activities in more stable middle income countries like
Peru). Therefore, it is fair to note that projects that did not fully achieve their targets, are not necessarily a failure or lack of success.
In all case studies, there are examples of projects that did not implement what
was proposed, or did not fully achieve each and every target, but nevertheless
generated results that are in line with PSOM/PSI objectives, significantly contributing to development. Box 10 provides examples of such cases.
Box 10: Success despite targets
Project successful with a very different business than envisaged
In Egypt there is the extreme example of PSOM/PSI support (2000) to a poultry farm that was
successfully turned into a fish feed producer. The poultry farm was initially successful, exceeding its targets, however, the 2006 avian flu hit the company hard. The Dutch embassy
actually provided the company with the idea to go into aquaculture, which the joint venture
took up through a pilot scale production facility. The company produced both poultry feed
and fish feed (2/3 of its turnover). The joint venture has now been taken over by a large multinational company.
Project did not succeed, but components are used elsewhere
In Uganda one of the PSOM/PSI projects that was successfully finalised (vegetable production) had to close its business operations after a few years because of difficulties with production circumstances (temperature for tomatoes, plant diseases, payment collection). However, the greenhouses and most employees are now working for a different business (flower
cuttings), also supported by PSOM/PSI.

There are of course also real ‘failures’ of PSOM/PSI, i.e. projects that should have
been avoided (e.g. through better selection processes or monitoring). In this category fall projects that are fraudulent, where there seems to be an intentional inappropriate use of subsidy for the benefit of the applicants. However, of the 49
case studies, only one such suspected case was identified.127 Another example of
a failed project is where the PSOM/PSI subsidy was not provided to a company,
but rather to a development project, for which this subsidy was certainly not intended.

127

Currently under investigation

Evaluation PSOM/PSI | 85

There are, however, also some projects where the incentives associated with
PSOM/PSI might have contributed to the lack of success.128 These are projects
that were set up too ambitiously for their own good in response to the amount of
funding available through the subsidy, rather than growing organically from
smaller initiatives that would have fitted better with the projects partners’ capacities, demand, market opportunities and other business-related factors. In these
projects there is evidence of overinvestment in:
 Amount and sophistication of hardware (e.g. unsuitable for the local
company or country context)
 Employment (to many staff recruited at once)
 Training (e.g. extensive travel)
Several of these projects are among those where there are doubts about the additionality of the PSOM/PSI support as well (chapter 3.6.3). Box 11 provides illustrations of projects that the evaluators did consider to be ‘true’ failures.
Box 11: Failures
Example of an unsuccessful project
In one of the country case studies, the evaluators visited a private clinic supported through
PSOM/PSI. During the visit, there was no single patient in sight nor was there evidence of any
activity ongoing or having been going on recently. The Dutch embassy confirmed this lack of
success. Both the applicant and the local partner benefited from the project even before its
implementation, through managerial fees (applicant) and equipment sales (local partner).
5.2.4 Predictors of success
Monitoring data collected by RVO was combined with the ranking scores in order
to investigate whether these scores predict success. The definition of success is
whether a project was implemented according to plan (i.e. subsidy paid out). To
find potential determinants of success profiles of projects were constructed to see
which characteristics of granted projects differed significantly between successful
and unsuccessful ones. Moreover, logistic regression and discriminant analysis
were used to determine which set of project characteristics best distinguished between success and failure (more detail on the methodology can be found in annex
V).
These analyses rely on the ranking scores of granted PSI project applications because the PSOM ranking scores are too incomplete to be useful. Moreover, even
128

86

This category is hard to determine through interviews but at least in six cases did company visits and interviews provide an indication that slower and lower investments
might have been more appropriate and effective.

|

2016

among the PSI data only the 2010 and 2011 tenders used consistent ranking systems which can thus be used for statistical analysis.
Profiles
As a starter, financial profiles were constructed for all finalized (100) and stopped
(139) PSI projects (annex V provides an overview).129 Ongoing projects were not
considered as it is too early to determine whether they are successful or not.
Of all the ex-ante financial indicators,130 only the applicant’s balance (total of assets) and equity ratio (this ratio measures the proportion of the balance financed
by the applicant) differ significantly between stopped and finalized projects, i.e.,
lower for stopped projects than for finalized projects.131
Table 5-7: Financial indicators with a significantly different mean for finalized and
stopped PSI projects (n = 100 finalized and 139 stopped PSI projects)
Finalized

Stopped

Variable

N

Mean

N

Mean

Balance applicant

82

€ 8.720.452

113

€ 5.504.171

Equity ratio applicant

82

11,69

113

6,77

Logistic regression
Annex V describes the different ways in which an attempt was made to estimate
the chance of success or failure based on the scores of a project proposal given by
RVO before the APSI ranking. A main challenge was to obtain a sample size that
was sufficiently large without losing too much detail with regard to the independent variables. This was an issue even though the analysis focused on the scores
from the 2010 and 2011 ranking forms (N = 113 PSI projects), a period in which
there is a relatively large number of finalized and stopped projects in both tenders

129

These profiles are based on company information from both partners and proposed
project characteristics reported in the application form. We constructed also constructed financial profiles for all 616 selected PSI projects and a sample of 249 rejected PSI projects.
130
Originally committed subsidy, Project characteristics (intentions), Hardware (%), Technical assistance (%), Internal rate of return, Division of shares applicant (%), Division of
shares local partner (%) / Characteristics applicant (before application): Own contribution, Equity, Balance, Turnover, Net profit, Cash flow, Number of employees, Equity
ratio, Financing ratio, Solvency ratio, Liquidity ratio / Characteristics local partner (before application): Own contribution, Equity, Balance, Turnover, Net profit, Cash flow,
Number of employees, Equity ratio
131
At a 5% confidence level. At a 10% confidence level, equity and turnover are also significantly lower.

Evaluation PSOM/PSI | 87

(compared to the 2012 till 2014 tenders, where most projects are still ongoing, or
earlier tenders when scoring was much less elaborate).
Different success indicators were analysed, such as the project status given by
RVO; the number of proposed targets that were achieved;132 the percentage of
originally committed subsidy that was disbursed, and the accomplishment of each
of the proposed targets separately - employment, training, outgrowers, subcontractors, females, sales and follow-up investment.
Inspection of the correlation matrix suggests that the accomplishment of each of
the proposed targets is correlated with scores on specific components (score for
innovativeness in marketing or the score on proposed number of high or medium
level trainings). However, remarkably, a project's total score is not in any way significantly correlated with any of the success indicators analysed. Similarly, the logistic regression did not identify any variables in the ranking that significantly influences achievement of targets.
Discriminant analysis
Discriminant analysis is another way to analyse the predictability of success. This
type of analysis has the additional advantage of being able to construct a continuous indicator of success from the resulting coefficients. This analysis used goal
achievement (i.e. difference between target proposed and target achieved relative to target proposed) with regard to employment, training and sales as predictors of success (projects being finalized or stopped). Other achievement indicators, like female employment, outgrowers, subcontractors and follow-up
investments, have too few observations to allow for a reliable analysis. In total
372 out of 721 finalized and stopped PSOM/PSI projects have information on all
three key indicators. The resulting scale can take any value, positive or negative.
Figure 5-9 plots the total ranking score against the resulting discriminant score, as
an indicator of success. The total ranking score is the weighted sum of the scores
that applicants get on several items (see par. 3.2). While one would expect that
projects with a higher ranking score would also have a higher success score,
Figure 5-9 does not show such a pattern. There are projects that have a total ranking score of 0,6 or higher who did not realise the targets as proposed. The majority of the projects that did meet their targets have a total score between 0,3 and
0,5, which is certainly not the highest score possible.

132

88

Achieved is defined as target proposed ≤ target realized. The targets are employment,
training, sales, females, outgrowers, subcontractors and follow-up investments.

|

2016

Figure 5-9: Scatterplot discriminant score vs. total score ranking (n = 59 projects
from tender 2010 and 2011)
Level of success (discriminant score)

8
7
6
5
4
3
2
1
0
-1 0
-2

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

Total ranking score (standardized)

Source: monitoring data, analysis APE

Hence, whether a project comes to full completion (finalized rather than stopped)
or reaches its targets (or the majority of them) cannot be predicted on the basis of
the ranking score at application. This is not surprising given that the application is
a snapshot, aimed at obtaining a subsidy, and after application the indicators can
easily change. Moreover, there are many external factors, not included in the
scoring, that affect whether or not a project achieves its targets.
On the other hand, proposals are assessed on the basis of their expected viability,
and the ranking scores are presumed to indicate the project’s potential success.
The absence of a (statistical) relationship between the ranking scores and the
eventual success shows that these scores lack predictive power. The analysis
based on correlations in the previous section suggests that some of the item
scores may be better predictors than the weighted sum across these items.
5.2.5 Conclusions on goal achievement
The following table provides an overview of the projects that achieved its separate targets, as monitored by RVO.133 There are several possibilities (e.g. all or part
of the targets reached, or accomplishment on each category of targets separately), though each of them has implications for the sample size as the monitoring by
RVO has been adapted over time and not all targets are available for all projects.

133

Achieved is defined as target proposed ≤ target realized.

Evaluation PSOM/PSI | 89

Table 5-8 shows the number of finalized projects that achieved their target, per
programme, sector and region (see also Table 5-1). As the monitoring data is incomplete, goal achievement is only reported for finalized projects for which information, both on target proposed and target realized, is available. The extent to
which achievement rates are different for projects with unknown data is of course
unclear. Likewise, it is unclear whether the outcomes in Table 5-8 are biased.
Table 5-8. Goal achievement, total finalized project is 428134
Employment

Training

Outgrowers

Sales

PSOM

152 (238)

140 (226)

6 (9)

90 (173)

Follow-up investment
57 (146)

PSI

69 (92)

68 ((89)

25 (28)

49 (84)

13 (38)

Total

221 (330)

208 (315)

31 (37)

139 (257)

70 (184)

Agriculture

128 (200)

121 (192)

27 (33)

83 (157)

50 (122)

Industry

66 (94)

62 (89)

4 (4)

43 (73)

16 (44)

Services

27 (35)

25 (34)

-

13 (29)

4 (16)

Africa

85 (135)

85 (129)

16 (18)

58 (116)

33 (85)

Asia

68 (106)

62 (100)

8 (8)

39 (68)

15 (54)

Central and Eas- 18(22)
tern Europe
Latin America
35 (47)

17 (21)

1 (2)

11 (18)

3 (7)

29 (43)

4 (5)

21 (36)

16 (29)

MENA

15 (20)

2 (4)

10 (20)

3 (9)

Sector

Region

15 (20)

Overall, two thirds or more of the finalized projects achieved their targets on employment, training, or outgrowers. Finalized projects have a lower goal achievement rate on sales and follow-up investments. In short, the projects are more
successful in reaching development goals than in achieving business targets.

134

90

Percentage goal achievement for a particular target is defined as the number of finalized projects that had a higher target realized than their target proposed as a share of
all finalized projects with information, both proposed and realized, on that particular
target. For instance, 128 projects in the agricultural had a higher number of jobs created than they proposed. Both employment proposed and employment realized is
registered for 201 agricultural projects. Therefore, the percentage goal achievement is
64%.

|

2016

5.3

Contribution PSI/PSOM
5.3.1 Introduction
In this Section we investigate to what extent PSOM/PSI grants have contributed to
the goals achievement rates discussed in the previous Section. The evaluation matrix for the evaluation of PSOM/PSI 1999-2014 refers to the question of attribution of the observed changes to the intervention. However, as noted above, given
the multitude of external factors that affect the outcome of PSOM/PSI projects, it
is hard to determine the exact attribution of desired changes to PSOM/PSI. This is
particularly so because there is no baseline for new projects and it is virtually impossible to construct a counterfactual for the PSOM/PSI projects, i.e. find innovative projects/companies that have the same characteristics (firm-level, sector and
market), but did not use the support of PSOM/PSI. Therefore, it is more appropriate to refer to the contribution or influence of PSOM/PSI to the outcomes of the
projects supported.135
In this evaluation, an attempt is made to establish the contribution of PSOM/PSI
in two ways (methodologies are described in chapter 1.3):
1. Qualitative evaluation based on country case studies
2. Quantitative evaluation based on RVO monitoring data
The rationale is that the qualitative information complements the statistical analysis, by providing more in-depth insights into the way in which PSOM/PSI contributed to the goal achievements. Moreover, the qualitative analysis could be used
to validate the findings of the quantitative analysis. As will be discussed here below, the quantitative analysis is, however, limited, so that the evaluation relies
mostly on the qualitative analysis for drawing conclusions on the contribution of
PSOM/PSI.
5.3.2 Quantitative data analysis on PSOM/PSI contribution
As discussed above, most information is available for the targets achieved with
regard to the key indicators employment, training and sales, as these data were
collected throughout the PSOM/PSI evaluation period. Different contribution
models have been used to determine what caused changes in these outcome variables.136 Possible determinants of these key indicators were:137
135

i.e. PSOM/PSI is said to have had an influence on the project, without deriving this
through comparison with other similar projects
136
i.e. these three indicators are the dependent variables in the models described in the
subsequent paragraphs. The logarithm of sales realized are used in order to normalize
the error terms.

Evaluation PSOM/PSI | 91






Committed subsidy for hardware and training (variations),
Sector (industry, agriculture, services),
Regions (Africa, Asia, Latin America, Central and Eastern Europe, Middle
East and North Africa),
Specific tenders138

Only for PSI projects is it possible to distinguish between subsidy committed for
hardware and for training (as proposed in the application form). With only 100 finalised PSI projects with information on goal achievement, it was not possible to
maintain this distinction in the modelling because 100 projects is too small a sample to reliably establish the contributory effect of a PSOM/PSI grant.
In the analysis we include an indicator for probability of being selected 139 in order
to take account of selection bias. Selection bias may be present if, for instance,
projects are selected on the basis of proposed targets. In that case proposed targets cause projects to be awarded a grant, instead of the grant contributing to
target achievement. To correct for such ‘built-in’ effects we include an estimate of
the probability to be selected.
The results are summarized in Table 5-9 (the full result tables can be found in Annex V). The PSOM/PSI subsidy significantly contributes to the achievement of the
employment target. This effect is robust against different ways of modelling. This
outcome implies that if PSOM/PSI committed subsidy is raised by ten percent (say
from € 550.000 to € 605.000), the number of jobs created at the end of the project is increased by almost 4, keeping other things equal. A higher subsidy also
translates into more sales, albeit at a lower level of confidence: 10% more subsidy
means 5.3% more sales. PSOM/PSI is not associated with more training, as shown
by the high P-value of the coefficient estimate. 140

137

An agricultural project in Africa selected in 2009 is the reference project, described by
the intercept.
138
Dummy variables for tenders up to 2012. No dummies for 2013 and 2014 are included
because there are no finalized projects from these tenders yet.
139
Constructed for PSOM and PSI projects by regressing a dummy for selection on dummies for sector, region and tender. Financial indicators were not included as this
would lower the sample size by too much.
140
As a reminder: this analysis assesses the effect of a change in the size of the subsidy
(rather than with or without subsidy).

92

|

2016

Table 5-9. Summary statistical analysis impact PSOM/PSI
Committed subsidy impact on:

2

Coefficient

P-value

N

R

Employment

39,31**

0,02

332

0,131

Training

-37,80

0,82

327

0,127

Sales (log)

0,53*

0,10

264

0,142

**significant at a 5% confidence level; * significant at a 10% confidence level.

From the case studies emerge indications that this result might be because the
partners responsible for the projects have more influence on the employment and
sales targets than on training targets. Actually, there are examples of projects in
which the PSOMPSI targets were said to have been an incentive to maintain staff
members through periods of crisis. For example, in some projects in Sierra Leone
during the Ebola crisis, project implementation was on hold, but staff was retained given the targets (albeit with adapted contracts and reduced wages). However, the case studies also identified projects where maintaining the employment
target was commercially not optimal, in which case employment was reduced
shortly after the ending of the project (e.g. IT company in Bosnia Herzegovina).
This reasoning implies that PSOM/PSI would have a significant influence on training as well, as this target might be even more in the hands of the project management (more flexible and short term than employment is). On the other hand,
employment and sales are closer linked to programme implementation and will
vary with the size of the projects, while training is relatively cheap and a less inherent part of production.
5.3.3 Case study results on PSOM/PSI contribution
The case studies, which include document review, interviews with applicants and
local partners, and company observations, were used to reconstruct the
PSOM/PSI contribution for a representative sample of 49 projects. Box 3 in chapter 1.2.4 describes the way in which contribution was assessed in this evaluation.
In short, a distinction was made between
a) minimal contribution, whereby PSOM/PSI is one of many factors contributing
to the success of the project (merely through the financial contribution); or
b) decisive contribution, whereby PSOM/PSI is considered by the project partners to be one of the main factors, going beyond the mere financial support,
including less tangible contributions such as risk-reduction, catalyst role and
positive effects of the international partnering that are all non-financial
components of the programme.
In sum, as depicted in Table 5-10 , PSOM/PSI has had a decisive contribution to
the projects in 19 out of 49 of the cases. In these cases, the conclusion of the case
studies is that PSOM/PSI had a decisive influence on the project, and was of signif-

Evaluation PSOM/PSI | 93

icant importance for its outcomes (without PSOM/PSI the outcome might have
been markedly different).141 In 28 out of 49 cases PSOM/PSI was thought to have
contributed, at least through its financial contribution, but there were other,
much more important factors that determined the outcomes of the projects.
Table 5-10: Summary of PSOM/PSI contribution
Bangladesh
PSOM/PSI one of
many factors contributing

5

BosniaHerzegovina
1

PSOM/PSI one of
the main factors
contributing

3

6

Too early to tell

1

Peru
(9)
8

Sierra
Leone
6

Egypt

Uganda

Total

2

6

28

1

2

6

1

19

1

2

From the case studies follows that the contribution of PSOM/PSI to projects happened through (Box 12 provides some examples):
Financial support for hardware and training: In principle, all companies would
benefit from a substantial subsidy as provided through the PSOM/PSI programme.
The amount of subsidy ranges between 50% and 60% of the total project budget.142 Though for some companies this grant support is more important than for
others that might have been able to access funding through other sources, it is for
all companies rare to receive such financial support for free.143 As such, PSOM/PSI
is considered to in principle always be one of the (many) factors contributing the
projects’ outcomes.
With PSOM/PSI it is possible to obtain the hardware for the project from the applicant. Because of the difficulty of following the money in PSOM/PSI projects, it is
unclear whether this contribution is always matched with own contributions.
Moreover, as noted above, there are cases in which the investments in hardware
seem to have been excessive compared to true need, and driven by the subsidy
available.
In principle, PSOM/PSI allows partners to spend more on training than would otherwise have been the case. There is evidence that such training has benefited, for
141

This does not imply that this influence is necessarily additional, i.e. could not have been
achieved through other means.
142
RVO monitoring data
143
This point links to the evaluation of additionality, as discussed in chapter 3.6.

94

|

2016

example, farmers engaged as outgrowers. Employees of projects have also benefited from more training than usual (e.g. with regard to health). At a management
level, there are examples of spending funds for travel between the country of the
applicant and the country where the project is located (e.g. technicians that provide training on hardware visiting more often and for longer periods of time).
While those who manage the projects state this kind of training has been very
useful, it is hard to determine with certainly to what extent such extra training is
more effective than what would have otherwise been provided.
Risk reduction: PSOM/PSI was also said to have been important because the subsidy allowed the project to overcome high initial investment costs and thus start
up a risky, innovative project quicker and bolder than would have been otherwise
possible. In particular the applicants noted that the subsidy of PSOM/PSI allowed
them to accept the risk associated with the project or country-context. For local
partners, an additional benefit was the way in which the international partnership
had a positive effect on the reputation of the project and thus the local partner.144
Once the PSOM/PSI project had taken off, interest in follow-up investment could
be generated, including from local sources.
Knowledge transfer: Not all the transfer of knowledge occurs through the training
component of the PSOM/PSI subsidy. The international partnership, which is part
of the design of the PSOM/PSI programme, in itself can stimulate the transfer of
knowledge, technology and innovation through working together and using joint
systems and production methods. Several local partners called it a ‘Dutch way of
working’ that was transferred through the partnership and was thought to have
contributed to the success of the project (e.g. anti-corruption, more participatory
management styles). There are other examples where the partnership has led to
improved management standards and professionalization of local businesses. On
the other hand, local partners provide the applicants (Dutch or otherwise) with
crucial information and knowledge as well (e.g. on context-appropriate management styles, access to local networks, knowledge of business environment). This is
an indirect contribution of PSOM/PSI, which is not directly linked to the subsidy,
but facilitated by the requirement of establishing a longer term international
partnership.
Indeed, the idea that local companies in PSOM/PSI target countries benefit from a
joint venture with international (non-PSOM/PSI) countries through knowledge
transfer is an implicit part of the PSOM/PSI philosophy145. However, this is certain144

Several projects actively promote this international character or the association with
the country of the applicant (e.g. use of flags, in logos)
145
For example see: PSOM tender documents 1999-2008 and PSI Government Gazette
2009-2014

Evaluation PSOM/PSI | 95

ly not applicable to all local entrepreneurs and in all PSOM/PSI countries. In a
middle-income country with relatively high education levels as Peru, the local entrepreneurs certainly do not lack access to up-to-date knowledge and technology.
There are also examples where the applicant plays a very minimal role, which
would by nature also limit knowledge transfer either way. Moreover, if there is a
lack of knowledge, this could potentially be accessed through other sources (e.g.
consultants, courses, acquisition, suppliers of hardware). For example, one of the
joint ventures in Egypt made use of the PUM programme.
Additional support from the Netherlands: Some of the projects have received
crucial support from Dutch embassies, even projects that did not have any connection with the Netherlands apart from being subsidized by PSOM/PSI (nonDutch applicant and local partner). Some embassies supported the projects, e.g.,
in official business affairs (e.g. approval for land acquisition). In one case the embassy actually provided the idea with which the original project plan could be
transformed into a successful one after it had run into problems (see box 9).
Though it is not always directly attributable to PSOM/PSI, several of the projects
have been supported by other Dutch actors as well. For example, Dutch NGO
Wilde Ganzen has supported a project in Bangladesh with community outreach (in
kind and financially). Furthermore, a few projects have benefited from the services of PUM, Netherlands Senior Experts programme, which provides technical
assistance in management. There is also an example whereby a project cooperates with the Dutch university, Wageningen (WUR).

96

|

2016

Box 12: Example of contribution PSOM/PSI
Industrial production, Bosnia Herzegovina
The applicant was working in China, and wanted to relocate to Eastern Europe. The
PSOM/PSI subsidy was instrumental to the decision to move operations to BosniaHerzegovina and not to another, non-PSI, country (e.g. Poland or Check Republic). Bosnia-Herzegovina was thought to be riskier than the other countries, but the PSI grant
compensated for this. This joint venture allowed the local partner to expand its market.
Industry, Sierra Leone
The applicant of this project worked with a local partner who had lived in the Netherlands before. Together they set up a joint venture (with a long-term loan from the applicant to the local partner), supported by PSOM/PSI. According to the local partner,
the joint venture and close working relationship that this implied led to a new way of
working, e.g. with regard to the relationship with workers, stance against corruption
and the ‘step-by-step’ approach (rather than quick wins). This was actively disseminated by the local partner, e.g. through university lectures, within the business community
and through interns. So apart from the training required to operate the business (including retraining of personnel), the joint venture – supported by PSOM/PSI - was
thought to have contributed to more significant knowledge transfer. Without
PSOM/PSI, less collaboration might have resulted in less of such transfer.

5.4

Effectiveness: combining success and contribution
As discussed, goal achievement and commercial success of PSOM/PSI projects
have to be combined with a significant contribution of PSOM/PSI for PSOM/PSI to
be considered a truly effective subsidy programme. In the longer run, impact and
sustainability are additional requirements for the effectiveness of the programme.
RVO collects monitoring information on goal achievement and commercial success (targets and ‘finalised projects still active’). However, the availability of monitoring data differs strongly between the indicators. For employment 67% of finalized project for which data was reported reached the target, for sales 54% of
projects reached the target, and for training 87%. For other indicators data was
reported for less than 100 projects. More generally, given the situation that monitoring data on goal achievement is only available for about half of the finalised
projects the question arises how representative this data is.
Despite their shortcomings these monitoring data were used to analyse the effect
of the subsidy on the employment, sales and knowledge transfer targets. The
PSOM/PSI subsidy appears to significantly contribute to the achievement of the
employment target. A higher subsidy also translates into more sales, albeit at a

Evaluation PSOM/PSI | 97

lower level of confidence. PSOM/PSI subsidies are not found to be associated with
more training.
This evaluation was intended to evaluate whether PSOM/PSI was an effective
programme, which requires the PSOM/PSI support to have in some way contributed to the successful continuation of the projects. Moreover, an additional requirement for an effective subsidy programme is that the financial support is additional, i.e. that the projects would not have happened at the same scale without
this support (the assessment of ex ante additionality is discussed in chapter 3).
Box 13: Defining success
RVO monitoring of projects =
% of goals achieved (monitoring data)
% of subsidy disbursed (financial information)
‘Finalised still active’ in the longer term (spin-off survey)
Effectiveness of PSOM/PSI programme =
Operational business + Significant contribution PSOM/PSI
(case studies and quantitative analysis limited by data availability)
Market and sector impact + sustainability in the longer term (not assessed
per project in this evaluation)

Therefore, to evaluate the effectiveness of PSOM/PSI as a programme, and to answer these extra questions of additionality and contribution, this evaluation relied
on case studies, assessing 49 projects in 6 different countries, with projects starting in 2000 and some running until 2017. As described in Section 5.3.2, statistical
analysis was also used to assess the contributory effect of PSOM/PSI, which could
be determined for employment and sales, though not for training.
What do the case studies tell us in this regard (see Annex VII for an overview of
projects)?
4. Firstly, 33 of 49 projects visited were operational businesses. In five
cases it was too early to tell. However, this 75% (33/44) cannot be
compared with RVO monitoring information on goal achievement because of a bias in the sample as operational businesses can be visited,
while non-performing projects are by nature less accessible.

98

|

2016

5. Secondly, in 18 out of all the 44146 projects (41%) evaluators concluded based on interviews with applicants, project partners, file review
and company visits that the PSOM/PSI programme played a crucial
role in the resulting company (i.e. decisive contribution without which
the outcome of the project might well have been different). In the
remainder of the projects visited, PSOM/PSI was seen to have been
only one of many factors contributing (at least by providing a grant),
but this support was considered to not have been decisive for its
achievements.147
6. Finally, if we then combine this information, out of the 49 projects visited, 17 projects that were operational businesses, received a contribution from the PSOM/PSI programme that was considered to be decisive - 5 projects started too recent to assess (39% or 17/44).148
The effectiveness of PSOM/PSI depends on the ability of RVO to select the right
projects to invest in or to collaborate with, i.e. those that have the potential to be
commercially successful and for which PSOM/PSI can make a difference (ex-ante
additionality as well as decisive contribution). For example, in a country like Peru,
it is relatively easier to identify potentially successful entrepreneurs (given skills
level and market development), but more difficult for PSOM/PSI to make a decisive contribution. In this context a subordinate loan might have been more effective (both with regard to additionality and significance of contribution). In Sierra
Leone, on the other hand, it is much more difficult to facilitate a successful business (given skills and market development and country-specific risks), but
PSOM/PSI will more likely e a decisive factor in the achievements of the projects.
In such contexts, the PSOM/PSI grants seemed to have more potential for effect
(albeit if adapted to the size of the companies as the Uganda case shows). A more
tailor-made programme (with regard to what is offered) might have been more
appropriate given the variety of country contexts.

146

For ease of reading we exclude the 5 cases in which it was too early to tell whether a
company was operational or not, though that does not necessarily exclude an assessment of the contribution (which can be made at very early stages).
147
See box 3 for the assessment of contribution as either decisive (main factor, going beyond the financial contribution) or minimal (one of many factors, merely a financial
contribution) for the achievement of results.
148
If you would subtract from the 49 project those considered not to be ex ante additional
(see chapter 3 on relevance), 14 projects would be left that are operational and to
which PSOM/PSI made a significant contribution (32% or 14/44).

Evaluation PSOM/PSI | 99

Table 5-11: PSOM/PSUI contribution
Business status

PSOM/PSI
contribution

Operational

Total

Not-operational

Decisive

17

1

Not decisive

16

10

33

11

44

Box 14: Significant but not additional contribution of PSOM/PSI
In case of a decisive, but not additional contribution of PSOM/PSI, case studies indicate that
there has been a decisive influence of the PSOM/PSI programme on the project. However,
with regard to financial resources, risk or knowledge (i.e. aspects in which PSOM/PSI is to
add extra value or be additional), the partners could have done without PSOM/PSI. This results in the rare combination (3 cases) of a decisive contribution that was nevertheless not
additional (ex-ante).
Industry, Sierra Leone
In this case, PSOM/PSI is thought to have significantly contributed to capacity development
in Sierra Leone, even though the project itself would probably have happened at a similar
scale and pace without PSOM/PSI. Because without PSOM/PSI there is a reasonable chance
that the partnership between the applicant and the local partner might have been a different, e.g. a less reliant and close partnership, which would have led to less knowledge transfer (beyond the PSOM/PSI training) and cooperation between partners. As such, PSOM/PSI
provided a decisive contribution (i.e. more than other factors and beyond the financial contribution), even though it is considered not ex ante additional.

100

|

2016

6

Impact
As listed in the evaluation matrix, the impact of the programme includes:
 Impact of PSOM/PSI on:
 Corporate social responsibility policies as applied by the project, including capacitating employees and good working conditions (with special attention to gender),
 Sector (value chain) and market (country) development in the country of
implementation.
Moreover, sustainability was also included among the evaluation topics, measured by:



continuation of completed projects and assessment of the economic viability of ongoing projects,
factors ensuring the economic viability of the projects on the long run.

This chapter describes, based on the country case studies, the projects’ possible
impact on CSR policies and practices, especially conditions (6.1). This part of the
evaluation is based on company visits and interviews with local partners and applicants, as well as group interviews with employees. The chapter subsequently
discusses the effects of PSOM/PSI projects on the wider environment, meaning
beyond the project itself. This includes the sector (value chain) and the market
(country), as described in 6.2. Sustainability is described in 6.3.

6.1

CSR, including working conditions
With regard to corporate social responsibility (CSR) in general, the main conclusion from the case studies is that the role of PSOM/PSI (directly through selection
criteria and targets in this area, or indirectly through knowledge transfer in the
partnerships) is limited. This is partly due to the design of the programme, in
which attention to CSR is one of the selection criteria. As a result, PSOM/PSI works
with entrepreneurs that already take CSR into account (e.g. applicants have CSR
policies or are at least positively disposed to CSR). Furthermore, the case studies
reveal that longer-term attention to CSR is strongly driven by the market, i.e. the
demands of the customer of the PSOM/PSI projects, rather than by PSOM/PSI.
This is particularly true for companies exporting agricultural produce. These are

Evaluation PSOM/PSI | 101

nearly always subjected to some sort of certification incorporating CSR (no child
labour, decent work condition, sometimes organic production or fair trade. So
though PSOM/PSI might provide financial means for certification, the fact that the
company complies with the regulations for such certification is regardless of
PSOM/PSI.
Nevertheless, the case studies found some examples where influence of
PSOM/PSI on applying CSR policies was noted. PSOM/PSI had strong influence on
CSR in only 4 out of 49 projects, and ‘some’ influence in 19 out of 49 (see Table 61). Such influence ranges from companies discussing CSR amongst the partners
(including some transfer of knowledge), to an agricultural project in Sierra Leone
that does no longer accept child labour from communities ‘because we are not allowed by PSOM/PSI’ (however this company was already committed to CSR from
the start). Strong influence happens in cases where PSOM/PSI seemed to have
been a driving factor for CSR e.g. in some cases organic produce was a requirement for PSOM/PSI support, in others PSOM/PSI financial support was used for
certification of products or other community-related activities (e.g. training facilities used by the community).
Working conditions garnished most attention, including employment for female
employees, and community involvement. In almost all cases RVO included conditions with regard to wages. For example wages needed to be a specified percentage above the minimum wage. Much less attention was given to CSR in the area
of environmental or ecological impact.
The spin-off survey, conducted by RVO (2015) also addressed CSR. The CSR component of the spin-off analysis mostly focusses on certification and chain responsibility. Companies have certification for quality149 (46%), environmental safety150
(27%) and social responsibility151 (14%). The spin-off survey confirms the findings
of the case studies that the need for certification seems to be driven by clients on
export markets (rather than PSOM/PSI). As for chain responsibility, 20% of the respondents set CSR standards for their suppliers. These mostly concern prohibition
of child labour or forced labour; environmental measures and certification. The
report also lists a few challenges that were identified by the respondents: work
safety; environmental protection; facilitating payments152. The country case stud-

149

For example, HACCP, ISO 9001 and ISO 22000.
For example, organic, Global Gap, ISO 14000 and MPS.
151
For example, Fair Trade, ETI and SA 8000.
152
Facilitating payments are defines as ‘small payments that facilitate certain processes’.
They are not considered as corruption (RVO 2016 PSI spin-off analysis).
150

102

|

2016

ies confirmed lack of work safety and environmental protection during the company visits (see 6.1.1. on working conditions, including safety).
6.1.1 Working conditions
Working conditions in PSOM/PSI projects are in general equal or above what is
common in the countries studied. In all but one case it was possible to conduct a
group interview with employees without attendance of management, which is already a positive sign. In general, employees spoke out frankly and freely, both on
positive and negative aspects of their employment with the PSOM/PSI project.
Box 15 provides some interesting points from the group discussions in Sierra Leone, Bangladesh and Egypt.153
In four cases working conditions seemed to be less than what is to be expected
given the country context. In several projects, in particularly in the lower income
countries, the evaluation team observed that safety measures were not strictly
followed (e.g. use of protective gear). This is, however, common practice in many
countries, which does not seem affected by a once-off grant such as PSOMPSI
(e.g. requiring adaptations in regulation). In one case, those closely involved were
of the opinion that the attention to CSR (requested by PSOM/PSI), in the form of
training and involvement of outgrowers in the surrounding villages, actually negatively affected the commercial viability of the project due to the increased costs
and limited commercial opportunities (e.g. bananas production in Peru).

153

Peru and Bosnia Herzegovina were generally positive without particular points of attention.

Evaluation PSOM/PSI | 103

Box 15. Some points of interest from worker interviews154
Bangladesh




According to the workers in all of the companies in the sample, salaries
are well above the minimum wage ceiling. The salary and over-time
payment cycle is considered to be regular. In general, the basic components of the country’s labour laws are adhered to.
However, in none of the companies visited was there trade union participation (either workers were not allowed to join or discouraged or
unaware of the possibilities), even in a factory size of over 2000 workers. Workers’ Participation and/or Welfare committees did exist in 3 of
the 8 cases. In general, workers were not aware of workers’ rights issues.

Egypt
 According to the workers there was no discrimination. However, with
regard to women employment, women’s ability to work or perform
certain tasks and shifts was repeatedly said to be restricted due to stated cultural, religious or customary rules. No specials measures were
taken to expand such boundaries to improve women’s employment
opportunities.
 In one group interview the workers felt the local company was trying to
replicate the sister Dutch company, not only in the technology used but
also in the work conditions. Two companies (of the 8) supported
schools in the area to serve staff and people (often temporary workers)
in the neighborhood.
Sierra Leone
 In at least two cases, the workers voiced concerns about their working
conditions (e.g. lack of contracts and limited availability of safety
measures). Compensations for injuries and accidents at work varied a
great deal between the different cases (with several examples of no
compensation and lack of insurance cover). Observations revealed that
protective gear was often missing or not being used.
 The Ebola crisis obviously also impacted on working conditions and
workers-management relations. In several cases, the crisis was weathered by (temporarily) reducing staff and salaries. For example, in one of
the projects staff was called to a meeting to choose between either
terminating contracts or accept temporary halving of monthly salaries.
After two months, the company repaid the missing half and provided a
bonus for the hard work during the Ebola emergency response.

154

104

Because of the varying nature of the companies visited (various sectors, technologies,
sizes, countries), it is not possible to synthesis the findings of the workers’ group discussions. The findings are incorporated in the evaluation of corporate social responsibility and some of the most interesting points are summarized in this box.

|

2016

6.1.2 Community involvement
Several visited projects were involved in community development practices in the
direct vicinity of the businesses (e.g. supporting health centers, setting up charities, offering new employment opportunities). There are examples of the
PSOM/PSI grant being used to build communal training centers, however, in most
of these cases such activities were financed from own resources and already integral part of the business practices of the local partners. They were therefore not
necessarily a result of PSOM/PSI support.155
Table 6-1. PSOM/PSI impact on CSR
CSR

B

Little evidence of CSR
CSR, but no influence

156

BiH

P (9)

SL

E

U

Total

1

3

3

2

1

10

1

2

5

1

2

3

14

6

5

4

3

1

19

1

1

4

PSOM/PSI
CSR, some influence by
PSOM/PSI
CSR, strong influence

1

1

PSOM/PSI

6.1.3 Gender
It is striking how little attention there has been for gender in the PSOM/PSI programme. Gender has, for example, for a long time not been an important issue in
the application process (e.g. scoring). With the launching of PSI targets for female
employment were set, albeit with limited weight.
None of the 49 cases studied have female applicants or local partners. Only in two
cases was the interview conducted with a female project manager.
The case studies reveal that an explicit target with regard to female employment
(at management levels) might result in increase in the number of female employees (at least during project duration). However, such targets were often seen as a
target to comply with to be eligible for PSOM/PSI grant and were not necessarily
sustained after the grant period. In one case, however, project managers claimed
155

Given the fungibility of PSOM/PSI support (i.e. it shifts expenditure from the recipient
towards areas that are not funded), it is difficult to assess whether this own contribution is nevertheless linked to PSOM/PSI. However, often local partners would also engage with the community in their other businesses or declared that community involvement was part of necessary business operations, in which case the link with
PSOM/PSI was considered less strong.
156
Uganda misses two observations because in those cases it was considered too early to
determine the influence of PSOM/PSI on CSR.

Evaluation PSOM/PSI | 105

that hiring women to comply with PSOM/PSI targets made them aware of the
benefits thereof.
As such, following the DCED, PSOM/PSI can be considered a ‘gender aware’ programme (i.e. disaggregating data) but definitely not yet ‘gender mainstreamed’
(i.e. explicitly benefiting men and women) or ‘women targeted’.157 The focus of
PSOM/PSI is on the target (i.e. number of women), rather than on policies and
practices required to employ and retain women and improve decent working
conditions. Often project managers referred to the limited opportunities for
women to be employed (e.g. due to cultural norms, safety, conventional
male/female job categories) in their sector or country, instead of actively addressing barriers to female employment in the PSOM/PSI supported projects (e.g.
through transport, child support, training and sensitization).

6.2

Impact on market and sector development
The terms of reference for this evaluation requested evidence of the extent to
which PSOM/PSI worked as a catalyst to develop a market158 or a sector159 in the
country, beyond the direct effect on the supported project and company. Measuring such development impacts is beyond the scope of this evaluation. This would
have required an in-depth study and analysis of suppliers, clients and competitors
of the PSOM/PSI projects. Moreover, such impacts are expected to occur long after the PSOM/PSI grant period has expired, while projects visited were most often
ongoing or recently finalised. For example, in some cases there were potential
(temporary) negative welfare effects due to substitution of smallholder farmers
for larger commercial enterprises (e.g. poultry, pigs). The ultimate effect can only
be assessed in the longer run and with a broader evaluation perspective.
The country studies revealed that to date 18 out of 49 projects had some effect
on the sector performance (in particularly by working with of outgrowers and local suppliers). Four projects provided evidence of some impact on the market at
large, mainly through import substitutions for some goods (eggs, herbs, textile),
but also by offering consumers a more reliable or safer energy source (gas rather
than coal).160
157

Market, 2014, Guidelines for measuring Women’s Economic Empowerment in Private
Sector Development http://www.enterprise-development.org/wpcontent/uploads/Measuring_Womens_Economic_Empowerment_Guidance.pdf
158
Links with suppliers, clients
159
Import substitution, technology and innovative products for use in local market
160
Note the difference with chapter 3.4.3 on relevance, which described the extent to
which PSOM/PSI projects are rightly supported given their potential development impact. Out of 49 projects, 39 were thought to have a potential impact on the sector (in

106

|

2016

The ultimate impact on the local sector and market will among others depend on
whether the PSOM/PSI project is actually serving the local market or focused on
exporting. Several cases were identified where the local company was just a production unit for the applicant, often being the sole client. This certainly limits the
external effects of the PSOM/PSI project in a country. All in all, 13 out of 49 projects were directed solely at exports. Those projects would by nature have less effect on market development, but can still have a sectoral impact (e.g. suppliers)
over time.
Local production and import substitution are not necessarily positive developments. The impact of supported projects will depend on the efficiency of production and the ability to maintain competitiveness over foreign products. In several
cases, cheaper imports proved to be a significant barrier to longer term commercial success (e.g. eggs in Sierra Leone, pork in Bosnia). The extent to which
PSOM/PSI projects gain a competitive advantage over others in the country also
matters for the sectoral impact (as discussed in 3.6). Again, evaluating the ultimate impact would require an evaluation after a longer period of time and with a
wider perspective (e.g. including all competitors).
Box 16: Impact pilots and PSD apps
PSOM/PSI focusses on project-level interventions. Wider sector developments
are mainly influenced indirectly via the project. One of the conclusions of the
2008 evaluation was that the programme could do more to contribute to sector effects that encompass the individual project level. The evaluation recommended involving local stakeholders, embassies, and other private sector organisations. As a result of these recommendations RVO launched several
‘impact pilots’ in 2011. These pilots were aimed at improving the business climate in target countries. They were not evaluated during this evaluation. The
business environment activities now fall under PSD Apps, a RVO programme
aimed at supporting embassies with their business environment activities.

6.3

Impact on applicants
The effect on the applicants, in particular those in the Netherlands, could not be
assessed within the scope of this evaluation. In general, it is safe to assume that
the applicant (irrespective of their nationality) applied for PSOM/PSI only if they
particular through outgrowers), while 19 out of 49 projects could potentially have an
impact on the broader market (e.g. import substitution).

Evaluation PSOM/PSI | 107

were convinced of the business case of the project, which implies there is an expected benefit for the applicants. The expected benefits identified are (i) direct financial return on (financial) investments made in the local companies (increase in
share value) (ii) news sales opportunities for the applicant (e.g. seed potatoes exported from the Netherlands), or (iii) secured supplies of cheap products (e.g.
herbs, onions, furniture exported the Netherlands). In a few cases, explicit altruistic motives were stated as reasons for applicant’s involvement (e.g. piggery in
Bosnia Herzegovina, poultry in Sierra Leone).
The RVO survey among successful projects demonstrated that over half of the responding applicants experienced additional positive effects of PSOM/PSI on their
initial business (in the Netherlands). These include increases in turnover (50%),
stronger market position (39%) and increased range of products/markets (31%).

6.4

Sustainability
In the terms of reference for this evaluation, sustainability related to the continuation of the projects after the PSOM/PSI grant period. RVO monitoring information, including the spin-off survey recently conducted, was expected to provide
evidence about the continuation of completed projects. It was, however, not possible to conduct statistical analysis of those projects to determine what factors
determine the economic viability of the projects on the long run (this would have
required information from a counterfactual group to control for external factors
influencing longer-term success).
The RVOs spin-off survey revealed that 75% of the respondents continued to invest in their joint venture after successful completion of the PSI grant. The average follow-up investment amounted to € 1,000,000 per project. Most often,
(74%), resources from one of the partners or cash-flow from the joint venture
were used for the investment as opposed to external funds. For the survey sample
each 1 euro of PSI subsidy has had a leverage of 2.7 euros of additional capital, including the initial investments of the project partners.161 No such number are
known for the total all PSI projects combined nor for PSOM projects.
For PSOM/PSI to have a lasting impact, the grant must be converted into equity
for the joint venture and as such provide a longer term financial structure (rather
than merely serving shorter term expenditures). However, the assessment of this
potential longer term impact of PSOM/PSI is hindered because of the lack of a
proper audit trail in PSOM/PSI project (tracking the PSOM/PSI grant once it has

161

RVO 2016 PSI Spin-off report

108

|

2016

been transferred to the international applicant). Upon receipt of (advance) payments of grant funds the applicant subsequently transferred payments directly to
the local implementing partner or to the joint venture, often together with some
of its own investments, but without explicitly accounting properly for such transfers. Applicants tended not to request from local partners to show how funds
transferred and received were recorded. As the applicant formally remains accountable for the grant funds until final grant amount has been defined, such accounting would might not seem problematic. In practice, however, clear audit
trails cannot be made in the absence of proper recording procedures. For example, of the nine projects visited in Peru, only in two cases were management able
(or willing) to clarify how PSOMPSI funds were recorded and and even then it was
difficult to discern properly between PSOM/ PSI funds and other investments
made.
As a result of the PSOM/PSI procedures, the ownership of the funds, and related
assets, is difficult to determine. Technically, RVO remains de jure owner until it
has determined the final subsidy amount and sent a letter to the applicant confirming that RVO has indeed granted the subsidy. Only at that moment can the
applicant decide to pass on ownership of the funds (and assets procured with
those funds) to the joint venture. At that moment, part of the funds could have
been used for short term expenditures (e.g. project establishment costs, consultants) and not be traceable in financial reports. How grant funds have resulted in
strengthening the financial structure of local company or joint venture could thus
not be determined.
On first sight, country case studies indicate that after PSOM/PSI, it is business as
usual. The companies that were economically viable during the PSOM/PSI period
remain so afterwards, obviously depending on all external factors that determine
company success (e.g. economic developments, political and other crisis such as
Ebola and the Egyptian revolution). Similarly, attention to CSR continues, in particularly if it was there before and if there is a clear business case (e.g. customers
requiring certification).
Several companies, in particular in more difficult contexts such as Sierra Leone, do
point out problems with financing of follow-up investments and further growth of
the projects. Several entrepreneurs suggested a programme that would provide
loans for successful PSOMPSI projects to finance further growth.

Evaluation PSOM/PSI | 109

Box 17. Longer term sustainability of PSOM/PSI support
The evaluators had the benefit of reviewing some PSOM projects that had been
completed long ago. Examples were Skretting in Egypt (PSOM 2000), EPEC
Egypt (PSOM 2004), Sekem Lotus Egypt (PSOM 2005), all of which had hugely
expanded through repeated rounds of reinvestment, now reaching sales some
10x the volume at PSOM spin-off. Likewise, there are examples of PSI projects
in BiH that in fact follow from an earlier PSO project (e.g. Bema, Lockwood),
which also show tremendous progress over the years.
What all of these have in common was a slow start and initial underperformance (compared to PSOM plan), later more than made good once the initial
commercial and technical challenges have been overcome. This holds lessons
for the recent PSI projects that somewhat lag behind: a slow start does not forbid successful operations and growth in the future. A second lesson is that a
small number of successful PSOM/PSI projects due to their leverage can compensate for a larger number of failures.

6.5

Conclusions
The role of PSOM/PSI projects in promoting CSR is limited. CSR is often market
driven which limits the scope for PSOM/PSI to contribute to CSR. In half of the
cases studied there is some (19 cases) or strong (4 cases) influence of PSOM/PSI
on CSR.
Certification and chain responsibility are found to be promoted by PSOM/PSI.
Working conditions are mostly above the country’s average but this is partly due
to RVO’s selection procedure.
PSI focuses on female employment in numbers but much less on policies and
practices required to employ and retain women and improve decent working
conditions.
To the extent that sectoral effects could be observed 18 out of 49 cases were
found to have an effect on outgrowers and local suppliers. On the other hand, 13
out of 49 of the applicants were found to use the project to enhance their production capacity.
Ultimately, whether a business venture is commercially sustainable depends on
the entrepreneurial skills of those who become the owners of the project, once
the grant period is concluded. Several examples show that PSOM/PSI contributions have helped to launch a project that proved successful.

110

|

2016

7

Lessons learned
This evaluation has shown that in many respects PSOM/PSI has been a relevant
and effective programme. Several lessons can be learned that may be of value for
current or future PSD programmes. This Chapter discusses those lessons and proposes five related recommendations.

7.1

International partnerships
From the case studies, we learn that most of the selected projects are innovative
within context and commercially viable. 61% of the projects would not have been
realised on the same scale without the financial contribution of PSOM/PSI. The
subsidy contributes directly to employment through the newly established joint
ventures. Moreover, in some cases and contexts these international partnerships
are seen to have contributed to a type of ‘professionalization’ of the local business environment, for example by improving production processes, quality standards or management practices. While the subsidy is not large enough to distort
markets, it has the potential to develop import substitution and thus strengthen
the local economy (e.g. diversification). Similarly, quality improvements generated
through the international partnerships enabled entrepreneurs to become more
competitive internationally (e.g. packaging sector). As recognized by RVO programme managers, the interpersonal relationship between the applicants and the
local partners seems to be an important determinant of longer term success of
the joint ventures supported by PSOM/PSI.
Recommendation 1: To generate benefits to the local economy, international PSD
programmes require strong partnerships between international and local stakeholders. In most cases these partnerships will exist before the subsidy (e.g. commercial and trade relations). Moreover, subsidies and grants for medium enterprises aimed at promoting the application of innovative technologies requires
attention to the local capacity to absorb these innovations and thus professionalize (e.g. qualified and trained staff).

Evaluation PSOM/PSI | 111

7.2

Adaptability
The six country case studies revealed that one programme design (e.g. tools,
M&E, intervention logic, objectives and results) was used for PSOM/PSI in a very
heterogeneous set of countries (e.g. regarding stability, income, entrepreneurship, business environment). Minor adaptations were made for the Arab region
and fragile states (PSI Plus), and the local context was considered in the selection
process and project management based on experience of RVO programme staff
and information from embassies. Nevertheless, the PSOM/PSI programme approach provides all projects with the same funding model (grants), expecting the
same type of results (albeit with different targets). The evaluation identified considerable differences in the business environments, the capacity of local entrepreneurs, the access to finance and country risks among PSOM/PSI countries, all
of which require a more tailor-made approach. In Uganda, for example, local entrepreneurs could barely mobilize their share in the joint venture investment. In
Peru, on the contrary, the use of grants seemed to have been unduly generous,
thus limiting the relevance of the programme. This lack of flexibility regarding the
type of support affected the additionality and relevance of the PSOM/PSI grants.
Recommendation 2: International programmes such as PSOM/PSI, that aim for
results in rather distant environments, require an elaborate context and problem
analysis to inform the programme design, including adjustments to changes in the
context. When applied to such a heterogenous group of countries, a PSOM/PSIlike programme should have an adaptive programme design that responds to
country and sector specific needs (e.g. including a suite of types of support, offering choice between subordinate loans and grants, including capacity development).

7.3

Gender
Before PSI, the programme did not pay particular attention to gender. With PSI an
important step was taken to include the assessment of gender in the selection
process and add targets for female employment. As discussed in the report, while
this moved the programme towards being ‘gender aware’, it is still far off from being able to make a substantial difference for women. Indeed, the evaluation found
that there were very few female entrepreneurs associated with the PSOM/PSI
programme. Attention to female employment increased due to targets, but this
did not lead to changes in the opportunities of women (including increased employment or greater opportunities for growth, changes in management or other
ways of facilitating female workers). Often the context was used as an excuse not
to make an extra effort.

112

|

2016

Recommendation 3: A programme like PSOM/PSI has an important role to play in
stimulating gender mainstreaming (i.e. ensuring benefits for both women and
men) and perhaps even gender targeting (i.e. extra effort to include women).
From a development perspective, and in line with the programme objectives, a
more convincing focus on gender is desirable to achieve an inclusive PSD programme. This requires stimulating grantees to undertake real actions to promote
female employment, regardless of the sector and country context, such as:162
a) Encourage company policies that facilitate women in the workforce and in
management positions (e.g. flexible working hours, transport, safety
measures);
b) Encourage women as project partners (applicants or partners or otherwise involved), e.g. by increasing the weight in the selection process;
c) Pay attention (leverage support, open dialogue) to issues that affect the
position of women in the sectors in which the programme is active (e.g.
land rights, access to credit, difficulties in connecting to the market).

7.4

Trade
The 'untying' of the PSOM/PSI subsidy from Dutch entrepreneurs may run against
current practices, while at the time this change in the programme was recognised
as a good practice and in line with international agreements to make aid more effective by reducing ties with donors’ interests. Moreover, opening PSOM/PSI internationally also reduced the potential market disruption of the subsidy.
At the same time, fitting with the current Aid & Trade agenda, PSOM/PSI also provides sufficient cases in which the subsidy strengthened the position of the Dutch
entrepreneurs abroad and enabled the internationalisation of Dutch mediumsized enterprises. The position of the local entrepreneur was strengthened as well
as he got access to a subsidy that he could not have accessed otherwise.
This ‘aid and trade’ effect could have been stronger if PSOM/PSI would have been
linked more closely to other (Dutch-funded) programmes in the same sectors, including private sector development and economic diplomacy instruments supported by Netherlands embassies. Although all embassies are contacted by RVO
during the selection process, afterwards cooperation varies. Some embassies
were very active in promoting PSOM/PSI and monitoring results (e.g. Egypt, Bosnia and Herzegovina), others were unaware of PSOM/PSI operations in the country. In all of the country case studies other donors were also involved with private
sector development, but not in conjunction with each other (and thus not with
PSOM/PSI either).

162

Several guidelines for gender mainstreaming exist (e.g. from UNIDO, World Bank, GTZ,
EU).

Evaluation PSOM/PSI | 113

Recommendation 4: Create stronger links between the different Dutch (private
sector) development programmes, centrally managed or through the embassies.
Where possible and relevant, link with the work of other international actors in
the same field and sectors (e.g. other bilateral and multilateral donors, development banks) to divide labour and leverage results. In certain country context,
more connection between the Dutch programme and national government might
also be useful (e.g. using PSOM/PSI to leverage improvements in the business environment).

7.5

Boosting an enterprising culture
The evaluation of PSOM/PSI revealed that the programme has been administered
as a subsidy programme for projects, rather than a subsidy programme for new
companies being set up or expanded. Although entrepreneurs generally appreciated the rigour and discipline that PSOM/PSI imposed (e.g. in interaction between
partners but also due to reporting requirements), there can at times also be friction between the entrepreneurial way of working and the design of the subsidy
programme (e.g. regarding risk sharing and development targets). As a result, the
evaluation noted that often consultants were used to translate the language of
the programme to the entrepreneurs involved. Of course, some aspects of the
PSOM/PSI reporting and funding approaches are inherent features of public sector programmes (e.g. results monitoring on different indicators than business
would use), by nature distinguishable from, e.g. the approach of banks (who
would, however, not have financed the same projects).
Recommendation 5: Similar programmes might be able to find more of a middle
way between a business and a public sector approach to the management and
administration of the programme. This requires certain capacities of staff and advisors, but might also have implications for risk sharing arrangements. Also, the
use of consultants in different stages in PSOM/PSI-like programmes needs to be
re-assessed to ensure such programmes remain demand-driven.

7.6

Monitoring and Evaluation
This evaluation is intended to learn about the PSOM/PSI experience for the improvement of future programmes, such as the Dutch Good Growth Fund. Such a
results evaluation requires reliable information, whether qualitative or quantitative methods of analysis are used. Unfortunately, the data relating to PSOM/PSI
were found to be incomplete. To some extent, this is due to the relatively long
evaluation period, which covered intermittent changes in programme objectives
and M&E data requirements. Such changes limited the benefit of having a long
evaluation period over which longer term changes could be tracked. However,

114

|

2016

there were also data gaps that resulted from difficulties within RVO to keep the
records complete and up to date, in particular for the more recent projects. Furthermore, some key information for evaluating results and programme contribution had not been systematically collected. For example, consistent information
on rating scores and the reasons for rejecting applicants would have enabled a
more thorough quantitative investigation. At the level of grants, the focus on the
supported joint venture as a project risks missing out information about the results of the programme (e.g. regarding the ownership, longer term sustainability
of the PSOM/PSI project as an enterprise).
Recommendation 6: A sound M&E system is preferably built around the agreed
intervention logic with results categories that are broad and solid enough to withstand some changes in the programme over time (e.g. indicators can be added,
some definitions can be expanded if necessary). Not only the indicators linked to
this intervention logic, but also the data collection tools should be firmly in place
from the outset of the programme.
Monitoring indicators are to be defined and systematically collected at the levels
of activities and outputs. However, the system should also provide for links with
outcomes (e.g. through reporting by the grantees, maintaining links after projects
have officially closed), which are to be used in evaluation of longer term impacts.
In case of private sector development, monitoring (financial) inputs of different
stakeholders requires extra attention. Audit trails provide information on how
and which assets have been financed and how ownership has changed over time,
would be useful to assess the full contribution of PSD programmes such as
PSOM/PSI (e.g. regarding improvements in financial structures through increased
equity).
The M&E indicators should also be collected (and externally verifiable) without
unreasonable effort and cost for RVO, grantees and evaluators. This implies that
the indicators must remain limited in number and be simple by nature (which also
aids in their flexibility), and recorded in a way that facilitates collating and analysing information (rather than being partly digital and partly paper-based, and definitely not overwriting old values). Such a system should also allow for collecting
qualitative information.

Evaluation PSOM/PSI | 115

Annex I Interviews
People interviewed in the Netherlands
Name

Function

Date

Buchem, S.M. van

Team manager RVO

10 November 2015

Dijksterhuis, R.G.

Member of the board International Programs RVO

25 November 2015

Freitas de Sousa, L. de

Financial advisor RVO

10 November 2015

Hamers, K.

Team manager RVO

10 November 2015

Huntjens, E.N.

Project Advisor RVO

10 November 2015

Paalman, B.

APSOM/APSI member

12 November 2015

Vink, M.

Project Advisor RVO

10 November 2015

Bleeker, P.

Avance Consulting

June, 2016

Veul, J.

Head private sector development MFA

20 June 2016

Evaluation PSOM/PSI | 117

Annex II: Terms of Reference
Background information
General
The Programme for Co-operation with Emerging Markets (PSOM) was established by
the Minister for Development Co-operation at the end of 1998 with an initial budget
of 41 million euro for the period 1998-2001. The programme supported initial investments in innovative business-to-business ventures in a number of developing
countries. Over time the programme was enlarged, because of its success, both in
scope and budget and had in 2013 a budget of 90 million euro. In 2008 PSOM was
stopped, because of judicial problems. The programme could no longer be implemented in the form of an assignment, because it had all the characteristics of a subsidy. A new programme was set-up: the Private Sector Investment programme
(PSI). The final tender of PSI closed in March 2014.
The goal of the PSOM/PSI Programme was to stimulate sustainable economic development by means of fostering innovative private sector investments in developing
countries. PSI was a grant programme and focused on economic growth, creation of
employment opportunities, income generation and knowledge transfer. The grant
was accessible for (Dutch and non- Dutch) companies wishing to make an innovative
investment, in cooperation with a local business partner, in one of the PSOM/PSI
countries. The programme intended to encourage Dutch and foreign entrepreneurs to
start an investment project in developing countries in partnership with local entrepreneurs. The programme aimed at triggering an investment project that otherwise
would not have been realized because of the high product/market risks.
PSI consisted of two sub-programmes: PSI regular and PSI Plus. PSI Plus focused on
fragile states/regions, the subsidy percentage was 60%, offered extra arrangements
for security costs and more flexibility in combinations of project partners. For both
sub-programmes the same budget ceiling was applicable.
Since 2012 a separate budget and ranking was available for projects in the Arab region. The criteria for projects in the Arab region were almost the same as for projects
in other countries. The only difference was that for PSI Arab there was a preference
for projects that focused on employment of young people and women and projects
with a Local Partner owned by young and female entrepreneurs. Otherwise, the procedures and documents were the same as for projects in other countries.
The programme is implemented by Netherlands Enterprise Agency (RVO.nl) on behalf
of the Ministry of Foreign Affairs and International Cooperation.
The following PSOM/PSI phases can be distinguished.
Table 1

1

PSOM/PSI phases

PSOM/PSI period

Period

Total budget
available

Countries

PSOM 1st phase

1998-2001

€ 37.9 million

8

Evaluation PSOM/PSI | 119

2

PSOM 2nd phase

2002-2004

€ 129 million

2002: 11
2003: 17
End of 2003: 21

3

PSOM 3rd phase

2004-2006

€ 45.9 million

42

4

PSOM 4th phase

2007-2010

€ 235 million

53

5

PSI 1st Phase

2009-2010

€ 140 million

51

6

PSI 2nd phase

2011-2014

€ 232 million

59 (see country
list in Appendix
1)

Previous evaluations
In 2005 the PSOM programme was evaluated by Ecorys. The evaluation consisted of
desk research and a field study. For the desk research files of 47 completed projects
were studied elaborately. Then, 22 of these studied projects in five countries were
selected to be visited in the field. The countries visited were Ghana, Thailand, Indonesia, Tanzania and Mozambique.
In 2009, an evaluation of PSOM/PSI over the period 1999-2009 was implemented by
Triodos Facet BV. The PSOM/PSI evaluation consisted of a desk study on 60 completed projects, telephone interviews and desk research on 25 stopped projects, assessment of the EVD organisation through desk study and staff interviews and field visits
to 32 projects in six countries (Ethiopia, Ghana, Indonesia, Mozambique, Suriname
and Vietnam). The report can be found on the website of the Ministry of foreign af163
fairs .

Current evaluation
These ToR describe the expectations, scope, questions and process for the evaluation
of the PSOM and PSI projects that were selected for PSOM or PSI financing in the period 1999 -2014 and the implementation of the programmes by RVO.nl.
As a preparation for this evaluation, a quality assessment of the M&E system used by
PSI and of the data collected was carried out by an independent organisation. The
results of this study are available (Appendix 2). Also, two parties performed a study
on the possibilities for using control groups and baselines in evaluations of PSI activities, which proves to be a challenge. The results are attached in Appendix 3 and 4.
Furthermore, an inventory among projects not selected has been implemented by
RVO.nl. A copy of the findings is also enclosed (Appendix 5). Finally, in 2015 a survey will be carried out among projects that are successfully finalized and a desk
study among projects prematurely finalized. The set-up for the data analysis will be
provided as Appendix (Appendix 6). The results will become available by the end of
2015. In Appendix 7 you will find a report which provides an overview of PSOM and
PSI projects over 1999 - 2014 and an analysis of successful, unsuccessful and ongoing projects.

163

www.minbuza.nl/binaries/content/assets/minbuza/nl/import/nl/producten_en_diensten/evaluatie/afgeronde_onderzoeken/2
010/07/evaluatie_psom_psi_1999_2009_en_mmf/evaluation-psom-programma-samenwerking-opkomende-markten-psiprivate-sector-investment-programme-and-mmf-mmf-matchmaking-facility

120

|

2016

Table 2

Completed and stopped projects until December 2014

Africa

Central
and
Eastern
Europe

Asia and
the Middle East

Latin
America

Total

Total
cumulative

1999

-

-

-

-

-

-

2000

-

-

-

-

-

-

2001

-

-

-

-

-

-

2002

7

-

3

10

10

2003

16

-

4

20

30

2004

16

-

10

26

56

2005

18

-

7

25

81

2006

12

-

21

3

36

117

2007

28

-

29

6

63

180

2008

23

3

32

4

62

242

2009

24

5

24

10

63

305

2010

28

6

19

10

63

368

2011

37

4

22

10

73

441

2012

24

11

21

10

66

507

2013

44

16

14

9

83

590

2014

34

7

18

12

71

661

Number
of projects

311

52

224

74

661

661

Source: PSOM/PSI 2 Annual reports from 2000 to 2014

PSOM & PSI M&E plan
The PSOM/PSI programme was set up in 1998. Throughout the fifteen years of implementation of PSOM/PSI, Monitoring & Evaluation of results (M&E) was continuously developed and improved. In October 2011 the Ministry of Foreign Affairs, Department Sustainable Economic Development (DDE) as principal of the programme,
requested further enhancement of the M&E of its instruments focussed on private
sector development. Based on a protocol provided by the independent Policy Operations and Evaluation Department IOB from the ministry, the existing PSI M&E system
was further systematized, aligned and professionalized. New elements in PSI M&E are
impact measurement and improved focus at Quality Assurance in project monitoring.
The PSI beleidsmemorandum gave input for the identification of the majority of result indicators at output and outcome level. PSI committed itself to the main targets
set for these indicators. However, PSI cannot commit itself to any (potential) targets
of indicators at impact level, since these effects depend on a variety of circumstances
which cannot be influenced by PSI and PSI project partners.
For the PSOM programme no well elaborated M&E plan was developed. For that reason, information on some indicators mentioned in the PSI M&E plan has never been

Evaluation PSOM/PSI | 121

gathered and is not available for PSOM projects. However, as mentioned before a
survey is carried out in 2015 to capture some of this missing information.
PSI’s Result chain
The result chain was established in 2012 and has slightly been adjusted in 2015 in
order to align better with the OECD/DAC criteria.
In accordance with the standard OECD/DAC guidelines, PSI’s result chain distinguished inputs, activities, outputs, outcomes and impacts. At the input level, the result chain discerned the (total) budget and RVO.nl (human) resources. RVO.nl activities were leading to projects subsidized by PSI (RVO.nl output). The PSI subsidy was
input for the investor for the establishment of innovative businesses. For PSI this is
the outcome level: with the PSI subsidy DGIS and RVO.nl aim at influencing the behaviour of (potential) investors, stimulating them to invest in innovative businesses,
creating jobs and ensuring the transfer of technology, etcetera.
Intermediate and long term effects of the individual projects are impacts which are
expected to occur, but they are beyond the direct control of PSI and the investor. For
PSI projects, the majority of these changes are measured two years after a project
has been positively finalized (in the spin-off phase). PSI is not legally involved in the
project anymore and the project partners are fully responsible for the additional
changes they make with additional investments. The impact represents the consolidation phase of PSI projects and gives the extent to which projects are locally embedded.

Figure I Results chain, including the different levels of change

At each of the above mentioned levels of change (or result) indicators were formulated to measure results. The indicators are described below.

122

|

2016

Inputs
The PSI inputs were given facts which may change over period of time. These inputs
were: the PSI grant budget, the PSI programme budget, amount of full-time employees for execution of PSI programme, type of available knowledge and expertise
among PSI programme personnel base.
Table 3

Inputs

Input

Indicator

Grant budget

 Allocated grant budget per year

Data available at
RVO.nl

 Committed grant budget per year
RVO.nl programme

 Programme budget

Data available at

budget

RVO.nl
 Budget specified per hour, activity and pay level

RVO.nl HR & Expertise

 Amount of FTE specified per level

The availability of financial means and knowledge among project partners who execute the projects is checked during the selection procedure of PSI. This is not monitored throughout the project implementation.
Activities and output
Programme activities
The inputs are used for activities at RVO.nl as PSI promotion, project appraisal, project monitoring, financial management, project evaluation and programme revision.
The direct output consists of the projects selected and awarded with the PSI subsidy.
This can be measured according to the following indicators.
Table 4

Activities and output

Programme activity

Indicator

PSI Promotion/

 Amount of workshops given

Data available at

Communications
leads to submission

RVO.nl
 Amount of promotion sessions

of sufficient qualitative proposals
Selection/approval of

 Amount or project proposals received

Data available at

PSI projects

RVO.nl
 Amount of projects which are declined because of the
ranking
 Amount of projects which are approved (per sector /
country)
 Average programme budget spent on appraisal

(Financial) project
supervision

 Amount/percentage of failed projects vs amount of money spent (failed projects stopped in time)

Data available at
RVO.nl

 Average PSI hours spent per project per activity (ap-

Evaluation PSOM/PSI | 123

praisal, supervision, M&E)
 Average grant budget spent per project (vs planned)

Project activities
At company level inputs are used for the execution of projects. During the selection
of PSI project, PSI judges the efficiency of planned project activities. If the costs of a
project are too high, then the budget is reduced. Also during project implementation
hardware purchase is controlled by market conformity checks by an independent
body. Also project execution is supervised by verification of project sub results. PSI
will not do extra efficiency measurement during project implementation and no indicators at this level are formulated.
In January 2014 a client satisfaction study was executed. The results of this survey
are available.
Outcomes
The anticipated effects of each PSI intervention are the investment in commercial activities (establishment of PSI projects). These activities create direct development results at outcome level which are monitored and verified by PSI throughout the implementation of PSI projects. At first this leads to the establishment of innovative
businesses in a development country (1). Secondly this leads to job creation and
knowledge transfer (2). Thirdly at outcome level the company implements responsible policies with regards to CSR, gender and environmental sustainability (3). The indicators at each level of change are the following and are monitored and measured
via progress reports and project visits.
Table 5

Outcomes

Outcome

Indicator

Innovative business-

 Successful finalization of project (per sector per coun-

es established

try) (coincides with DCED indicators)

Data partially available
at end of 2015 at
RVO.nl

 Annual turnover of project
Job creation and

 Number of (direct/indirect) net (additional) jobs as well

knowledge transfer

as suppliers/outgrowers of the project (coincides with

Data partially available
at end of 2015 at

DCED indicators)

RVO.nl

 Average investment per created position
 Amount of high level and low level employees / suppliers / outgrowers trained
 Percentage of total project inputs/raw materials from
local suppliers / outgrowers
CSR, gender and en-

 Amount of female employees in project

vironmental business
policies implemented

at end of 2015 at
 Amount (and type) of implemented environmental mitigation measures in project
 Amount (and type) of obtained international certificates

124

|

2016

Data partially available
RVO.nl

in project that proof qualitative business standards
 Existence (and components) of implemented HRM/CSR
policies
 Employees: percentage of income on top of minimum

Suppliers: no data
available

loan
 Suppliers: percentage earnings on top of minimum
market prices (only if control data is available)
 Existence of an implemented outgrower scheme, collecting scheme or related scheme

Impact
At impact level, the above mentioned outcome likely leads to intermediate term results. At this level PSI measures if the development effects are locally embedded and
consolidated. Therefore, a few indicators which are measured at outcome level are
also measured at impact level. However, at impact level growth or improvement of
the indicators can demonstrate durability and consolidation. The outcome Job creation and knowledge transfer will on the intermediate term likely lead to the impact of
improved job opportunities (5). It is PSI’s objective that PSI projects do reinvestments, to increase development results. Growth indicators can say something about
the strength and consolidation of the PSI projects. It is likely that The Establishment
of Innovative businesses will on the intermediate term lead to embedding of (innovative) business(es) (4). Finally it is plausible that the Implementation of CSR, gender
and environmental business policies leads to the impact of embedded CSR, gender
and environmental business policies (6).
PSI aims to create leverage and amplify development effects on the long term over
sectors or regions. The majority of the impact indicators can only be measured on
the longer term. This will mainly be done by means of external evaluations after the
spin-off period (two to five years after the finalization of the PSI project). Nevertheless, impact measurement will also be done with the spin-off report. However, at that
point project have just finalized the consolidation phase and impact effects are expected to be low.
At first it is expected that improved job opportunities in terms of more FTE, good salaries and good prices for outgrowers are created. On the longer term PSI hopes that
it will also lead to improved job opportunities (5) for outgrowers in terms of increased sales and production volumes. Secondly, it is plausible that the embedding of
(innovative) business(es) will on the long run lead to the Expansion and catalysing
effect of (innovative) business(es) (7). PSI projects can be a showcase to other companies. With the expansion of a business, companies tend to influence governments
and/or other entities to come to an optimum working environment. New entering
companies can profit for this path which has been paved. Thirdly, embedded CSR,
gender and environmental business policies will on the longer run lead to the increased local attention for CSR, gender and environmental business policies (9). Altogether, this can contribute to the PSI objective to promote local economic sustainable development.
Table 6

Impact

Evaluation PSOM/PSI | 125

Impact

Indicator

Embedding of (inno-

 Continuation of business after the spin-off phase

vative) business(es)

Data partially available at end of 2015 at

 Spin-off (average) annual turnover in EUR (expected, re-

RVO.nl

alized at closing, realized in spin-off and at time of
measurement)
 Annual net (additional) income in EUR of the company
(coincides with DCED indicators) (expected, realized at
closing, realized in spin-off and at time of measurement)
 Amount of follow-up investment (expected, realized at
closing, realized in spin-off and at time of measurement)
 Additional amount of EURO investments which are done
for every subsidy EURO (input additionality)
Improved job oppor-

 Number of follow-up (direct/indirect) net (additional)

Data partially availa-

tunities for local pop-

jobs as well as suppliers/outgrowers (coincides with

ble at end of 2015 at

ulation / suppliers /

DCED indicators)

RVO.nl

outgrowers

Data not available
 Amount of people who moved to better positions in the
PSI project because of improved skills

CSR, gender and environmental business

 Amount of employees who work in improved working

Data not available

conditions (baseline needed)

policies embedded
 Amount of female managers
 Amount (and type) of additional international certificates
that proof qualitative business standards
 Amount (and type) of additional environmental measures
 Application of additional components in HRM/CSR policy
 Employees: percentage of income on top of minimum
loan
 Suppliers: percentage on top of minimum market prices
(if control data is available)
Expansion and catalysing effect of (inno-

 Number of events in which the project shared technical
knowledge with third parties

vative) business(es)

Data partially available at end of 2015 at
RVO.nl

 Number of times that the project is copied

Data not available

 Number of times that the project (partners) put a topic
with regards to the improvement of the business climate
on the political agenda (e.g. via newspaper, appointment
at government level)
 Amount of products/services which contribute to import
substitution (with use of new technologies)
 Amount of improved products/services for consumers on

126

|

2016

Data not available
Data partially available at end of 2015 at
RVO.nl

the local market (which are not being imported either)

Assumptions
It is the assumption that PSI projects comply with several pre-conditions. Otherwise,
projects will not be sustainable and will not be able to achieve the foreseen development results on the longer term. The pre-conditions are verified during project appraisal phase. The assumptions for proper project implementation and achievement
of development results are the following.
Table 7

Assumptions

Assumptions

Indicator

Partners

 Project partners have the capacity (financially as well as in terms of knowledge
and expertise) to carry out the project
 PSI project partners have a common goal and aim for a long term collaboration
 The PSI project partners work according to high CSR standards (ILO labour
standards, OECD guidelines and UN convention on biodiversity)

Business case

 Business plans of PSI projects are financially viable
 The technologies used in the PSI projects are suitable and commercially proven
 There are good market opportunities for the products/services of the PSI project
 The project risks are manageable and can be mitigated

Other

 The embassy has the capacity to advice during project appraisal

Scope, specific objectives and evaluation questions
Scope of the evaluation
The purpose of the 5 yearly evaluation which is tendered through these ToR is:
1. To assess the outcome of the PSOM/PSI selection process (Appendix 8), and
the additionality ex-ante of the PSOM/PSI financial contribution for the realisation of the projects; did PSOM and PSI select the right projects to achieve the
goals of the PSOM/PSI programme; do the projects selected differ from the
projects rejected, would the project not have been realized, or only on a
smaller scale, in a slower pace or with less impact, without the financial contribution of PSOM/PSI (additionality ex-ante – see note developed by DCED,
Appendix 9)
2. To assess the goal achievement of PSOM/PSI projects, which has been monitored by RVO.nl or collected in the 2015 survey, for a sample of projects.
3. To assess the effectiveness of PSOM/PSI projects, which has been monitored
by RVO.nl or collected in the 2015 survey, for a sample of projects.
4. To analyse the impact of PSOM/PSI projects (through case studies) on:
a. employees (improvement of income, working conditions, knowledge and/or
job opportunities) and

Evaluation PSOM/PSI | 127

b. clients, suppliers and/or competitors (to what extent has the investment
worked as a catalyst to develop a sector or a market in the country).
The evaluation of the programme includes several components:
the assessment of PSOM/PSI monitored output and outcome results
field studies among a representative sample of projects for project evaluation.
o This includes an investigation whether the output and outcome data on paper correspond with reality.
o It includes an assessment of the effectiveness at project level (see IOB criteria).
o collection, measurement and evaluation of – longer term - impact effects.
The PSOM/PSI evaluation must cover a representative sample of projects and countries involved in the programme. Specific questions that will be treated will vary according to the nature of the projects (stopped or completed) and must be elaborated
in the inception report (see paragraph 4.1). When selecting countries to be included
in the field work, the experiences gained with the previous PSOM/PSI evaluations of
2005 and 2009 must be taken into account.
−
−

Geographic coverage
Whereas the evaluation will cover all countries where PSOM and PSI projects have
taken place, it should concentrate on a number of countries (5-7), selected on basis
of the following criteria:
a) Geographical coverage: including regions Africa, Latin-America, Asia, Eastern
Europe;
b) Project status: at least 5 completed projects in a country;
c) Development and income levels: including at least 3 Lower Income countries
(LICs), 1 Lower Middle Income Country and 1 UMIC;
d) Fragility: inclusion of at least one fragile state;
e) Sector coverage;
f) At least one country of the PSI Plus country list and at least one country from
the PSI Arab list.
In the 2009 evaluation the following countries were included: Ethiopia, Ghana, Indonesia, Mozambique, Suriname and Vietnam.

Evaluation questions
The evaluation will be based on the OESO/DAC criteria. This resulted in the following
basic research questions that need to be answered. However, the evaluator is invited
to propose other questions to be able to better answer the main evaluation questions.
Basic research questions are:
1. Relevance of the PSOM/PSI programme: i.e. are the PSOM/PSI projects consistent with the country needs and is the intervention appropriate to realise
the objectives and did the PSOM/PSI subsidy fill a gap?
2. Efficiency of the country list and the involvement of 3rd country companies.

128

|

2016

3. Effectiveness of PSOM/PSI, i.e. determine if and to what extent:
A. the completed PSOM/PSI projects have achieved intended results and
B. the subsidy was required to realise these results.
4. Impact of the PSOM/PSI, i.e. what where the impacts on employees and on
the sector in the country?
5. Sustainability: continuity of activities.
Table 7

Evaluation questions

Criteria

Questions

Possible design

Relevance

1.1. Do the selection criteria and process ensure the selec-

Document analysis,

tion of projects in line with the objectives of PSOM/PSI?
1.2. Are the supported investments in line with the objec-

interviews, benchmark

tives and priorities of the PSOM/PSI country?
1.3. Would the projects have been realized on the same
scale, in the same pace and with the same impact without the financial contribution of PSOM or PSI (additionality ex ante according to DCED)
Efficiency

Effectiveness

Impact

2.1 How is the efficiency of the implementation of the PSI
programme by RVO affected by:
a. the PSI country list (the number of countries on the
list)
b. applicants from third countries

Document analysis,

3.1 To what extend are the targets of the projects met
(goal achievement) in terms of
− number and type of jobs created by the PSOM/PSI
supported companies?
− number of outgrowers contracted
− number of people trained / type of trainings implemented
− extend of innovativeness of projects in the PSOM/PSI
country
3.2 What factors contribute to success of projects or to failure of projects?
3.3 What are the changes in the effect variables in comparison to the situation at the start (baseline)?
− which is the attribution of the observed changes to
the intervention?

Assessment of output

4.1 What is the impact of the projects on the development

Data collection and

interviews, benchmark

and outcome monitoring data
Data analysis, Contribution analysis,
Qualitative Comparative Analysis

of the sector in the country of implementation (vertical

analysis, Contribution

linkages)?

Analysis , Process
tracing.

4.2 What is the impact of the projects on the development
of the market in the country of implementation (horizontal linkages)?
4.3 What is the impact on the employees of the PSI supported company? (when possible disaggregation between male/female employees and special attention to
impact on female employment and on decent working
conditions for women)
4.4 What is the impact of projects on the implementation of

Evaluation PSOM/PSI | 129

CSR principles? How has this affected employees/environmental issues/other local enterprises, etc?
4.5 Which other (non-intended) effects can be attributed to
the projects?
a. on actors or aspects in the PSOM/PSI country?
b. on the business of Dutch companies (applicants) in
the Netherlands (e.g. effect on employment, effect on
trade flows and/or volumes)
Sustainability

5

Given the information available (gathered by RVO.nl

Document analysis,

through monitoring and the spin-off survey) on

interviews, literature

stopped, completed and ongoing projects; how sustain-

review.

able are the PSOM/PSI investments?
a. what is the available evidence about the continuation
of completed projects?
b. which factors were important for ensuring the economic viability of the projects on the long run?
c. is it possible to give an assessment of the economic
viability of ongoing projects based on the available information?

Approach and methodology
The evaluation should provide answers on the evaluation questions. The evaluators
are supposed to use different techniques including a desk study of available (statistical) data and documentation; a survey among users; interviews of key informants,
etc. Triangulation of data is deemed essential. The Inception Report should include a
detailed approach for the evaluation.
The remainder of this section includes some ideas for the evaluation. Tendering parties are free to offer an alternative approach.
1. Assessment of the outcome of the PSOM and PSI selection process, and the
additionality ex-ante of the PSOM contribution and PSI subsidy for the realisation of the projects.
The evaluator should propose an evaluation method to assess the additionality exante of the PSOM contribution or the PSI subsidy. Additionality is defined as “the net
positive difference that is expected to result from a donor-business partnership. The
extent to which activities (and associated results) are larger in scale, at a higher
quality, take place quicker, take place at a different location, or take place at all as a
result of a donor intervention” (DCED). The DCED report attached in Appendix 9 describes eight criteria of additionality and eight principles for assessing and enhancing
additionality ex-ante. The evaluation should focus on the criteria 3 and 5:
Criteria 3: Without the public subsidy, the company would be unwilling to implement the proposed business model and/or changes in operational
standards because of a perceived negative balance of costs/risks and
benefits.
Criteria 5

130

|

2016

The cost-shared project does not displace other companies already
operating in the market, or that are ready to undertake the same project without public support.

One question the evaluator might ask, is whether these criteria were sufficiently addressed by the PSI programme ex-ante (during the project selection process).
There are several ways for assessing the additionality. First of all, the evaluators
have to analyse the additionality for a sample of projects (using document analysis,
interviews and literature on the type of investment). In addition, it may be possible
to assess the additionality by analysing the portfolio and comparing selected projects
with proposals that have not been selected If additionality plays an important role in
the selection process, it might be possible to find differences between selected projects and rejected proposals. An option seems to be assessing the probability of being considered additional as a function of a number of project characteristics (such as
country and sector of investment).
2. To assess the goal achievement of PSI projects, which has been monitored by
RVO.nl or collected in the 2015 survey, for a sample of projects.
In 2015 RVO.nl will carry out a spin-off survey. This survey is mainly directed to
gather information on outputs and outcomes of successfully finished projects. The interviews will be conducted by the current project advisors and will be integrated in
the monitoring activities of their country portfolios.
The evaluator should verify the monitoring data collected by RVO.nl, and the data
collected through the spin-off survey in 2015.
3. Assessment of the effectiveness of PSOM and PSI projects, which has been
monitored by RVO.nl and gathered via the 2015 survey, for a sample of projects164;
The evaluator should assess the effectiveness of PSOM and PSI, using monitoring data collected by RVO.nl, and the data collected through the spin-off survey in 2015,
through desk study and data collection during field visits for a sample of projects. It
is suggested to make use of surveys and/or semi-structured interviews for data collection. Draft survey/interview protocols should be provided during the Inception
Phase, allowing the Reference Group to give feedback.
4. Gather and analyse data on the impact of PSOM and PSI projects (through
case studies) on:
a. employees (improvement of income, working conditions, knowledge
and/or job opportunities if feasible with gender disaggregation) and
b. clients, suppliers and/or competitors (to what extent has the investment worked as a catalyst to develop a sector or a market in the country).
Although the PSOM and PSI programme monitors the direct effects of projects (which
will be verified by the evaluator as described above), there is limited knowledge on
the impact that projects have on their employees and on the sector/market. Although
these impacts are highly project specific, the evaluator is asked to explore the impacts through case studies of several projects. This can be done rigorously (e.g. by

a.

164

To assess additionality ex-post, the evaluation should ideally make use of an explicit counterfactual.
However, a counterfactual might be a serious challenge, given the nature of the PSI programme. In
2012 AIID and Erasmus University have been asked to give an advice on using baselines and control
groups in the evaluation of PSI. It was concluded that as many PSI activities created new firms, baselines were not possible. The feasibility of using control groups for impact evaluation was highly project
specific and dependable on the PSI indicator. A copy of both reports is included (Appendix 3 and 4).
The evaluator is invited to come up with a proposal on how to address the additionality question and
to perform an evaluation of sufficient quality given the circumstances.

Evaluation PSOM/PSI | 131

making use of a counterfactual scenario, for example at the employee-level, or at the
level of clients, suppliers (e.g. smallholders) and/or competitors), or less rigorously
(e.g. with qualitative interviews). Although more rigorous methods are preferred, the
evaluation budget might not allow for this.
The evaluator is asked to come up with a method to assess the impact of PSOM and
PSI projects on employees and/or the sector/market to the best extent possible given the budget.
The impact evaluation is supposed to combine monitoring information with field
work, using a larger sample (based on monitoring data) for statistical analysis and a
smaller sample for more detailed research. For the field work a sample of projects is
required. While most projects are still being implemented, the evaluation should focus especially on completed projects. For reasons of efficiency, it is proposed to focus
on countries with relatively many (completed) projects. The evaluation will require
about 5-7 countries for the field work.

Deliverables and schedule
The proposed deliverables are presented in the schedule below, as well as an estimation of the planning. The evaluator is free to comment on the proposed planning.
Table 8

Deliverables and schedule

Subject

Deliverable

Estimated / pro-

Responsible

posed Timing
Expression of interest

These Terms of

(EoI) sent

Reference

Receipt EoI

Response by email

June 22rd, 2015

Meeting Committee

Ministry of Foreign Affairs

June 30st, 2015

Parties invited
to send in EoI

Request for Concept

These Terms of

notes (CN)*

Reference

Receipt of CN

Standard Template

July 3th, 2015

Ministry of Foreign Affairs

July 17th, 2015

Parties invited
to send in CN

Invitation to Tender

These Terms of

th

August 6 , 2015

Reference
Final date for posing

List of questions

questions by tenderers

about the ToR

Ministry of Foreign Affairs

st

August 21 , 2015

Parties invited
to send in proposal

Receipt of Full Pro-

Proposal (Standard

posals

Template: Appendix

th

August 28 , 2015

to send in pro-

10)

posal
st

Selection of evaluator

Parties invited

September 21 , 2015

Ministry of For-

Yes

eign Affairs in
cooperation with
reference group
PHASE 1

Inception phase

Start

Proposal for the
process in Phase 1
and 2

132

|

2016

September 2015

Evaluator

Yes

Sample selection: Se-

Proposal for sample

September - October

IOB / DDE / ex-

lection of representa-

selection

2015

ternal evaluator

Desk study (over all

Validation of pro-

October – November

Evaluator

projects) for

jects output and

2015

 validation of input

outcome

Yes

tive sample of projects
and project clusters for
project evaluation

and outcome data
 implementation by
RVO.nl
Discussion draft Incep-

Draft Inception Re-

tion

port (including fea-

Report with reference

sibility report on

group

how a counterfac-

December 2015

Evaluator

Yes

January 2016

Evaluator

Yes

January – June 2016

Evaluator

January – June 2016

Evaluator

tual can be
achieved)
Receipt final work plan

Work plan (includ-

for phase 2

ing indicators, research questions,
selection of projects)

PHASE 2

Evaluation

Kick-off
Field research of repre-

Evaluation of pro-

sentative sample of

ject sample

projects for validation
of input and outcome
data and evaluation of
efficiency and effectiveness
 Data collection of

Project impact

sample of PSI pro-

measurement and

jects, clusters and

evaluation

stakeholders of projects which passed
the spin-off phase
 Data collection control groups (if applicable)
 Comparison to baselines and control
groups (if applicable)
 Consideration of
other indicators

Evaluation PSOM/PSI | 133

which can attribute
to impact
 Data measurement,
analysis and evaluation
PHASE 3

Reporting

Discussion Draft Final

Draft Final report

September 2016

Evaluator

Yes

Final report & ac-

October 2016

Evaluator

Optional

report
Receipt Final report

companying
Database

*When less than four (4) parties send in an Expression of Interest, the Request for Concept Notes can be
omitted and interested parties will be requested to directly send in their Full proposal.

Phase 1 Inception Phase
Given the complexity of the evaluation we want to start with an Inception phase. The
aim of this phase is to make an inventory of the available information and to prepare
the design and methodology of the evaluation. The inception phase will result in a research protocol as part of the inception report. The inception phase covers the following topics:
−

The evaluation team will be briefed by the Ministry of Foreign Affairs and the
PSOM/PSI team at RVO.nl. The evaluation will be accompanied by a reference
group established and chaired by DDE. The evaluation team will start the
evaluation at RVO.nl office in The Hague with the examination of relevant
background materials and documentation. The main outcome of this phase
will be the elaboration on and development of a detailed evaluation matrix.

For the assessment of the effectiveness and relevance of the programme a
desk study will be performed on a certain number of completed PSOM/PSI
projects in the 5-7 countries that will be visited. For each country to be visited
a number of projects will be selected from the sample of projects reviewed in
the desk study. For the assessment of the effectiveness and efficiency of implementation by RVO.nl an extra sample of stopped and running projects will
also be viewed by the evaluators, through desk study and interviews. No client satisfaction survey is required as this will be done separately by the
RVO.nl. The results of this survey are available.
During the inception phase the evaluation team will have the opportunity to discuss
the details and logistics of the subsequent phase of the field work in consultation with
the reference group and the Embassies concerned.
−

At the end of the inception phase the consultants are expected to submit an inception
report, which will contain:
 Detailed evaluation matrix, evaluation criteria and methodology;
 Proposal how to include control groups in the evaluation (if applicable);
 Detailed proposal for the field studies to be undertaken;

134

|

2016

 Questionnaire for the survey to be conducted for the projects in the 5-7 countries;
 Detailed schedule of work, including field studies in the 5-7 countries and proposed projects to be evaluated, and list of main persons to be interviewed;
 Any further data requirements from RVO.nl in order to fulfil the project;
 Proposed table of contents for a draft version of the final report, including a
brief overview of suggested structure and content of each chapter. Suggestion
of lay-out of the project data sheet per project visited.
The inception report will be discussed with the reference group. A final agreement on
the details of the evaluation methodology needs to be reached between the reference
group and the evaluators.

Phase 2 Evaluation
The field studies will concern visits to the selected PSOM/PSI projects in the 5-7 developing countries, to the Netherlands Embassies and the Government counterpart
ministries or other relevant institutions in recipient countries. The evaluation team
will brief the Ministry of Foreign Affairs (DDE) and RVO.nl on the data it has collected
after the first country visit(s).
The team will present one or more interim reports based on the field visits to the reference group for comments.
The reports should be written in English and comply with the IOB 'Evaluation policy
and guidelines for evaluations'. The reports should be understandable for people who
are not familiar with methodologies used in impact studies. The methodology used
should be presented in appendixes.
Furthermore, it should be clear what the judgement of the interviewee is and what
the judgement of the evaluator is. The source of the information provided should be
clearly indicated.

Ownership and usage of data
Evaluation reports will be published in name of the evaluator as an independent project evaluation. After approval of the report by the Reference Group the report will be
published on the PSI website. Discussions on certain topics between the evaluator
and the Reference Group will be addressed in an appendix to the report.
The report and all underlying data (survey data and other data collected as part of
the evaluation) will be owned by the Ministry of Foreign Affairs. The (anonymized 165)
data will become publicly available at the time of publishing of the report. As soon as

165

If it is impossible to anonymize the data at a specific level, anonymising of the publicly available dataset
may be ensured by aggregation or by combining regions. Researchers with full access to the data will
be in a position to analyse the data at the lower level of aggregation for the report, provided that they
respect the agreed anonymity.

Evaluation PSOM/PSI | 135

the report and data are published by the Ministry, the evaluator is free to use reports
and data for other scientific uses166.

Institutional arrangements and guiding principles
Guiding principles and values
The evaluator should adhere to the guiding principles and ethical standards described
in Chapter 5 of the IOB 'Evaluation policy and guidelines for evaluations' and should
make explicit in the proposal which code of ethics they adhere to (and attach a
signed version).

Reference Committee
For this project evaluation, a reference committee will be formed, with the following
representatives:
Antonie de Kemp – IOB
Geert Thijssen - Quality Assessment Unit, RVO.nl
Job Runhaar - DDE
Frans Baneke - Member APSI
Sylvia van Buchem - RVO.nl
Marjolein Vink - RVO.nl
Els Huntjens – RVO.nl

Support for local clearance
PSI will, through the Applicants of the projects, support the evaluator in receiving
clearance for performing local data collection (surveys) and meeting the local stakeholders of the projects.

Submission Guidelines and Assessment
Communication
166

For this reasons, two amendments have been made to the ARVODI:
- Notwithstanding Article 11.5 of the ARVODI the following applies: The contractor is allowed to remain
working papers containing copies of relevant documents in respect of the engagement. The working
papers are the property of the Contractor.
- Notwithstanding Article 23 ARVODI the following applies. The Contracting Authority is the owner of all intellectual
property rights that may be exercised now or in the future in relation to the results of the Services performed by
the Contractor, irrespective of where and when they may be exercised. However, the Contracting Authority will
not make changes to the reports written by the Contractor and published in name of the Contractor. The Contracting Authority (and Reference Committee) can ask the Contractor to make amendments to the report before
approval and publication, in case of incorrect data, methodological errors and other mistakes or lack of clarity in
the proposed report. Moreover, the Contracting Authority, taking into account the advice of the Reference Committee, decides about publication of the report. The report may not be published if the quality is below standard.

136

|

2016

All communication relating to this tender needs to take place with the following contact persons at the Ministry of Foreign Affairs, Mr. Job Runhaar
(job.runhaar@minbuza.nl) and at RVO.nl, PSI unit, Els Huntjens
(els.huntjens@rvo.nl).

Lodging the quotation
Proposals should be submitted using the template attached in appendix 10 (Guidelines full proposal). The final deadline for submitting the quotations is August 28 th,
2015 at 12.00h CET, this is a strict deadline. The quotation must be transmitted by
email to Job Runhaar and Els Huntjens. Quotations which are not sent in the prescribed manner and/or are sent after the final delivery date and time will be put
aside.
The template in Appendix I0 should be used for the quotation. The quotation needs
to be complete; that is to say, all requested supplements, or other information, need
to have been included. If in your opinion there are other matters of interest, you can
raise these matters in a separate supplement.
We stress that the tenderer is responsible for the completeness of its quotation. An
incomplete quotation can lead to exclusion from the tender process. Your quotation
needs to be signed by the authorized official.

Assessment of the quotations
The decision will be announced latest September 21st, 2015 through an email to the
contact person given in the quotation. The contract will be awarded to the economically most advantageous tenderer, based on the award criteria, including the relevant
weighting as described in Chapter 9 below. The economically most advantageous
tenderer is the tenderer with the highest definitive final score. The contracting authority rounds off tenderers’ final scores to one place after the decimal point. To determine definitive final scores, the marks are not rounded off.

General
The contracting authority reserves the right to stop this tender completely or partially, temporarily or permanently, up to the moment of signing the Agreement. In such
a situation, tenderers are not entitled to any compensation or to reimbursement of
any costs incurred within the framework of this tender. By lodging a quotation, the
tenderer declares to be in agreement with these conditions and all other conditions
referred to in this call for tenders.
Delivery, payment and/or other conditions - however referred to – from the tenderer
are expressly excluded. The General Government Terms and Conditions regulating
the issue of assignments to provide services (ARVODI 2011, English version) are applicable as stated in the Framework Agreement Article 7.1.
The contracting authority is not responsible for any costs involved in the preparation
and release of a quotation, including the provision of any further information. Any
costs and/or losses which (can) arise from this tender not being granted (to the tenderer) are at the risk of the tenderer.

Evaluation PSOM/PSI | 137

Any dispute between those involved in the tender process which might arise by reason of this tender process will only be brought before the authorized Court in The
Hague. Only Dutch law is applicable.
By lodging a quotation, the tenderer guarantees the accuracy of all requested and
delivered data and agrees with all specifications and conditions indicated in this document.
The tenderer’s quotation will not contain any reservations. By issuing a quotation the
tenderer declares that he agrees with this condition and all other specifications from
this tender. A quotation containing one or more reservations will be excluded.
The contracting authority reserves the right to put a quotation aside, and to exclude
the tenderer from any further participation in the tendering process if a quotation
does not conform to the specifications contained in this chapter.

Questions
The final date for lodging questions relating to the tender is August 21st, 2015 12.00h
CET. All answers will be sent by email as well as made available to all organisations
invited to submit a proposal. Questions can be directed through email. The answers
(Summary of Information) will constitute an integral part of this call for tenders.
The contracting authority will assume that there is no lack of clarity relating to components about which no questions have been raised. In case of inconsistencies between the call for tenders and the summary of information, the summary of information will prevail.

Contract and payment
Budget
The indicative budget for the evaluation is € 300,000 (VAT included). This budget includes all deliverables described in Chapter 4; an inception report, a baseline report,
midterm report and a final report. Tenderers may want to offer additional evaluation
possibilities over and above the specification (such as additional midterm report(s)
and/or additional studies), in which case these optional proposals should be budgeted
separately.

Contract
A contract will be signed with the winning tenderer for the evaluation.
The evaluator will perform the Services specified therein for a fixed maximum price,
including VAT, or should mention VAT reverse charge if applicable, as given in the
quotation following these ToR. It is possible that the scope and budgets for the different phases are changed due to progressive insights of the evaluator. In this case
the evaluator will advise the Ministry of Foreign Affairs of the changes. The evaluator
will only implement the changes after the reference group gives its consent. The
evaluator guarantees that the sum (the fixed maximum price) will not be exceeded.
The Ministry of Foreign Affairs may also cancel specific deliverables or parts thereof
due to progressive insights. Consequently the budget for these individual deliverables
can be cancelled or adjusted by the Ministry. Inflation can only be corrected according to inflation rates in the Netherlands. The fixed maximum price should include all

138

|

2016

the deliverables as described in chapter 4 above. The tenderer is free to include other
deliverables (such as additional midterm report and/or additional studies), which will
be considered as optional. The Ministry can decide whether or not they will make use
of these services. The evaluation of the price (please refer to chapter 9) will only be
based on the (non-optional) deliverables for phase 1, 2, 3 and 4 as described in
chapter 5 above.
The tenderer should guarantee that the project leaders mentioned in the proposal will
be available for the project. Changes related to project leaders should be presented
to and accepted by the reference group.

Award criteria
Award criteria

Weighting
factor

1

Evaluation team

45%

Coverage required expertise by team. Expertise means both availability and contribution of team members on one hand and level and type of relevant available
knowledge on the other.
a

Expertise international team leader

15

 CV (max. 1)
 Interview team leader
b

Expertise other international team members

10

 CVs (max. number will be indicated)
c

Expertise local team leader / members (if applicable)

5

 CVs (max. number will be indicated)
d

Elaboration on how team composition and available expertise (excluding meth-

10

odology) will guarantee results
e

Input on this subject has to match the information given in the LoI,

5

or Concept Note.
2

Methodology: technical proposal

45%

a

Address evaluation quality criteria

10

b

Additional specific methodology and elaboration on how results are guaranteed

20

c

Implementation plan and calendar

5

d

Elaboration on who is responsible for which aspect of the project

5

execution (methodology).
e

Input on this subject has to match the information given in the LoI,

5

or Concept Note.
3

Price: Financial proposal

10%

Evaluation PSOM/PSI | 139

Ad 1: The qualifications and experience of the team as a whole (if more than one
person is required). This is assessed on the basis of CVs.
In general:


Existing collaboration between network partners on the specific subject will
be an advantage in the assessment of the proposal;



the technical ability and experience of the team will be assessed on the basis of the CVs and interview (team leader);



Quality measurements must be taken in order to guarantee the required
quality level (see schedule IV and schedule V).

Ad 2: A technical proposal will be requested in which the candidate elaborates a
more detailed methodology and planning of the evaluation. This is used to
judge the tenderers understanding and approach of the assignment.
Ad 3: A quotation should be provided that falls within the maximum range stated in
the tender document and provides details about man-days, fees for the various evaluators and details of all other costs. The quotation is used to assess
the cost of the evaluators and the total price, as well as the overall feasibility
of the proposal. Also a maximum fee can be set as a requirement per impact
evaluation.

140

|

2016

Annex III: Evaluation team and reference committee
The evaluation team was composed of:







Philip de Jong (APE)
Phil Compernolle (APE)
Bert van Manen (Timpoc)
Klaas Molenaar (Timpoc)
Karen Rijen (MDF)
Mike Zuijderduijn (MDF)

The reference committee was composed of:
 Anthony de Kemp (IOB)
 Geert Thijssen (RVO)
 Job Runhaar (DDE)
 Frans Baneke (Member APSI)
 Sylvia van Buchem (RVO)
 Marjolein Vink (RVO)
 Els Huntjens (RVO)

Evaluation PSOM/PSI | 141

Annex IV: Case study interview guide
This is the interview guide as well as the reporting format for the interviews with the local management. The interview with the applicant can be reported in this same format (but please reference those statements separately). The interview is semi-structured so the order of questions
may be changed as seen fit during the interview. List and rankings (green sections) to be done by
the interviewer, reflecting perceptions of the interviewed persons (please color your own observations to distinguish from the quotes/information from those interviewed). Separate reporting
format for interviews with employees.

General information 167
Date of interview
Project/Company name
Names informants + functions + contact details
Personal involvement of those interviewed with PSOM/PSI?168
(If so, what function)
Remarks169

PSOM/PSI origins170
1
a

b

Relationship with applicant:
When/how did you meet
your
(Dutch/foreign)
partner (the applicant)?
Did you work together
prior to PSOM/PSI? If so,
how?
What kind of arrangement did you make for
the project?171 Why?

167

For general information on the company, see files RVO (to be read before the visit).
For example, did the person being interviewed also participate in the design, in the implementation of the project,
do they personally know the applicant or was that his/her predecessors?
169
Points of attention with regard to the company/project/applicant (from files) that need to be taken into consideration throughout the interview/report should be mentioned here (e.g. tensions between applicant/partner, unfortunate events…)
170
Part of the assessment of relevance
171
For example, legally established joint venture, nothing formal, subcontract, MoU, …
168

142

|

2016

c

When was it formalised?172
On what date was the first
investment made (with
PSOM/PSI)?
How is the financing determined (use of subsidy)?
How is the current rela- None / good / bad
tionship?
Clarification/evidence173

2
a

Origins of PSOM/PSI project
When and how did you 
first
hear
about
PSOM/PSI

b

What were the main mo- 1. Access to foreign investment
tives to join the pro- 2. Access to technical assistance and training
3. Access to foreign exchange
gramme
4. Access to innovative processes/products…
5. Access to new markets
6. Expand existing commercial relations with foreign partner
7. Other (specify)

c

How and by whom was
the project proposal developed? How much
were you involved?
Agency NL / EVD/ RVO
What is your experience None / good / bad + clarification/evidence175
with Agency NL / EVD/ 
RVO?174


3

Who is involved (or could be)? – additionality
4

Apart from PSOM/PSI, what other (public and private) parties are or were involved with the
project? With what inputs/activities (investment, knowledge transfer, CSR incentives, innovation promotion, other)?176

172

This question is added because several projects stopped shortly after having been selected because partners could
not come to an agreement about the division of tasks and funds (despite having submitted a ‘joint’ proposal). Answer to be compared with RVO monitoring data.
173
Examples that provide evidence of the state of the relationship (e.g. contact moments, appreciation or not…)
174
The first point of contact for RVO is with the applicant. However, in many cases, RVO did visit the projects (and
might for example have been asked to follow up on specific issues).
175
For example, knowledge of RVO, contact moments, appreciation
176
‘None’ is also a possible answer.

Evaluation PSOM/PSI | 143

e.g. Government, Private investors, Business associations, Other partners177
Organisations
Input/activities
1
List
5

What would have been other sources of investment/knowledge transfer if not PSOM/PSI?
178

Why not used? e.g. Government, Private investors, Business associations, Other partners
Organisations
Type of funding
1

Comment on rating

6

Would the project have happened without PSOM/PSI? What would have been different?
Prompter: would you have applied for a PSOM/PSI loan rather than a grant?

Effectiveness: project goals
7

Comparison of goals with achievements179
(using the project document and the company fiche that lists key anticipated results)
Per target: Not achieved / partly achieved / fully achieved
Target
Achievement

1
Notes

2
Notes

3
Notes

177

Initially the other actors were to be compared to PSOM/PSI PSOM/PSI as possible substitute (S) or complement (C),
and being less (1), equally (2) or more (2) important for the project’s success than PSOM/PSI. However, this option was dropped from the questionnaire.
178
‘None’ is also a possible answer.
179
Targets to be extracted from project documentations. Rows can be added (if need be during the interview to document targets not mentioned in project documentation).

144

|

2016

4
Notes

8

Other results achieved (not anticipated or not listed in fiche)

9

List critical success factors:
1.
2.
3.
4.
5.

10 List major barriers to success:
1.
2.
3.
4.
5.
11 Extent to which PSOM/PSI contributed to these achievements (how and if not why not)?180
a
Investment in hardware:
1. Negligible / 2.one of the factors / 3. one of the main factors / 4.crucial
Clarification
b

Knowledge transfer, training and technical assistance:
1. Negligible / 2.one of the factors / 3. One of the main factors / 4.crucial
Clarification181

c

Direct employment creation (per skills level):
1. Negligible / 2.one of the factors / 3. One of the main factors / 4.crucial
Clarification

d

Other:182
1. Negligible / 2.one of the factors / 3. One of the main factors / 4.crucial
Clarification
Assessment of the evaluator based on observations:
0 = unsuccessful project
1 = Negligible contribution of PSOM/PSI to results

180

PSOM/PSI = subsidy as well as cooperation with applicant
Through training, (management) cooperation, experience, other?
182
Can be more than one
181

Evaluation PSOM/PSI | 145

2 = PSOM/PSI one of many factors contributing to results (not crucial)
3 = PSOM/PSI one of the main factors contributing to results
4 = PSOM/PSI crucial to achievement of results (indispensable)
Clarification (incl. key area of contribution?)
12 How innovative is this project?
a
Y Innovation of product / service / management / production method / means of service provision
N Not particularly innovative
Clarification
b

Are there comparable projects/products in the country (examples)?

c

What were the main competitors at the start of the programme? And how is the current situation?
Assessment of the evaluator based on observations:183
0 = not innovative in any way
1 = new for the company (in the country of application)
2 = new for the company and new in the sector (in the country of application)
3 = new for the company, new for the sector and new for the country
4 = a world first (new globally)
Clarification

13 Corporate social responsibility
a
How active was your company with regard to CSR before the project?
Not / somewhat / very + examples
- Working conditions
- Environment
- Gender
- Other aspects?
b

How important is CSR in the project?
Not / somewhat / very + examples
- Working conditions
- Environment
- Gender
- Other aspects?

c

Has the project affected CSR practices in other parts of your company? In what way? Or why
not?
Not / somewhat / very + examples

183

Categorisation roughly based on AECF rating. Innovation serves as a proxy indicator for (-) market displacement

146

|

2016

d

Working conditions
Environment
Gender
Other aspects?

Since you started with CSR, has this been taken on board by others in your chain, market, sector (e.g. copied)? In what ways? Or why not?
Not / somewhat / very + examples
- Working conditions
- Production processes
- Environment
- Gender
- Other aspects?
Assessment of the evaluator based on observations:
0 = little evidence of CSR
1 = no added value PSOM/PSI with regard to CSR
2 = CSR of project supported by PSOM/PSI (some influence)
3 = CSR of project strongly influenced by PSOM/PSI
4 = CSR of project influenced by PSOM/PSI and spill over to others
Clarification

IF PROJECT STILL ONGOING during/after subsidy184
14 Key data project185
Value sales

Target

Current

Employment (fte of the Target
project)
Current jobs
Male/Female

Current

High/low skill level
Flexible/Fixed contracts
Younger than 25 years
Evidence of upscaling
Follow up186 (# and Target
sum)

Current

184

The analysis will take into account differences between categories of companies (e.g. ongoing projects or those
that have been finalised for several years but are still active).
185
Some of this information is in the company fiche, in which case this serves as a validation. Other information
should be collected during the interview.

Evaluation PSOM/PSI | 147

# of outgrowers/contractors
# of locations
Other
15 What are your expectations for the future of the project (in case of still active)

Wider impact
16 Has the project affected other parts of your company?187 If so, how? With examples
1.No / 2.somewhat / 3.a lot / 4.crucial
17 Has the project affected others in the sector (e.g. suppliers, outgrowers clients)? If so, who
and how?188 With examples
1.No / 2.somewhat / 3.a lot / 4.crucial
18 Has the project affected others in the market/country? If so, who and how?189 With
examples
No / somewhat / a lot / crucial

Wrap-up question
19 What do you recommend for future investment programmes? What should definitely
continue / be stopped?

Summary and personal assessment by evaluators190
Relevant (Does the company and project fit the selection criteria and objectives of PSOM/PSI?)
Y/N + clarification
Additionality (Could the project have happened without PSOM/PSI at about the same scale,
186

New investors
E.g. employment, working conditions, technologies, innovation…
188
E.g. employment, new technologies adapted, working conditions
189
E.g. replication of business models and/or of technology, changes in working conditions, laws and regulation
187

190

These answers will be discussed in depth during the workshop for the country experts to ensure consistency in assessments. The information above will substantiate the seemingly simple answer Yes/No (as there is obviously
always more nuance and middle ground). However, providing clear answers to the main questions, will facilitate
the synthesis of findings within broad categories that will be elaborated on with more detailed examples and clarifications.

148

|

2016

pace and impact?)
Y/N + clarification
Effectiveness (expected results achieved)
Y/N

Contribution PSOM/PSI191
No / one of many / crucial

Turnover/sales
Direct employment
Knowledge transfer
CSR
Other
Impact (evidence of results beyond project)
Y/N

Contribution PSOM/PSI192
No / one of many / crucial

Local company
Sector
Market
Other
Embeddedness of the business (Is the project sustainable after PSOM/PSI or does it seem to
have the potential to be sustainable?)
Y/N + clarification
Other remarks by evaluator

191
192

No causal link / one of many factors / significant causal link
No causal link / one of many factors / significant causal link

Evaluation PSOM/PSI | 149

Annex V: Employee survey guidelines
Topic sheet group interview employees on working conditions
The interview with a group of employees will be used to validate the information with regard to
working conditions and CSR, obtained through the PSOM/PSI reporting and interviews with the
local project management and applicant.193 This sheet should be used to report the answers to
the questions posed by the interviewers in ways that are most appropriate for the specific context and culture.194195
The group consists of 5-10 employees together, both male and female, (having been) working in
the PSOM/PSI project at different levels (non-management).196
After introduction, description of purpose of interview, confidentiality,
List people in the group, names, age, sex and function:

Y N Comments197
Work
People in this company
earn at least minimum wage198
What is this wage (more or less)?
Get paid on a regular basis
Daily? Weekly? Monthly?
Always get paid for
overtime
193

The aim of this exercise is to confirm that the working conditions in the PSOM/PSI projects are as expected. It is difficult to measure change, and in particular attribute this to PSOM/PSI (even more so because CSR is a precondition as well as an expected output, moreover the attention to CSR varies over time and within RVO). Nevertheless, at the end of the discussing an attempt will be made to determine changes in the past years, which the
evaluators can link to the PSOM/PSI timeline.
194
The questions draw on ILO’s Decent Work questionnaires (abbreviated), using internationally accepted aspects of
decent working conditions.
195
Questions can be posed directly to the group (‘how many of you…’) or asked in general for the company. Y is the
answers if the majority of people answer positively, or if the group agrees this is the case for the company (with
possible exceptions). Otherwise N.
196
Given the scope and nature of this evaluation (with a focus on the PSOM/PSI programme managed by RVO and
funded by the Ministry of Foreign Affairs, rather than being focused on CSR of participating companies), it is unavoidable that these employees will be selected by the project management, interviews are conducted on-site and
the evaluation does not allow for third party involvement (e.g. trade unions).
197
Statements to be answered with Y/N, nuances (e.g. differences between levels, people) to be noted in comment
section. It is important to probe/double check answers (examples provided in the list, to be added depending on
type of project/sector/country). Observations of interviewers also to be noted in the comment section (in a distinct colour).
198
To be determined and assessed by the local expert (Note: there might be differences between legal minimum wage
and wat is considered a minimum living wage. In that case living wage is to be assessed.).

150

|

2016

Are always over 15
years old
How old is the youngest employee? What does he/she do?
Are allowed to terminate employment
at will or after serving a notice
Have the right to
join a trade union199
What is the union called?
Y N Comments
Holidays
People in this company
Get a weekly rest
period of at least 1
day a week
What days are off?
Y N Comments
Employment security
People in this company
Receive a written
statement of particulars at the start of
employment200
Does everybody have a contract? Signed?
Can work on tasks
of a permanent nature without fixed
term contract (i.e.
as contract workers)
Are there more flex or fixed contracts? What tasks are flex contractors?
Get due notice before termination of
employment contracts
Receive severance
pay in case of sudden termination of
employment
Do you know of people who have received this severance pay?
Y N Comments
Sickness and employment injury benefits
People in this company
Receive paid sick
leave during the
first months201
199

Obviously, this question is only to be asked in appropriate context (i.e. when legal in the country)
contract, appointment letter or other
201
Note % of wage and for how long?
200

Evaluation PSOM/PSI | 151

Do you know of people who had this happen to them? In this company? Elsewhere?
Are adequately
compensated in
case of an accident/work injury or
occupational disease
Do you know of people who had this happen to them? In this company? Elsewhere?
Y N Comments
Maternity
Women in this company
Can adapt their
work (e.g. hazardous work, night
shifts) during pregnancy
Are protected from
dismissal during
pregnancy
Have maternity
leave (at least partly
paid)202
Do you know of people who had this happen to them? How long is maternity leave?
Y N Comments
Health and safety
People in this company
Have a safe and
healthy workplace
Use protective
equipment, including clothing, free of
cost to them
Examples of safety measures (and unsafe situations)?
Y N Comments
Fair treatment
People in this company
Receive equal pay
for similar/equal
work without discrimination
Are treated equally in employment opportunities (appointment, promotion, training…) without discrimination of:
Sex/gender
Religion
Political opinion
Place of birth
Trade union mem-

202

Note % of wage and for how long?

152

|

2016

bership
Other?
Do you have examples of ways in which women are supported in their work?
Do you have examples of discrimination that was halted/punished?
Training
What types of training have the employees received in the past 5 years?203
Was this useful and Y N Comments
why (not)?
Health and Safety
…
…
Change over time
In the past few years, what have been the main changes in the areas discussed (positive/negative)?204 How did this come about?

Overall comments of interviewers on process205
Topic sheet group interview employees on working conditions
The interview with a group of employees will be used to validate the information with regard to
working conditions and CSR, obtained through the PSOM/PSI reporting and interviews with the
local project management and applicant.206 This sheet should be used to report the answers to
the questions posed by the interviewers in ways that are most appropriate for the specific context and culture.207208
The group consists of 5-10 employees together, both male and female, (having been) working in
the PSOM/PSI project at different levels (non-management).209
After introduction, description of purpose of interview, confidentiality,
List people in the group, names, age, sex and function:

203

Or during the PSOM/PSI project period.
If possible, to be linked to the PSOM/PSI project period.
205
Attitude of the group, possible differences between answers of men and women, credibility of the answers,…
206
The aim of this exercise is to confirm that the working conditions in the PSOM/PSI projects are as expected. It is difficult to measure change, and in particular attribute this to PSOM/PSI (even more so because CSR is a precondition as well as an expected output, moreover the attention to CSR varies over time and within RVO). Nevertheless, at the end of the discussing an attempt will be made to determine changes in the past years, which the
evaluators can link to the PSOM/PSI timeline.
207
The questions draw on ILO’s Decent Work questionnaires (abbreviated), using internationally accepted aspects of
decent working conditions.
208
Questions can be posed directly to the group (‘how many of you…’) or asked in general for the company. Y is the
answers if the majority of people answer positively, or if the group agrees this is the case for the company (with
possible exceptions). Otherwise N.
209
Given the scope and nature of this evaluation (with a focus on the PSOM/PSI programme managed by RVO and
funded by the Ministry of Foreign Affairs, rather than being focused on CSR of participating companies), it is unavoidable that these employees will be selected by the project management, interviews are conducted on-site and
the evaluation does not allow for third party involvement (e.g. trade unions).
204

Evaluation PSOM/PSI | 153

Y N Comments210
Work
People in this company
earn at least minimum wage211
What is this wage (more or less)?
Get paid on a regular basis
Daily? Weekly? Monthly?
Always get paid for
overtime
Are always over 15
years old
How old is the youngest employee? What does he/she do?
Are allowed to terminate employment
at will or after serving a notice
Have the right to
join a trade union212
What is the union called?
Y N Comments
Holidays
People in this company
Get a weekly rest
period of at least 1
day a week
What days are off?
Y N Comments
Employment security
People in this company
Receive a written
statement of particulars at the start of
employment213
Does everybody have a contract? Signed?
Can work on tasks
of a permanent nature without fixed
term contract (i.e.
as contract workers)
210

Statements to be answered with Y/N, nuances (e.g. differences between levels, people) to be noted in comment
section. It is important to probe/double check answers (examples provided in the list, to be added depending on
type of project/sector/country). Observations of interviewers also to be noted in the comment section (in a distinct colour).
211
To be determined and assessed by the local expert (Note: there might be differences between legal minimum wage
and wat is considered a minimum living wage. In that case living wage is to be assessed.).
212
Obviously, this question is only to be asked in appropriate context (i.e. when legal in the country)
213
contract, appointment letter or other

154

|

2016

Are there more flex or fixed contracts? What tasks are flex contractors?
Get due notice before termination of
employment contracts
Receive severance
pay in case of sudden termination of
employment
Do you know of people who have received this severance pay?
Y N Comments
Sickness and employment injury benefits
People in this company
Receive paid sick
leave during the
first months214
Do you know of people who had this happen to them? In this company? Elsewhere?
Are adequately
compensated in
case of an accident/work injury or
occupational disease
Do you know of people who had this happen to them? In this company? Elsewhere?
Y N Comments
Maternity
Women in this company
Can adapt their
work (e.g. hazardous work, night
shifts) during pregnancy
Are protected from
dismissal during
pregnancy
Have maternity
leave (at least partly
paid)215
Do you know of people who had this happen to them? How long is maternity leave?
Y N Comments
Health and safety
People in this company
Have a safe and
healthy workplace
Use protective
equipment, including clothing, free of
214
215

Note % of wage and for how long?
Note % of wage and for how long?

Evaluation PSOM/PSI | 155

cost to them
Examples of safety measures (and unsafe situations)?
Y N Comments
Fair treatment
People in this company
Receive equal pay
for similar/equal
work without discrimination
Are treated equally in employment opportunities (appointment, promotion, training…) without
discrimination of:
Sex/gender
Religion
Political opinion
Place of birth
Trade union membership
Other?
Do you have examples of ways in which women are supported in their work?
Do you have examples of discrimination that was halted/punished?
Training
What types of training have the employees received in the past 5 years?216
Was this useful and Y N Comments
why (not)?
Health and Safety
…
…
…
Change over time
In the past few years, what have been the main changes in the areas discussed (positive/negative)?217 How did this come about?

Overall comments of interviewers on process218

216

Or during the PSOM/PSI project period.
If possible, to be linked to the PSOM/PSI project period.
218
Attitude of the group, possible differences between answers of men and women, credibility of the answers,…
217

156

|

2016

Annex VI: Overview PSOM/PSI contribution

ID

Operational
(Y/N/0 =
too early)

Additionality
(Y/N)

Contribution
PSOM/PSI (one
of Many factors/Crucial
factor/0 = too
early)

B1
B2
B3
B4
B5
B6
B7
B8
BH1
BH2
BH3
BH4
BH5
BH6
BH7
BH8
E1
E2
E3
E4
E5
E6
E7
E8
P1
P2
P3
P4
P5
P6
P7
P8
P9
SL1
SL2
SL3
SL4
SL5
SL6
SL7
SL8
U1
U2

Y
Y
Y
N
N
Y
Y
Y
Y
0
Y
Y
N
Y
Y
N
0
Y
N
Y
0
Y
Y
Y
Y
Y
Y
0
Y
Y
N
Y
Y
Y
Y
Y
N
N
Y
N
Y
N
Y

N
Y
N
Y
Y
Y
Y
Y
N
Y
Y
Y
N
Y
N
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
N
N
Y
Y
N
Y
N
N
N
N
Y
Y
N
Y
N
N
Y
N

M
C
M
M
M
C
C
M
C
0
C
C
C
C
C
M
C
C
M
C
M
C
C
C
M
M
M
M
C
M
M
M
M
M
M
C
M
M
M
M
C
M
M

Evidence
of sector
impact
(Potential
Yes / No /
Yes)
PY
N
N
N
PY
N
Y
Y
Y
Y
Y
Y
N
Y
PY
Y
Y
Y
N
Y
Y
Y
Y
Y
PY
N
N
PY
PY
Y
Y
N
N
Y
Y
Y
Y
Y
PY
PY
Y
Y
Y

Evidence
of market
impact
(Potential
Yes / No /
Yes)
N
N
N
N
PY
N
Y
PY
N
N
PY
N
N
N
PY
Y
N
Y
Y
Y
Y
N
N
N
PY
PY
N
N
N
N
N
N
PY
Y
N
N
N
PY
PY
N
N
Y
N

Evaluation PSOM/PSI | 157

U3
U4
U5
U6
U7
U8

158

0
Y
Y
Y
Y
N

|

2016

N
Y
N
Y
Y
Y

0
M
M
M
C
M

Y
PY
PY
Y
PY
PY

N
N
PY
N
N
PY

Annex VII: Quantitative methodology and analysis
Monitoring data
Data
The evaluation team used the inception phase to make an inventory of available data, in order
to assess the feasibility of the methodology described in the proposal.
Descriptive data concerning project characteristics (sector, country, tender) are available for all
PSOM/PSI projects. The data covers employment, female employment, sales, training, follow-up
investments and number of outgrowers for all projects. Additional data on employment level
and subcontractors was collected for PSI projects.
RVO collects these impact indicators throughout the whole project. But these monitoring data
are overwritten every time RVO receives an update about these impact indicators. Therefore only the target proposed and the target realized, at the end of the project or the last time results
are reported by a project, are registered in the BAS monitoring system.
Furthermore, APE has collected information about the partnership and financial characteristics,
like balance, turnover and number of employees at the start of the project, on all selected and a
sample of rejected projects to add to the analysis. 250 projects were randomly selected from the
946 proposals rejected under PSI. Data was available for 186 of these proposals (no financial
characteristics were available in BAS for the other 64). The data from these 186 will be included
in the analysis of the selection process and ex ante additionality.
Project database
 RVO provided the evaluation team with a project database with all readily available data.
This database contains information on all 2912 PSI and PSOM projects, both rejected and
granted. Variables included are:
 Programme: PSOM, PSI regular, PSI Plus or PSI Arab
 Dossier status: rejected, ongoing, stopped or finalized
 Year of tender
 Country and region
 Fragile state
 Dutch applicant: yes or no
 Sector
 Start and end date of the project
All of these data can be used to make a quantitative description of the programmes. Additionally, impact variables and grant characteristics are also available for granted projects in this project database. Monitoring data is missing for the projects before 2003.
Additional information BAS monitoring system
The evaluation team has collected additional data for all 616 granted PSI projects from the BAS
monitoring system and project files in order to be able to do more complex analyses (as described in Sections 3.3, 5.3 and 5.4). The additional variables comprise data about the partnership and financial characteristics at the start of the project. So these data (except approved

Evaluation PSOM/PSI | 159

hardware budget) cover intentions, not realisations. The variables are shown in appendix III. For
the PSOM projects the data necessary for the more complex analyses were not available. The
evaluation team has also collected these additional data for a random sample of 186 rejected PSI
proposals. Further, the reason why RVO rejected the proposal is collected.
Ranking forms
Additionally, RVO provided the evaluation team with an overview of the ranking forms for all
projects across 2008-2014.

Multivariate analyses
Data preparation
Some data cleaning was done before the quantitative analyses were carried out. Outliers and
the number of missing values were identified by means of descriptive statistics and bar graphs,
like Table A.5.1. Furthermore, we used the information gathered during from interviews and
desk research. This approach led to the following decision rules for outliers:
 We see projects where employment proposed or employment realized (Figure A.5.1) exceeds 1,000 jobs as outliers.
Figure A.5.1: Employment realized (n = 555 projects)



160

Projects with a training component, both proposed and realized, higher than 10.000 people trained are, based on the five highest values in Table A.5.1regarded as outliers.

|

2016

Table A.5.1: Five highest values of training realized
Highest

Number of people trained

1

2,000,000

2

30,150

3

7,946

4

6,615

5

6,202










We consider projects where sales or investments are larger than € 25,000,000, proposed
and realized, as unrealistic because PSOM/PSI finances pilot projects.
Financial ratios and internal rates of returns with an absolute value higher than 100, or 1
depending on the measurement scale, are regarded as outliers. RVO determines the following financial ratios for each applicant and local partner at the moment of application: (i)
financing ratio, (ii) equity ratio, (iii) liquidity ratio and (iv) solvency ratio219.
Percentages, like the division of shares and the percentage of budget reserved for hardware and technical assistance, cannot exceed 100%.
We regard applicants with more than 10,000 employees, and local partners with more
than 1,000 employees as outliers. RVO favours SMEs during PSOM/PSI. Therefore, companies with such large number of employees are judged exceptional.
PSOM/PSI projects have a maximum project budget of € 1,500,000, RVO accounts for 50%
of this budget. Therefore applicant and local partner cannot invest more than € 750,000.
We regard applicants whose equity, balance, turnover, net profit or cash flow exceed €
100,000,000 as outliers. We use different boundaries for the local partner’s financial indicators. Balance or turnover exceeding € 50,000,000, equity exceeding € 25,000,000, net
profit or cash flow exceeding € 5,000,000 are regarded as exceptional based on the inspection of bar graphs.

Selection of projects: determinants of awards; common ground of awards and denials
We constructed financial profiles, where available, for all 616 selected PSI projects and a sample
of 249 rejected PSI projects. These profiles are based on company information from both partners and proposed project characteristics reported in the application form (Table A.5.2).
Table A.5.2 shows means for all registered financial characteristics about the applicant, local
partner and some proposed characteristics of the project for both selected and rejected projects. By means of a paired t-test for equal sample means, we analysed whether selected and rejected projects have significantly different means for these characteristics. In order to estimate
the t-value correctly, it is important to know whether the variances between the group of selected and rejected projects are equal. We used the Folded F test to determine this. If the Folded
F test indicated unequal variances at a 5% significance level we used the paired t-test with unpooled variance to take this into account. Otherwise we used the paired t-test with pooled variance. We also accounted for the fact that the number of selected projects for which the infor-

219

RVO defines these ratios – per partner - as follows:
 Financing ratio = (project duration in months/12) * (cash flow / own contribution)
 Equity ratio = equity / own contribution
 Liquidity ratio = short term loans / current assets
 Solvency ratio = total assets / equity

Evaluation PSOM/PSI | 161

mation from the application form is known is much larger than the number of rejected projects
with this information in calculating the (un)pooled variance.
Correlation matrix




Next we looked at the correlation matrix to determine whether the various financial
characteristics are associated with being selected. Being selected is positively correlated
with the applicant's net profit, cash flow, financing ratio and solvency ratio.
There is also a high (above 0.8) correlation between the financial ratios and some of the
firm characteristics: net profits, cash flow, balance, equity and turnover. This is not surprising because the financial ratios are based on these same characteristics..
The firm characteristics are also highly correlated with each other. For instance, the applicant's cash flow and net profit have a correlation coefficient of 0.88. This is also not
surprising because a successful company will do well on all financial components.

Logistic regressions
We analyse different models to determine the probability of being selected.220 A dummy variable which equals one if a project is selected is the dependent variable in all these models. The
reference project in these analyses is an agricultural project in Africa selected in 2009. Results
are in Table A.5.3.
Model (1)
This model is estimated based on the information of 616 selected and 249 rejected PSI projects.
Information on all financing ratios is available for 494 of these projects - 414 selected and 80 rejected. The financial ratios - financing, solvency, liquidity and equity ratio - for applicant and local
partner are used as independent variables. A dummy variable to indicate whether a third partner is present, the project's proposed internal rate of return, the proposed division of shares between applicant and local partner, and the number of employees of both partners are added to
the regression. Also dummy variables for region, sector, programme, tender year, and lower or
upper income countries are added as dependent variables.
The applicant’s financing ratio, which is the ratio between an applicant’s total cash flow during a
project and the own contribution he/she invests into this project, is the only significant financial
variable. It has a positive effect on the selection probability. So, a large total cash flow compared
to the contribution to the project, resulting in a high financing ratio, increases the selection
probability.
Model (2)
Model (2) is the same as Model (1) except that the ratios are replaced by the firm characteristics
underlying the ratios. Each characteristic - equity, balance, net profit, turnover and cash flow - is
entered separately because of the high correlation between these variables. No significant results are found in these regressions. Therefore, the results are not reported in Table A.5.3.
Model (3)
This model is estimated based on the data of all 2,912 PSOM/PSI projects. 1,107 of these projects were awarded. We only use dummy variables for region, sector and tender as independent
variables in this regression. This allows us to include all 2,912 projects.
220

162

Unfortunately, the ranking scores given by RVO could not be added as independent variables because
there are hardly any rejected projects with a completed ranking form.

|

2016

Almost all variables are significant in this model. Projects in services and industry have a higher
selection probability than the reference project. The selection probability is also higher for projects in Asia or Latin America. The selection probability is lower for projects situated in Eastern
and Central Europe or in the MENA region.
Kernel densities
We inspect the kernel densities for selected projects and rejected projects to investigate the discriminatory power of the selection process. The stronger this power the smaller the common
ground of rejected and selected proposals. We calculate kernel densities using Model (3) because this model has the most observations and the most significant variables.
Figure A.5.2 shows the kernel densities of the selected (pink) and rejected (blue) projects. We
use a t-test for equality of means to determine whether selected and rejected projects are socalled balanced – before and after matching – which is an indicator for discriminatory power. If
selected and rejected projects are balanced on all variables the difference in means between
both groups is insignificant and therefore selected and rejected projects are comparable.
Table A.5.4 shows the results of this t-test. We see that the mean for multiple variables, like
Asia, industry or services, differ significantly between selected and rejected projects before they
are matched. These significant differences become insignificant after matching. So, a valid control group of rejected projects could have been constructed. This could have been a basis to assess a net effect (additionality) of PSOM/PSI, were it not that we lack comparable data on outcomes for rejected projects.
Figure A.5.2 Kernel density estimates selected projects (pink, N = 1,107 projects) and rejected
projects (blue, N = 1,805 projects)

Evaluation PSOM/PSI | 163

Table A.5.2: Profiles selected and rejected PSI projects (n = 616 selected and 249 rejected projects)
Selected
Variable

N

Hardware (%)
Technical assistance (%)
Internal rate of return
Division of shares applicant (%)
Division of shares local partner (%)

517
517
516
498
496

Own contribution
Equity
Balance
Turnover
Net profit
Cash flow
Number of employees
Equity ratio
Financing ratio
Solvency ratio
Liquidity ratio

510
515
515
510
505
513
492
515
514
515
512

Own contribution
Equity
Balance
Turnover
Net profit
Cash flow
Number of employees
Equity ratio
Financing ratio
Solvency ratio

508
499
499
496
488
496
484
490
484
498

221

Mean

Std. Dev.
N
Mean
Project characteristics (intentions)
79.70
10.19
156
77.65
20.22
10.12
156
22.42
15.98
8.74
155
18.16
46.39
17.04
124
44.50
49.15
17.40
122
50.70
Characteristics applicant (before application)
€ 295,084
138,422
126
€ 268,129
€ 2,503,967
4,351,672
125
€ 1,912,533
€ 6,012,066
9,368,713
125
€ 4,770,984
€ 8,748,207
14,834,804
123
€ 6,210,224
€ 413,235
915,375
125
€ 148,566
€ 614,258
1,157,762
125
€ 237,650
79
264.96
113
60
8.47
14.49
122
5.74
4.92
9.11
123
0.94
0.44
0.31
124
0.44
4.26
10.14
121
4.97
Characteristics local partner (before application)
€ 287,512
150,404
124
€ 304,260
€ 1,090,939
2,016,167
115
€ 640,245
€ 2,388,800
4,124,753
113
€ 1,978,619
€ 2,414,578
4,860,329
109
€ 1,983,573
€ 170,694
371,103
110
€ 129,344
€ 245,662
424,930
111
€ 166,313
72
132.62
110
61
3.90
7.48
111
2.95
1.99
3.89
110
1.31
0.53
0.35
112
0.45

Rejected
Std. dev

Paired t-test
221
|T-value|

16.25
16.22
13.76
18.34
18.62

1.49
1.60
1.86*
1.09
0.87

145,043
4,090,561
9,659,023
12,131,587
1,105,988
1,017,368
192.65
10.58
13.70
0.32
11.07

1.94*
1.38
1.32
1.99**
2.47**
3.34**
0.88
2.36**
3.06**
0.11
0.68

165,315
1,322,676
4,026,115
5,698,050
461,809
514,085
121.21
6.32
3.24
0.43

1.09
2.95**
0.96
0.73
0.88
1.51
0.81
1.38
1.90*
1.81*

* means significance at a 10% confidence level, ** means significance at a 5% confidence level. We take unequal sample size into account in the t-test for equal sample means.

164

|

2016

Liquidity ratio

477

5.90

12.12

108

4.79

6.58

1.31

Table A.5.3: Results logistic regression selection
Variables
Intercept
Internal rate of return
Division of shares local partner
Division of shares for applicant
Equity ratio local partner
Equity ratio applicant
Financing ratio local applicant
Financing ratio applicant
Solvency ratio local partner
Solvency ratio applicant
Liquidity ratio local partner
Liquidity ratio applicant
Number of employees local partner
Number of employees applicant
Dummy variable for:
Asia
Latin America
MENA region
Eastern and central Europe
Industry
Services
Plus
Arab
Third partner

222

Coefficient
-1.66
-0.01
2.58*
2.47*
0.02
0.00
0.05
0.05**
0.27
0.16
0.01
0.01
-0.00
0.00
-0.45
0.49
1.25**
-0.11
1.73**
1.04**
-0.14
-0.80
0.79**

4

(1)
P-value
0.261
0.379
0.074
0.095
0.709
0.855
0.353
0.004
0.493
0.741
0.504
0.460
0.809
0.700
0.324
0.540
0.047
0.854
0.000
0.013
0.836
0.391
0.030

Coefficient
-0.48**

0.19*
0.28**
-0.14
-0.48**
1.13**
1.00**

222

(3)
P-value
0.002

0.086
0.029
0.337
0.002
0.000
0.000

* means significance at a 10% confidence level, ** means significance at a 5% confidence level.

Evaluation PSOM/PSI | 165

Lower middle income country
Upper middle income country
Fragile state
Tender 1999
Tender 2000
Tender 2001
Tender 2002
Tender 2003
Tender 2004
Tender 2005
Tender 2006
Tender 2007
Tender 2008
Tender 2010
Tender 2011
Tender 2012
Tender 2013
Tender 2014
2
R
N

-0.12
-0.48
-0.82

0.89
-0.26
0.11
-0.13
-0.49

0.794
0.385
0.189
-1.14**
-1.29**
-0.94**
-0.53*
-1.05**
-0.71**
0.14
0.13
-0.19
0.20
0.08
-0.45*
0.03
-0.53**
-1.08**

0.115
0.594
0.834
0.787
0.338
0.163
494

0.000
0.000
0.001
0.086
0.000
0.000
0.549
0.554
0.347
0.438
0.683
0.019
0.860
0.004
0.000
0.086
2912

Table A.5.4: T-test for equality of means between selected and rejected projects before and after matching
Variable
Asia
Latin America
MENA region
Eastern and central Europe

223

Unmatched, Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched

Mean selected projects
0.25
0.25
0.15
0.15
0.08
0.08
0.08
0.08

* means significance at a 10% confidence level, ** means significance at a 5% confidence level.

166

|

2016

Mean rejected projects
0.20
0.24
0.11
0.14
0.11
0.09
0.10
0.08

223

P-value
0.001**
0.680
0.001**
0.520
0.033**
0.788
0.232
0.996

Industry
Services
Tender 1999
Tender 2000
Tender 2001
Tender 2002
Tender 2003
Tender 2004
Tender 2005
Tender 2006
Tender 2007
Tender 2008
Tender 2010
Tender 2011
Tender 2012
Tender 2013
Tender 2014

Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched

0.32
0.32
0.15
0.15
0.03
0.03
0.02
0.02
0.02
0.02
0.02
0.02
0.03
0.03
0.07
0.07
0.06
0.06
0.07
0.07
0.08
0.08
0.05
0.05
0.09
0.09
0.09
0.09
0.11
0.11
0.11
0.11
0.06
0.06

0.15
0.33
0.09
0.14
0.04
0.03
0.06
0.02
0.04
0.02
0.02
0.02
0.05
0.03
0.09
0.08
0.03
0.06
0.04
0.08
0.06
0.08
0.02
0.04
0.06
0.09
0.11
0.09
0.07
0.11
0.12
0.12
0.11
0.06

0.000**
0.729
0.000**
0.658
0.007**
0.999
0.000**
0.881
0.006**
0.766
0.260
0.982
0.002**
0.740
0.029**
0.580
0.001**
0.884
0.000**
0.812
0.092*
0.977
0.003**
0.320
0.000**
0.914
0.366
0.706
0.000**
0.896
0.523
0.834
0.000**
0.959

Evaluation PSOM/PSI | 167

Predictors of success
Profiles
As a starter, we constructed financial profiles for all finalized and stopped PSI projects (Table
A.5.6). Rejected and ongoing projects are left out as they have not (yet) failed or succeeded. The
same method, as for the profiles of selected and rejected PSI projects, is used.
The distinction between successfully finalized projects and stopped projects is difficult to make
using ex ante financial indicators for both partners as these values might only be valid at the
start of the project and be unrelated to further success or failure. However this is the only available information in the BAS system about applicant and local partner.
Only the applicant’s balance and equity ratio are lower for stopped projects than for finalized
projects.
Logistic regression
We used logistic regression to determine the effect of the partnership's financial characteristics
and the project's proposal scores, given by RVO before the APSI ranking, on project's success.
We only looked at the scores from the 2010 and 2011 ranking forms (N = 113 PSI projects) because there is a relatively large number of finalized and stopped projects in both tenders, compared to the 2012 till 2014 tenders, where most projects are still ongoing (Table A.5.5). Further,
the collected ranking scores are more comparable and elaborated for the 2010 and 2011 tenders, compared to the tenders in 2008 and 2009. Moreover, the scheme by which the ranking
scores were weighted is available for these two tenders. This is not the case for the 2008 and
2009 tenders.
Table A.5.5: Number of finalized and stopped projects per tender
Tender
2008
2009
2010
2011
2012
2013
2014

Finalized
25
47
31
19
3
0
0

Stopped
21
34
36
30
16
20
3

Standardized ranking scores were required to make the rankings of 2010 and 2011 comparable
because these tenders differed in the type and number of items scored and in the way the ranking scores were weighted. For instance, RVO gave scores for substantial impact on production
chain, sector or country for the first time during the tender of 2011224. Scores are standardized
by dividing the obtained score by the maximum score possible, based on the items that are recorded in both 2010 and 2011.
We analysed different success indicators: (1) the project status given by RVO, (2) the number of
proposed targets that were achieved, (3) percentage of originally committed subsidy paid and
224

168

Source: RVO ranking forms tender I & II in 2010 and tender I & II in 2011.

|

2016

(4) the accomplishment of each of the proposed targets separately - employment, training, outgrowers, subcontractors, females, sales and follow-up investment.
Inspection of the correlation matrix shows that the accomplishment of each of the proposed
targets225 is correlated with scores on specific components, like the score for innovativeness in
marketing or the score on proposed number of high or medium level trainings given during the
project. Remarkably, a project's overall score is not significantly correlated with any of the success indicators analysed.
Based on the correlation matrix and on the number of observations, we chose to use the following success indicators in the logistic regression: (i) a dummy variable which equals one if a project finalized according to the definition of RVO and (ii) a dummy variable which equals one if
four out of the seven targets - employment, knowledge transfer, outgrowers, subcontractors,
females, sales and follow-up investments - are achieved.
The following models are estimated. The reference project, described by the intercept, is always
an agricultural project selected in 2009.
Model (1A)
The dependent variable is a dummy variable which equals one if four out of seven targets are
achieved during the project. This dummy is regressed on the partnership's financial ratios from
the application form, the number of employees of both partners and the scores per component
given by RVO at the ranking form. Dummies for region, sector and tender are also included.
The number of observations is small in this model. For only 71 PSI projects we have scores for
each item of the ranking form, the financial ratios of the partnership and all seven targets. 60
out of 71 projects did not achieve four out of seven targets, 11 did achieve this. The model cannot be estimated properly due to validity issues226 as a result of this small number of observations.
Model (1B)
This model is the same as model (1A) except that the total score from the ranking form is used
instead of separate item scores. The validity issues are solved in this way but still only the estimate of the intercept is significant.
Model (1C)
This model is a simplification of model (1B). For the local partner, the number of employees and
the financial ratio are missing relatively often. Therefore, these two variables are left out of the

225

226

Achieved is defined as target proposed ≤ target realized. The targets are employment, training, sales,
females, outgrowers, subcontractors and follow-up investments.
The maximum likelihood estimate may not exist due to (quasi-)complete separation. This means that
there is an independent variable that (almost) perfectly predicts success. For example, if the number
of employees is lower than ten for all stopped projects and higher than ten for all finalized projects,
then number of employees predicts success perfectly and there is complete separation. Mathematically, this means that the coefficient for number of employees cannot be estimated. The higher the
coefficient for number of employees, the higher the likelihood will be. Therefore, the coefficient of
the number of employees tends towards infinity in case of complete separation. So, the coefficient estimate for number of employees in this example is incorrect but the coefficient estimates for other independent variables are valid. Removing the number of employees would lead to biased estimates
for these other independent variables.

Evaluation PSOM/PSI | 169

model to increase the number of observations. Unfortunately, this does not result in any significant estimates.
Model (2A)
The number of observations is low because we focus on the ranking form scores from tenders
2010 and 2011. In order to increase the number of observations and the reliability of our estimates, we also estimated a model based on the partnership's financial ratios at the moment of
application. Dummies are included for region, sector and tender. The dependent variable is a
dummy variable which equals one if the project is successful according to the definition of
RVO227 (as assessing targets met would again reduce the number of observations).
The number of observations is 158. Just as in Model (1A), this model results in validity warnings.
Even, if only one financial ratio and dummy variables are used as independent variables (N =
225), the issue is not resolved.
Model (2B)
Model (2B) has the same independent variables as model (2A) but has the dummy variable that
equals one if four targets are achieved as dependent variable. Here, the intercept and the dummy for Asia are significant.
So, in conclusion, the logistic regression does not identify any variables that influence success
significantly - except for model (2B) where the dummy variable for Asia is significant. Whether a
project will be a success, is difficult to predict by the ranking scores.
Discriminant analysis
Discriminant analysis is a technique which is often used in marketing research for profiling – how
do groups differ with respect to independent variables? -, differentiation – are differences
across groups significant?- and categorization – can we predict group membership based on certain independent variables? -. This technique is a special case of multiple regression when there
are only two groups to predict, as is the case here. We used discriminant analysis for categorization in order to construct a continuous indicator of success.
We focus on goal achievement with regard to employment, training and sales because information on those three key indicators is available for most projects. These three indicators are
monitored through the whole PSOM and PSI programme while others were added later. 372 out
of 721 finalized and stopped PSOM/PSI projects have information on the proposal and realization of all three indicators.
The project status as given by RVO is the dependent variable in this discriminant analysis. This
dummy variable equals one when a project has the status ‘finalized’ and zero when a project is
stopped. We analyzed two different discriminant models, based on all 372 PSOM/PSI projects
with information on employment, training and sales:
Model (1)

227

170

RVO considers a project unsuccessful if less than 20% of the financial support is disbursed. If more
than 60% the subsidy is disbursed, or if the majority of the results have been accomplished, the project is deemed successful . For the other projects success is determined on a case-by-case basis by
RVO.

|

2016

Firstly, we analyzed a model where the independent variables are three dummies that equal one
if the target on, respectively, employment, training or sales was achieved228.
This model does not result in a fully continuous success indicator. There are only eight possible
combinations based on three dummy variables.
Model (2)
Secondly, we analyzed a model where we used the difference between target realized and target proposed divided by target proposed as independent variables. This model results in a continuous success indicator. Therefore, we analyzed this one further in relation to the ranking
scores given by RVO.
Beforehand, one would suspect that the level of success would be higher if the total ranking
score would be higher (i.e. ranking is a predictor of success). Figure A.5.3 does not show this expected pattern. Here, the level of success is determined by means of discriminant analysis. The
higher the discriminant score is, the more successful a project is (i.e. larger positive difference
between target proposed and target achieved). A project is classified as successful when its discriminant score is higher than 0229. Figure A.5.3 shows that there are some projects that have a
standardized total ranking score of 0,6 or higher but are subsequently classified as unsuccessful.
The majority of the projects that are successful have a standardized total score between 0,3 and
0,5, which is not the highest score given.
Figure A.5.3: Scatterplot discriminant score vs. total score ranking (n = 59 projects from tender
2010 and 2011)
Level of success (discriminant score)

8
7
6
5
4
3
2
1
0
-1 0

-2

228
229

0,2

0,4

0,6

0,8

Total ranking score (standardized)

Achieved is defined as target proposed ≤ target realized.
This threshold is determined by means of the Mahalanobis midpoint with unequal group sizes.

Evaluation PSOM/PSI | 171

Table A.5.6: Profiles finalized and stopped PSI projects (n = 100 finalized and 139 stopped projects)
Finalized

Stopped

Variable

N

Mean

Std. Dev.

N

Mean

Std. Dev

|T-value|

Originally committed subsidy

82

€ 658,868

195,043

115

€ 649,197

179,988

0.36

Project characteristics (intentions)
Hardware (%)

82

76.68

13.15

114

78.16

10.33

0.85

Technical assistance (%)

82

22.96

12.82

114

21.82

10.35

0.66

Internal rate of return

82

16.63

10.27

114

15.93

9.42

0.50

Division of shares applicant (%)

76

45.85

14.77

107

48.8

17.93

1.18

Division of shares local partner (%)

76

49.43

15.79

107

46.63

17.23

1.12

Characteristics applicant (before application)
Own contribution

81

€ 310,231

154,220

111

€ 305,389

144,931

0.22

Equity

82

€ 3,633,957

6,144,134

113

€ 2,174,861

3,553,756

1.93*

Balance

82

€ 8,720,452

11,743,159

113

€ 5,504,171

9,330,563

2.05**

Turnover

81

€ 14,285,976

19,331,530

112

€ 9,323,010

13,861,873

1.97*

Net profit

81

€ 595,297

1,173,347

109

€ 433,001

978,243

1.04

Cash flow

81

€ 966,153

1,383,750

112

€ 668,429

1,356,423

1.49

Number of employees

79

46

74.85

110

80

305.69

1.10

Equity ratio

82

11.69

18.62

113

6.77

11.23

2.13**

Financing ratio

81

6.81

10.07

113

4.92

10.71

1.25

Solvency ratio

82

0.41

0.32

113

0.43

0.32

0.60

Liquidity ratio

81

3.79

10.24

113

3.96

9.10

0.13

Characteristics local partner (before application)
Own contribution

81

€ 271,126

157,482

111

€ 276,967

146,850

0.26

Equity

82

€ 1,172,531

1,975,120

109

€ 994,125

2,573,382

0.54

Balance

82

€ 2,220,651

3,696,418

109

€ 2,113,747

3,837,577

0.19

Turnover

82

€ 1,744,580

3,333,261

110

€ 2,229,581

4,211,781

0.89

Net profit

82

€ 137,724

346,035

106

€ 136,436

291,659

0.03

Cash flow

82

€ 187,214

369,660

109

€ 204,760

311,365

0.36

Number of employees

78

65

132

105

69

136

0.17

230

* means significance at a 10% confidence level, ** means significance at a 5% confidence level. We take unequal sample size into account in the t-test for equal sample means.

172

|

2016

230

Equity ratio

80

4.32

8.93

108

3.74

8.28

0.46

Financing ratio

79

1.54

4.15

107

1.68

2.81

0.26

Solvency ratio

81

0.54

0.34

109

0.55

0.33

0.10

Liquidity ratio

78

5.17

8.68

105

4.78

9.51

0.29

Evaluation PSOM/PSI | 173

Contribution of PSOM/PSI subsidies to outcomes
We focus in this section on the indicators employment realized, training realized and sales realized for the same reasons as explained above. These three indicators are the dependent variables in the models described in the subsequent paragraphs. We took the logarithm of sales realized to normalize the error terms.
We estimated the following three models for each indicator. An agricultural project in Africa selected in 2009 is, again, the reference project, its effect estimated by the intercept.
Model (1)
Each indicator is regressed on the logarithm of subsidy for hardware, the logarithm of subsidy
for technical assistance and dummy variables for sector, region and tender 2010 till 2012231.
The amount of subsidy for hardware and technical assistance is calculated by separating the
committed subsidy according to the ratio of hardware and technical assistance proposed in the
application form. We use the committed subsidy because this is the amount of money the partnership actually received from RVO. The logarithm is taken after this separation is made.
We estimated this model for finalized PSI projects because the ratio hardware and technical assistance is only known for PSI projects. There are only 100 finalized PSI projects.
Model (2)
No distinction between subsidy reserved for hardware or technical assistance is made to increase the number of observations. PSOM projects can now be included because the committed
subsidy is known for these projects though the ratio hardware and technical assistance is not.
Each indicator is regressed on the logarithm of committed subsidy and dummy variables for sector, region and tender 2003 until 2012 - except 2009.
Model (3)
Model (2) and Model (3) are almost identical, only the selection probability is added in Model
(3). In this way, the selection bias is removed from the contribution effect (i.e. to distinguish between the impact of PSOM/PSI on employment and other variables and vice versa, as those variables have also been a reason for PSOM/PSI support). The selection probability is constructed
for PSOM and PSI projects by regressing a selection dummy on dummies for sector, region and
tender. Financial indicators are not included in this regression, like the selection models in Table
A.5.3 because the number of observations would then be even smaller than in Model (1).
The results are shown in Table A.5.7 through Table A.5.9. The contribution of the committed
PSOM/PSI subsidy is only significant on employment realized232 and not on knowledge transfer

231

174

No dummies for 2013 and 2014 are included because there are no finalized projects from these tenders yet.

|

2016

or sales realized. The effect of the committed PSOM/PSI subsidy on employment is robust
against the inclusion of the selection probability. If PSOM/PSI committed subsidy is raised by one
percent, the number of jobs created at the end of the project is increased by 0.39 percent, keeping all other things equal. A higher subsidy also translates into more sales, albeit that the coefficient estimate for sales is somewhat less reliable: 10% more subsidy means 5.3% more sales. The
contribution of PSOM/PSI to training is insignificant.

232

The effect is not significant if a distinction is made between subsidy for hardware and technical assistance. However, the number of observations is much lower in this case (93 versus 332 if the committed subsidy is regarded in total).

Evaluation PSOM/PSI | 175

Table A.5.7: Contribution of PSOM/PSI subsidy to employment realized
Employment realized
(1)
233

Intercept
Log:
Hardware subsidy
Technical assistance
subsidy
Committed subsidy
Dummy variable for:
Asia
Latin America
MENA region
Eastern and central
Europe
Industry
Services
Tender 2003
Tender 2004
Tender 2005
Tender 2006
Tender 2007
Tender 2008
Tender 2010
Tender 2011
Tender 2012
Selection probability
2
R
N

233

Coefficient
141.35

30.59
-39.09

P-value
0.763

(2)
15
Coefficient
P-value
-414.56* 0.069

(3)
15
Coefficient
P-value
1,170.32** 0.007

0.327
0.232
38.36**

0.024

39.31**

0.020

0.084
0.025
0.145
0.151

15.83
27.67
-46.08
-52.65

0.681
0.529
0.280
0.239

-25.56*
-17.65
-38.74
-59.23**

0.062
0.314
0.127
0.015

-23.57*
-126.85**
-36.84
154.49

-29.72
-33.34

0.362
0.478

4.46
42.12
12.00

0.887
0.270
0.901

-24.39*
-55.10**
84.63**
24.03
8.10
10.43
-1.77
2.27
2.00
48.36*
29.41

0.066
0.004
0.002
0.267
0.713
0.617
0.934
0.933
0.935
0.095
0.690

-537.95** 0.033
-498.27** 0.023
507.14** 0.016
313.65** 0.029
9.57 0.663
10.69 0.606
-0.34 0.987
4.13 0.878
2.76 0.910
253.67** 0.016
29.36 0.689
1,871.38** 0.042
0.131
332

0.116
93

0.120
332

* means significance at a 10% confidence level, ** means significance at a 5% confidence level.

Table A.5.8: Contribution of PSOM/PSI subsidy to knowledge transfer realized

Intercept
Log:
Hardware subsidy
Technical assistance
subsidy
Committed subsidy
Dummy variable for:
Asia
Latin America
MENA region
Eastern and central
Europe
Industry
Services

Knowledge transfer realized
(1)
(2)
(3)
16
16
CoeffiP-value Coefficient P-value Coefficient
P-value
234
cient
3,775.76
0.3659
1,509.13
0.502
-6,701.09
0.121
-138.64
-79.19

0.617
0.785

234

0.779

-37.80

0.820

-515.83
-363.84
-548.33
-604.35

0.133
0.351
0.149
0.136

-184.13
-237.12
-397.11
-380.55

0.174
0.181
0.115
0.118

-164.30
-1,426.58**
-375.06
1,936.47*

0.222
0.012
0.134
0.070

-341.38

0.241

0.011

-5,923.02**

0.019

-639.87

0.127

334.97**
489.28**
496.50*
-376.91*
496.28**
-262.28
-352.92*
-479.84*
-327.08
-183.08
-363.99

0.010

-5,322.08**

0.015

0.067
0.080
0.023

5,092.23**
2,768.98*
-482.89**

0.015
0.053
0.026

0.213
0.097
0.072
0.180
0.530
0.619

-257.64
-336.73
-460.46*
-318.57
2,042.88
-362.59
20,371**
0.127
327

0.219
0.111
0.082
0.189
0.050
0.618
0.027

Tender 2003
Tender 2004
Tender 2005
Tender 2006
Tender 2007
Tender 2008
Tender 2010
Tender 2011
Tender 2012
Selection probability
2
R
N

-46.99

-348.76
-181.36
-400.83
0.107
92

0.213
0.597
0.127

0.113
327

* means significance at a 10% confidence level, ** means significance at a 5% confidence level.

178 |

2016

Table A.5.9: Contribution of PSOM/PSI subsidy to sales realized (logarithm)
Sales realized (logarithm)
(1)
(2)
235
17
Coefficient
P-value Coefficient
P-value
13.96**
0.019
6.49
0.132

Intercept
Log:
Hardware subsidy
Technical assistance
subsidy
Committed subsidy
Dummy variable for:
Asia
Latin America
MENA region
Eastern and central
Europe
Industry
Services
Tender 2003
Tender 2004
Tender 2005
Tender 2006
Tender 2007
Tender 2008
Tender 2010
Tender 2011
Tender 2012
Selection probability
2
R
N

235

-0.30

0.447

0.29

0.487

(3)
17
Coefficient
P-value
-7.08
0.411

0.52*

0.103

0.53*

0.101

0.46
-0.27
-0.26

0.357
0.628
0.612

0.43*
-0.32
0.04

0.100
0.316
0.931

0.46*
-2.31**
0.07

0.077
0.044
0.867

-0.17

0.764

0.34

0.444

4.26*

0.054

0.46
-2.33**

0.262
0.000

-0.21
0.03
1.60

0.592
0.949
0.174

-0.10
-1.25**
-0.35
-0.71*
-1.24**
-0.32
-0.51
-0.66
-0.20
-0.11
0.80

0.684
0.000
0.510
0.069
0.004
0.397
0.180
0.167
0.641
0.823
0.512

-9.52*
-9.36**
7.30*
4.56
-1.20**
-0.32
-0.49
-0.63
-0.19
3.65*
0.79
34.19*
0.142
264

0.067
0.037
0.086
0.120
0.005
0.398
0.194
0.186
0.652
0.086
0.515
0.070

0.267
85

0.130
264

* means significance at a 10% confidence level, ** means significance at a 5% confidence level.

Evaluation PSOM/PSI| 179

Annex VIII: Survey questionnaire
Question

Question text

Answer options

1a

Please confirm your identify

Company name: [Automatisch invullen]

1b

What is your role within the

1: General Management

company?

2: Marketing / Sales

nr.
Name: [Automatisch invullen]

3: Operations
4: Research & Development
5: Human Resource Management / Personnel
6: Logistics
7: Other: …
1c

According to our records your

1: Yes

company submitted a proposal

2: No

for a PSOM or PSI subsidy,
which was rejected. Did your
company submit a succesful
proposal for the same project in
subsequent subsidy rounds? If
multiple porposals meet this criteria, pick the most recent one.
1d

What was the main reason the

1: Partnership

proposal was rejected?

2: business plan
3: financing
4: development impact
5: other, please specify….

1e

Did your company carry out the

1: yes

project without the subsidy (in

2: No

the same or different form)?

Question

Question text

Answer options

How was the project executed

1: According to the original plan

nr.
2a

(more than one answer possible)? 2: With a different local business partner
3: At a smaller scale
4: At a slower pace
5: Other, please specify:

180 |

2016

2b

How was the project financed

1: Fully self-financed by partnership

(more than one answer possible)? 2: Using another government/ngo subsidy/grant ot soft loan. If so, which?
3: Using a privately financed loan (e.g. by a
bank/investment fund).
4: Other, Please specify….
2c

Is the project still operational?

1: Yes
2: No

2d

Would you consider the project

1: Yes it was successful

successful?

2: It was successful, but less successfull than
planned
3: It was not successful

Question

Question text

Answer options

What is the main reason the pro-

Open

nr.
3a

ject was not executed?
Question

Question text

Answer options

nr.
4a

What was the size of your company 1: Micro: < 10 employees
at the time of the applciaiton for a

2: Small: < 50 employees

PSOM/PSI subsidy?

3: Medium: < 250 employees
4L: Large: anything bigger than Medium

4b

In which sector did the proposed

1: Agriculture (primary production of crops,

project operate?

lifestock etc.)
2: Industry (agroprocessing is categorised as
industry)
3: Services (also contains tourism, energy, logistics, transport)

4c

Did you have experience with setting up projects in developing
countries before application?

4d

Did you do business with your local
business partner before the application for PSOM/PSI?

4e

Have you started a project in a developing country, since the
PSOM/PSI application?

Evaluation PSOM/PSI| 181

Question nr. Question text
Answer options
5a
Was the PSOM/PSI application 1:

Yes

procedure transparent (was it 2: No,why not
clear why the application was unsuccessful)?

5b

Did you contract an external con- 1:
sultancy to write the PSOM/PSI 2: No
project application?

182 |

2016

Yes

Annex IX: Case studies (overview visited projects)
Country

Start Date

Bangladesh

Dutch applicant?
Yes

Adjusted end date
30-4-2011

Dossier
status
Finalized

Actual status
Still active

Sector

Commitment

In spin-off survey RVO?

1-12-2006

Original end
date
30-11-2008

industry

504,000

yes

Bangladesh

Yes

1-1-2006

1-9-2008

18-3-2009

Finalized

Onbekend

industry

495,000

yes

Bangladesh

Yes

1-7-2008

30-6-2010

30-6-2012

Finalized

Still active

services

432,684

no

Bangladesh

No

1-1-2012

31-12-2014

31-12-2015

Ongoing

Ongoing

agriculture

720,430

no

Bangladesh

Yes

1-9-2007

31-8-2009

30-6-2010

Finalized

Still active

industry

483,241

yes

Bangladesh

Yes

1-12-2006

30-11-2008

28-2-2010

Finalized

Onbekend

industry

504,000

yes

Bangladesh

Yes

1-7-2009

30-6-2011

31-7-2012

Finalized

Still active

industry

744,305

no

Bangladesh

Yes

15-6-2009

31-5-2011

30-4-2015

Ongoing

Ongoing

industry

500,650

no

BosniëHerzegovina
BosniëHerzegovina
BosniëHerzegovina
BosniëHerzegovina
BosniëHerzegovina
BosniëHerzegovina

Yes

1-7-2007

30-6-2009

31-12-2013

Finalized

Still active

agriculture

708,000

no

Yes

1-7-2009

31-12-2011

30-6-2015

Ongoing

Ongoing

services

414,750

no

Yes

1-7-2009

30-6-2011

30-9-2012

Finalized

Still active

industry

213,870

yes

Yes

1-1-2011

30-9-2012

31-12-2013

Finalized

Still active

services

202,790

no

Yes

1-1-2012

30-6-2014

16-12-2013

Finalized

Still active

industry

495,500

no

Yes

1-1-2013

30-6-2015

31-12-2015

Ongoing

Ongoing

industry

730,825

no

BosniëHerzegovina
BosniëHerzegovina
Egypte

Yes

1-7-2012

31-12-2014

31-3-2016

Ongoing

Ongoing

industry

550,054

no

Yes

1-7-2013

31-12-2015

30-6-2016

Ongoing

Ongoing

agriculture

675,750

no

Yes

1-8-2011

1-2-2014

1-5-2015

Finalized

Still active

services

543,419

no

Egypte

Yes

22-7-2004

30-6-2006

31-3-2007

Finalized

Still active

agriculture

643,814

yes

Egypte

Yes

1-1-2012

30-6-2014

31-5-2015

Finalized

Still active

industry

743,750

no

Egypte

Yes

1-1-2013

30-6-2015

31-12-2015

Ongoing

Ongoing

agriculture

732,341

no

Egypte

Yes

1-1-2010

31-12-2011

30-6-2016

Ongoing

Ongoing

agriculture

712,500

no

Egypte

Yes

1-7-2010

31-12-2012

30-6-2013

Finalized

Still active

agriculture

331,946

No

Egypte

Yes

1-1-2006

31-12-2007

31-3-2008

Finalized

Still active

agriculture

621,939

Yes

Egypte

Yes

1-10-2000

1-4-2002

31-3-2003

Finalized

Still active

agriculture

680,467

Yes

Sierra Leone

No

5-6-2009

4-6-2011

31-1-2013

Finalized

Still active

industry

933,750

No

Sierra Leone

No

1-1-2012

30-6-2014

31-5-2016

Ongoing

Ongoing

services

954,338

No

Sierra Leone

No

1-9-2009

31-8-2011

30-12-2014

Finalized

Still active

services

890,470

No

Sierra Leone

No

1-9-2009

31-8-2011

30-10-2014

Finalized

Still active

industry

777,600

No

Sierra Leone

Yes

1-9-2014

28-2-2017

0-1-1900

Ongoing

Ongoing

industry

899,373

No

Sierra Leone

Yes

1-6-2010

1-6-2013

30-9-2015

Finalized

Still active

agriculture

898,800

No

Sierra Leone

No

1-8-2010

31-7-2013

0-1-1900

Finalized

Still active

industry

899,761

No

Sierra Leone

Yes

1-1-2013

30-6-2015

30-6-2016

Ongoing

Ongoing

agriculture

900,000

No

Peru

Yes

1-7-2009

31-3-2012

31-12-2012

Finalized

Still active

agriculture

750,000

Yes

Peru

Yes

15-12-2005

30-6-2008

30-8-2008

Finalized

Still active

agriculture

620,760

Yes

Peru

Yes

1-7-2012

31-12-2014

31-12-2015

Ongoing

Ongoing

agriculture

721,500

No

Peru

Yes

1-7-2011

30-9-2013

31-7-2017

Ongoing

Ongoing

agriculture

736,860

No

184 |

2016

Peru

Yes

1-7-2010

31-12-2012

31-5-2015

Ongoing

Ongoing

agriculture

749,963

No

Peru

Yes

1-1-2007

31-12-2008

30-6-2010

Finalized

Still active

agriculture

558,828

Yes

Peru

Yes

1-3-2014

28-2-2017

0-1-1900

Ongoing

Ongoing

agriculture

425,669

No

Peru

Yes

1-7-2011

31-10-2013

0-1-1900

Finalized

Still active

agriculture

661,150

Yes

Peru

Yes

1-1-2011

30-6-2013

31-7-2014

Finalized

Stopped

agriculture

750,000

No

Peru

Yes

1-9-2005

31-8-2007

30-6-2010

Finalized

Still active

agriculture

467,284

Yes

Peru

Yes

1-4-2004

17-10-2005

30-11-2005

Finalized

Still active

agriculture

492,500

Yes

Oeganda

Yes

1-7-2009

31-12-2011

30-4-2015

Ongoing

Ongoing

agriculture

748,643

No

Oeganda

No

1-3-2014

31-8-2016

0-1-1900

Ongoing

Ongoing

services

748,000

No

Oeganda

No

1-9-2014

31-8-2017

0-1-1900

Ongoing

Ongoing

agriculture

647,475

No

Oeganda

Yes

1-1-2008

31-12-2009

31-5-2010

Finalized

Still active

agriculture

494,770

No

Oeganda

Yes

1-7-2008

30-6-2010

31-12-2010

Finalized

Still active

agriculture

483,000

No

Oeganda

No

1-9-2005

31-8-2007

31-10-2008

Finalized

Onbekend

agriculture

509,847

no

Oeganda

No

1-3-2014

31-8-2016

0-1-1900

Ongoing

Ongoing

industry

200,588

no

Oeganda

No

1-7-2009

30-6-2011

30-6-2015

Ongoing

Ongoing

industry

511,360

no

Evaluation PSOM/PSI| 185

Selection criteria

Country priorities

Relevance

Ex-ante additionality

Monitoring data
Quantitative analysis
Qualitative analysis

Deskresearch

Indicators

Survey rejected applicants

Question

Interviews other

Aspect

Interviews companies NL

Criteria

Interviews companies local

Annex X: Evaluation matrix

Comparison of project selection criteria with
observations of company
Do the selection criteria and process ensure Criteria related to successful PSI/PSOM prothe selection of projects in line with the ob- jects (for all PSOM/PSI projects with the rejectives of PSOM/PSI?
jected ones) like annual turnover, financial ratios. The found relations can be field tested
for the 48 case studies
Are the supported investments in line with Role of country priorities in project selection
the objectives and priorities of the PSOM/PSI (does it play a role in project selection? How is
country?
it operationalized?)
Ability for self-financing based on financial data like leverage and financial strength ratios
(equity/project size)
Lack of knowledge based on knowledge score
ranking
Extent to which PSOM/PSI addresses prior barWould the projects have been realized on riers to project (as perceived by participants:
the same scale, at the same pace and with e.g. private investment, management opposithe same impact without the financial con- tion, unsuccesful previous investments, laws
and regulations)
tribution of PSOM or PSI (additionality ex
Alternatives to PSI/PSOM available/used:
ante according to DCED)
- other investors approached (public, private,
commercial or not)
- indication of % investments other parties
Market displacement
- comparable projects?
- competitors?
Evaluation PSOM/PSI| 187

Efficiency

Effectiveness

188 |

Costs

How is the efficiency of the implementation
of the PSI programme by RVO affected by:a.
the PSI country list (the number of countries
on the list)b. applicants from third countries

Goal achievement (outcome)

To what extent are the targets of the projects met (goal achievement) in terms of:

2016

Expected additional investments, other private
entities/public support as a result of subsidy
Ex-ante factors which increase the probability
of a project being successful (employment
growth, revenue growth, successful completion of project), based on regression analysis
of monitoring data and project files.
costs/subsidies and costs/total investments for
all PSOM/PSI projects (without rejected)
costs per country/subsidy per country or costs
per country/total investments mobilised per
country for all PSOM/PSI projects (without rejected)
costs/country (programms costs)
costs/application from third country (programm costs)
Perceptions on communications with RVO
(during selection and project implementation,
including reporting requirements)
number and type of jobs (direct jobs) created
by the PSOM/PSI supported companies (monitoring data, case studies)
number of outgrowers contracted (monitoring
data, case studies)
number of people trained / type of trainings
implemented (monitoring data, case studies)
extent of innovativeness of projects in the
PSOM/PSI country. (for the 48 case studies)
This can be judged based on the innovation
score given by RVO.nl on the ranking form
and/or the classification made by AECF (DCEDdemonstrating additionality guidelines, p.7.)
Sales compared to (initial) business plan for
project supported or projections (monitoring
data, case studies)
Investment (€) per job, compared to initial
business plan for the project supported and to
other PSOM/PSI projects. (monitoring data,
case studies)

Sales (€) per job, compared to initial business
for the project supported plan and to other
PSOM/PSI projects. (monitoring data, case
studies)

Success factors

What factors contribute to success of projects or to failure of projects?

Attribution

What are the changes in the effect variables
in comparison to the situation at the start
(baseline)?
− which is the attribution of the observed
changes to the intervention?

Assessment of innovativeness of project (using
the AECF classification)
Factors which increase (ex ante) the probability of a project being successful. Success of the
project can be operationalized in terms of RVO
definition, growth in annual turnover, growth
in number of jobs (low-level, medium/high
level) (quantitative analysis) These findings
can be cross-checked in the case studies.
% of annual turnover related to initial investment (subsidy, local and Dutch partner, third
parties)(quantitative analysis)
# of new suppliers compared to pre-PSOM/PSI
situation

Market development

impact

What is the external impact of the projects
on the development of the market in the
country of implementation (e.g. replication
of technology and innovation, improved
working conditions...)?

# of clients in general of local company compared to pre-PSOM/PSI situation

# of estimated additional jobs in suppliers
# of suppliers (or outgrowers) that adopted
new technology or commercial practices due
to the project
# of workers whose working conditions improved (e.g. safer) due to training and technology (for 48 case studies)
What is the impact of projects on the imple- # of workers whose responsibilities increased
mentation of CSR principles? How has this af- due to training and technology (for 48 case
CSR, gender, environment fected employees/environmental isstudies)
sues/other local enterprises? How have the
# of female managers (for 48 case studies)
projects influenced gender relations?
# of reported preceived changes in gender
equality (e.g training, promotion) (for 48 case
studies)

Evaluation PSOM/PSI| 189

# of environmental certificates obtained (for
48 case studies)
# of cases where the company shared technical knowledge with outsiders, not limited to
suppliers (for 48 case studies)

Which other (non-intended) effects can be
attributed to the projects?

Impact on employees

What is the impact on the employees of the
PSI supported company? (when possible disaggregation between male/female employees and special attention to impact on female employment and on decent working
conditions for women)

# of employees (male and female) with written
contracts (48 case studies)
Comparison of RVO project data with observations in company, with regard to
- training
- contracts
- wages
- working conditions

Given the information available (gathered by Projects completed
RVO.nl through monitoring and the spin-off Evidence of continuation after project comple-

Sustainability

190 |

# of innovative products / services introduced
on the local market. This can be judged based
on the innovation score given by RVO.nl on the
ranking form and/or the classification made by
AECF (DCED- demonstrating additionality
guidelines, p.7.)
# of products / services that substitute imports
Identification of (non-intended), other effects
on company and broader context
# of workers who received training
(male/female) (for 48 case studies)
Wages of (male and female) employees compared to ex-ante situation, by main labour categories (48 case studies)

2016

Incl.
Spin-off
survey

Non-intended effects

# of companies replicating the project technology

Interviews employees

Sector development

What is the external impact of the projects
on the development of the sector in the
country of implementation (e.g. adaptation
of technology, employment, improved working conditions...)?

tion:
- project continued? Growing?
- cooperation with NL partner continued? Expanded?
- follow-up investment realised for project?
- strength local company?
- commitment to CSR within (other parts of)
the company?
Projects ongoing
Evidence of intentions for continuation:
- viability of project
- strength of local company
- perceptions on longer term relationship between partners
- activities to acquire follow-up investments
- embeddedness of CSR within the company

Incl. Spin-off survey

survey) on stopped, completed and ongoing
projects; how sustainable are the PSOM/PSI
investments?
a. what is the available evidence about the
continuation of completed projects?
b. which factors were important for ensuring
the economic viability of the projects on the
long run?
c. is it possible to give an assessment of the
economic viability of ongoing projects based
on the available information?

Evaluation PSOM/PSI| 191