NL – Position Paper on Regulatory Sandboxes

09.2022

Position Paper – ‘Regulatory Sandboxes: Testing and Learning’

1. Introduction: The first EU sandbox framework
Regulatory sandboxes (hereafter: sandboxes) are not a new concept. They are ‘schemes that
enable organisations to test innovations in a controlled real-world environment, under a specific
plan developed and monitored by a competent authority.’ 1
Up until now, sandboxes have been a national phenomenon. The proposed Artificial Intelligence Act
(hereafter: AI Act) creates a European legal framework for sandboxes in Articles 53-54b. 2
NL is pleased that the framework in its current wording already provides a strong level of support
for developers that participate in sandboxes, especially for small and medium enterprises. NL
thinks that even more value can be gained from sandboxes by further utilizing the knowledge they
generate on a European level to improve regulatory practices and future legislation, i.e. through
regulatory learning.
2. The need for regulatory learning
Compliance with legislation for a quickly developing field of technology is challenging for both
developers and competent authorities. Regulatory learning can help them both.
In a sandbox, developers have the opportunity to receive compliance assistance by a competent
authority during their product development process. This results in better understanding of the
applicable rules for the developer, a reduction of development costs and a ‘safe space’ due to
supervisory flexibility in order to boost innovation. In turn, competent authorities can increase their
understanding of the legislation as well so that they can improve their future supervisory practices.
So why should we ensure that their lessons learnt are utilized to the best extent possible?
-

-

-

-

Improved supervisory practices. National Competent Authorities (NCAs) can learn from
each other’s sandbox experiences. Sharing technical and procedural knowledge between NCA’s
helps improve supervisory practices throughout the EU. This is especially prudent since it will
be necessary to quickly build the required knowledge about and experience with supervision on
AI in a quickly growing EU market.
Clarity about the law. If interpretations of the law are shared between NCAs and
communicated to the wider public, developers have better knowledge about what is expected
from them throughout the whole European Union. This strengthens the EU AI-market and
facilitates cross-border trade in the EU.
Evidence-based policy making. Various aspects of the AI Act can be amended through
delegated acts such as the areas of AI that are classified as high-risk. Practical experience in
sandboxes can provide insight into the current working of the AI Act. A coordinated learning
approach ensures a continuous feedback-loop between developers, NCAs and the Commission.
The soft law instruments of the AI Act, such as guidance, can also be informed by the lessons
learnt of sandboxes.
A level playing field. Sandboxes are scarce resources and therefore bear the risk to disrupt
the level playing field for market participants. Developers with access to a sandbox will reap
the benefits from their participation while other market-players are left empty-handed. To
justify this distinction between developers, the knowledge generated in sandboxes needs to
flow back into the market through information-sharing and utilizing obligations so that the
whole market benefits from improved supervisory practices, clarity about the law and better
future regulation.

Commission’s Better Regulation Toolbox (November 2021), p. 597.
Building on earlier reflections on a European framework for sandboxes in the Commission’s Better Regulation
Toolbox (November 2021 edition) and the Council Conclusions on regulatory sandboxes (12683/1/20 REV1).

1
2

NL – Position Paper on Regulatory Sandboxes

09.2022

3. How do we achieve regulatory learning?
Textual changes to the AI Act are possible to facilitate regulatory learning in addition to the current
set-up that already benefits participating AI-developers. Below we explain the main changes to the
text and why these are necessary, also listed in Annex I.
-

-

-

-

Explicitly name regulatory learning as a goal of sandboxes. In the latest draft text (June
2022), three objectives were removed from art. 53 (1b). These objectives specifically addressed
‘improved supervision’ and ‘evidence-based policy making’ and should therefore be returned
because these goals reflect the above-mentioned aspects that will make sandboxes even more
beneficial for the EU’s AI-market. Furthermore, we propose that the implementing acts of art.
53(6) also support, amongst other things, regulatory learning to the best extent possible.
Ensure that knowledge is documented. The lessons from sandboxes are only comparable if
there is a common framework for the learning aspect. We propose to lay down harmonized rules in
the AIA for the documentation inside sandboxes; strengthening the plan of participation and
introducing an exit report. These documents ensure that learning goals are set and evaluated,
without compromising on the flexibility of NCAs on how they want to engage with participants.
We value this flexibility in the design of sandboxes themselves, since this design should also be
amendable based on new insights. The Spanish AI Sandbox pilot, launched on 27 June, is a good
start of this learning-cycle. This timely pilot will be the first to define the needs of developers and
competent authorities alike in the design of AI regulatory sandboxes. These lessons can serve as a
solid basis for the modalities and conditions of the first EU sandbox.
Ensure that knowledge is utilized. In order to establish a feedback-loop and concretise learning
from a sandboxes, we propose to explicitly include the following obligations for the Commission,
the AI Board and the European Standardisation Organisations. If an expert pool is included in a
future text of the AIA, we suggest to lay a similar explicit (feedback-)link between sandboxes and
this pool.
o the Commission should consider annual reports and exit reports of NCAs when amending
Annex I, II, IV and when drafting guidance or common specifications;
o the AI Board should consider annual reports and exit reports of NCAs when carrying out its
tasks.
o the ESO’s should take these reports into account when developing harmonised standards.
Prevent sandbox-shopping while ensuring flexibility for NCAs. In line with the recent Czech
compromise proposal, we recognise the importance for NCAs to adjust their procedures to the
national context. To further stress the importance of this flexibility for NCAs, we propose to add
that the modalities and conditions that will be laid down in implementing acts ‘shall to the best
extent possible support flexibility for national competent authorities…’. Nonetheless, we propose to
lay down harmonized rules (instead of principles) for criteria for participation and the terms and
conditions applicable to participants. This prevents ‘sandbox-shopping’ between different Member
States and a consequential competition between NCAs to attract participants.
4. Testing in real world conditions
We see real world testing as an important tool for the development of high-quality AI-systems for
developers that are dependent on testing AI-systems outside a lab environment. We expect a high
demand for testing in real world conditions and are glad that there is flexibility to decide whether it
takes place inside or outside a regulatory sandbox.
In order to ensure the necessary level of protection, we propose to change Article 53(7) to apply
the relevant safeguards of Articles 54a and 54b to testing in real world conditions inside sandboxes
as well.

NL – Position Paper on Regulatory Sandboxes

09.2022

Annex I – text suggestions
Current AIA text (June 2022) and proposed
changes

Article 53
AI regulatory sandboxes
-1e
Regulatory sandboxes that consider AI-systems and
that are established under national law before the
entry into force of this Regulation shall be exempt
from the obligations concerned of paragraphs 2a, 2b,
4a, 5 and 5a of this Article and from the general
common rules that are adopted through
implementing acts under paragraph 6 of this Article.
This exemption lasts for a maximum period of two
years after the entry into force of this Regulation.
Member States shall endeavor to implement the
obligations of this Article into those already existing
sandboxes to the best extent possible.

-1a. National competent authorities may establish
AI regulatory sandboxes for the development,
training, testing and validation of innovative AI
systems under the direct supervision, guidance and
support by the national competent authority, before
their placement on the market or putting into
service. Such regulatory sandboxes may include
testing in real world conditions supervised by the
national competent authorities.

1b
The establishment of AI regulatory sandboxes under
this Regulation as defined in paragraph 1 shall aim

Comments

There have been signals that harmonized
rules may interfere with already existing AI
regulatory sandboxes that have been
established under national law.
With this paragraph, we aim to give Member
States some leeway when implementing this
Regulation. During the first two years of this
Regulation, they are encouraged to
implement the requirements of this Article
as much as possible into already existing
sandboxes but have no obligation to do so.
The obligations that those already existing
sandboxes are exempt from concern mostly
the design and rules of the sandbox.
Consequently, NCAs are also exempt from
paragraph 5 (annual reports) since these
already existing sandboxes cannot be
compared with the harmonized sandboxes
established under this Article.
Add “under the direct supervision, guidance
[..] by the national competent authority”.
The key element of supervision and
guidance by the national competent
authority was deleted by deleting the whole
article 53 (1) and should be returned.
Add “support”: Especially for start-ups it is
very important that competent authorities –
within their legal possibilities – act as
supporters in ensuring compliance, e.g.
through mentoring, personal exchange or
customized guidance. The impressive
examples of data regulatory sandboxes by
the French CNIL and the British ICO also
explicitly “support” the projects. The term
“support” is also used in EU Commission’s
Better Regulation Toolbox Tool #69 on
regulatory sandboxes (page 597).
Although the additional objectives of AI
regulatory sandboxes have also to some
extent been listed in Recital 72, we propose
to return these into the Article itself.

NL – Position Paper on Regulatory Sandboxes

to contribute to one or more of the following
objectives:
a) foster innovation and competiveness and
facilitate the development of an AI ecosystem;
b) facilitate and accelerate access to the Union
market for AI systems, including in particular
when provided by small and medium enterprises
(SMEs), including and start-ups;
c) improve legal certainty and contribute to the
shareing of best practices through cooperation
with the authorities involved in the AI regulatory
sandbox with a view to ensuring future
compliance with this Regulation and, where
appropriate, with other Union and Member
States legislation;
d) enhance authorities’ understanding of the
opportunities and risks of AI systems as well as of
the suitability and effectiveness of the measures
for preventing and mitigating those risks;
e) contribute to the uniform and effective
implementation of this Regulation and, where
appropriate, its swift adaptation, notably as
regards the techniques in Annex I, the high-risk
AI systems in Annex III, the technical
documentation in Annex IV;
f) contribute to the development or update of
harmonised standards and common
specifications referred to in Articles 40 and 41
and their uptake by providers.
d) enhance authorities’ understanding of the
opportunities and risks of AI systems as well as of
the suitability and effectiveness of the measures
for preventing and mitigating those risks;
e) contribute to the uniform and effective
implementation of this Regulation and, where
appropriate, its evidence based swift adaptation,
notably as regards the techniques in Annex I, the
high-risk AI systems in Annex III, the technical
documentation in Annex IV;
f) contribute to the development or update of
harmonised standards and common
specifications referred to in Articles 40 and 41
and their uptake by providers.
g) Contribute to the possible future evidence-based
advancement of this Regulation and, where
appropriate, of other Union and Member States
legislation.
2a

09.2022

This underscores the importance of
regulatory learning in sandboxes. To further
underscore this, we propose to add another
objective to this list. Regulatory sandboxes
should contribute to resilient and relevant
legislation through facilitating regulatory
learning.

First, we think it is important to limit the
participation to sandboxes to providers

NL – Position Paper on Regulatory Sandboxes

Access to the AI regulatory sandboxes and
supervision and guidance by the relevant authorities
shall be free of charge, without prejudice to
exceptional costs that national competent authorities
may recover in a fair and proportionate manner. It
Access to the AI regulatory sandboxes shall be open
to any provider or prospective provider of an AI
system who is established in the Union or has
appointed a legal representative who is established
in the Union and fulfils the eligibility and selection
criteria referred to in paragraph 6(a) and who has
been selected by the national competent authorities
or, where applicable, by the European Data
Protection Supervisor following the selection
procedure referred to in paragraph 6(b). Providers
or prospective providers may also submit
applications in partnership with users or any other
relevant third parties.
Participation in the AI regulatory sandbox shall be
limited to a period that is appropriate to the
complexity and scale of the project in any case not
longer than a maximum period of 2 years, starting
upon the notification of the selection decision. The
participation may be extended for up to 1 more year.
This period may be extended by the national
competent authority.
Participation in the AI regulatory sandbox shall be
based on
a specific plan referred to in paragraph 6 of this
Article that
shall be agreed between the participant(s) and the
national
competent authoritie(s) or the European Data
Protection
Supervisor, as applicable. The plan shall contain as a
minimum the following:
a) description of the participant(s) involved and
their roles, the envisaged AI system and its
intended purpose, and relevant development,
testing and validation process;
b) the specific regulatory issues at stake and the
guidance that is expected from the authorities
supervising the AI regulatory sandbox;
c) the specific modalities of the collaboration
between the participant(s) and the authoritie(s),
as well as any other actor involved in the AI
regulatory sandbox;

09.2022

established in the Union or has a legal
representative in the Union, similar to the
requirement to testing in real world
condition 54a(4)(d).
But more importantly, we want to return the
requirements to the specific plan in this
Article.
When NCAs and participants have to think
about the included elements before they start
their cooperation within a sandbox, they will
know exactly what the added value of
participation is. This is true for both the
participant and the national competent
authority.
The participant can check whether it is
necessary to enter into a time-consuming
process or whether the questions that they
have can be answered through readily
available information.
The NCA can help the participant with this
process and can also assess whether the
questions of the participants justify the
resource-heavy process of a sandbox.
Harmonizing the rules concerning this
specific plan of participation helps
regulatory learning as well.
- It is important to have a clear
objective in mind when operating a
regulatory sandbox.
- If the context of participation is
documented well, it is easier to
compare the results of the sandbox
with sandboxes that have taken place
under the supervision of other NCAs.
Additionally we propose a new provision
2a(bb). Note that this does not require
participants to have a novel regulatory issue
in order to participate in the sandbox.
Whether a regulatory issue is novel can also
become clear during the sandbox.
Furthermore, we also propose a new
provision 2a(f). It is important to think about
the information that has to be shared before

NL – Position Paper on Regulatory Sandboxes

09.2022

d) a risk management and monitoring mechanism
to identify, prevent and mitigate any risk
referred to in Article 9(2)(a);
e) the key milestones to be completed by the
participant(s) for the AI system to be considered
ready to exit from the regulatory sandbox.
a) description of the participant(s) involved and
their roles, the envisaged AI system and its
intended purpose, and relevant development,
testing and validation process;
b) the specific regulatory issues at stake, and the
guidance that is expected from the authorities
supervising the AI regulatory sandbox;
bb) the novelty of the specific regulatory issue,
compared to the annual reports referred to in Article
53(5), and whether analyzing this regulatory issue in
the regulatory sandbox contributes to the objectives
of Article 53(1b)(c) and (d);
c) the specific modalities of the collaboration
between the participant(s) and the authoritie(s),
as well as any other actor involved in the AI
regulatory sandbox;
d) a risk management and monitoring mechanism
to identify, prevent and mitigate any risk
referred to in Article 9(2)(a);
e) the key milestones to be completed by the
participant(s) for the AI system to be considered
ready to exit from the regulatory sandbox;
f) the information that has to be shared between the
participant(s) and the authoritie(s) to allow
proper evaluation of the project.

the project takes off. This is important for
both the participant (who then knows what
they have to share) as well as the
supervisory authority (to ensure that the
necessary information for evaluation is
collected during the sandbox).

2b
After an AI regulatory sandbox has ended, the
participant(s) and the national competent
authoritie(s) or the European Data Protection
Supervisor, as applicable, shall draw up an exit
report. This exit report shall contain as a minimum
the following:
a)
The plan referred to in paragraph 2a of this
Article;
b)
An evaluation of the specific regulatory issues
that were at stake during the AI regulatory sandbox,
including a problem definition and proposed
solutions;
c)
Whether the key milestones referred to in
paragraph 2a(e) of this Article have been completed;
d)
A conclusion on the lessons learnt, specified in
the following categories of use:

In various national regulatory sandboxes, it
is common practice to issue an exit report
after the sandbox has concluded. We
propose to include this practice in the AI Act
as well. The exit reports focus more
specifically on the case at hand, instead of
the more vaguely drafted ‘annual reports’
(which also focus on the implementation of
sandboxes).

It merely requires the participant and NCA
to think about whether this is the case. This
helps to assess whether the sandbox has
value for guidance or other lessons learnt
after the sandboxes has ended.

In order to truly utilize lessons learnt, they
must first be defined. The national
competent authorities are in the best position
to do this, right after a sandbox has ended.
Under paragraph 5a, the exit reports will
then be used by the AI Board and
Commission to improve interpretation,

NL – Position Paper on Regulatory Sandboxes

09.2022

a.
An improved understanding on the
implementation of the AI regulatory sandboxes;
b.
Improved methods of supervision by national
competent authorities;
c.
A revised or novel interpretation of this
Regulation.
4a.
Upon request of the provider or prospective provider
of the AI system, the national competent authority
shall provide, where applicable, a written proof of
the activities successfully carried out in the sandbox.
The national competent authority may also provide
an exit report detailing the activities carried out in
the sandbox and the related outcomes. Such written
proof and exit report could be taken into account by
market surveillance authorities or notified bodies, as
applicable, in the context of conformity assessment
procedures or market surveillance checks.

guidance, communication and amendments
regarding this Regulation.

5.
Member States’ National competent authorities that
have established AI regulatory sandboxes and the
European Data Protection Supervisor shall
coordinate their activities and cooperate within the
framework of the European Artificial Intelligence
Board. They National competent authorities shall
make publicly available publish on their websites
submit annual reports on to the Board and the
Commission on the results from the implementation of
those the AI regulatory sandboxes, including good
practices, lessons learnt and recommendations on their
setup and, where relevant, on the application of this
Regulation and other Union legislation supervised
within the sandbox. Those annual reports shall be
submitted to the AI Board which shall annually
make publicly available publish on its website a
summary of all good practices, lessons learnt and
recommendations. This obligation to make annual
reports publicly available shall not cover sensitive
operational data in relation to the activities of law
enforcement, border control, immigration or asylum
authorities. The Commission and the AI Board shall,
where appropriate, take the annual reports into
account when exercising their tasks under this
Regulation.

Small addition to ensure regular feedback
from the AIB.

Although we are glad that the ‘exit report’ is
mentioned in the latest compromise text, the
addition to art. 53(4a) does unfortunately not
support regulatory learning. It seems almost
identical to the already existing written
proof, and only relevant in the context of
conformity assessment procedures or market
surveillance checks.
We therefore propose to delete this new
addition and include the exit report as
proposed in our paragraphs 53(2b) (contents
of the exit report) and 53(5a) (utilization of
the exit report).

We welcome the last sentence of this
paragraph, which was added in the latest
compromise proposal. Nevertheless, we still
propose to introduce an exit report that
ensures that lessons are actually documented
and further utilized. This is also necessary to
draw up useful annual reports.

NL – Position Paper on Regulatory Sandboxes

09.2022

5a.

To ensure that sandboxes will deliver more
1. The exit report shall not be made accessible to than vaguely defined annual reports, this
the public, unless both the participant and the paragraph requires the AI Board and
national competent authority have explicity
Commission to utilize the exit reports that
agreed with this.
have been drawn by the national competent
2. After an AI regulatory sandbox has ended,
authorities.
the national competent authority shall share
the exit report referred to in Article 53(2b)
As these exit reports may contain sensitive
with the AI Board and the Commission,
information that should be kept confidential,
unless the participant explicitly objects to
an explicit reference to Article 70 has been
this.
made. If a participant still feels
3. The exit reports shall be shared on a
uncomfortable with sharing this exit report
confidential basis and in accordance with
with the AIB and the Commission, it may
Article 70.
object to this.
4. The AI Board shall use the annual reports of
paragraph 5 of this Article and the exit
Also, the exit report will only be made
reports it recieves according to paragraph 1
public if both the concerning participant and
in the exercise of its tasks as listed in Article
the national competent authority have given
58.
consent to this.
5. The Commission shall use the annual reports
of paragraph 5 of this Article and the exit
This also prevents a situation in which
reports it recieves according to paragraph 1
participants may be reluctant to participate
in the exercise of its tasks in Articles 4, 7,
in sandboxes because they are afraid that
11(3) and 58a.
their trade secrets or other sensitive
information will be made public.
6.
We propose to return to the previous text.
The detailed modalities and the conditions for the
The lessons from sandboxes are only
establishment and of the operation of the AI regulatory comparable if there is a common framework
sandboxes under this Regulation, including the
for the learning aspect. ‘Common main
eligibility criteria and the procedure for the application, principles’ may result in differently
selection, participation and exiting from the sandbox,
organised sandboxes throughout Europe.
and the rights and obligations of the participants shall
be set out in implementing acts. Those implementing
We included an additional sentence to
acts shall be adopted through implementing acts in
ensure the objective to ensure flexibility for
accordance with the examination procedure referred to
national competent authorities, foster AI
in Article 74(2). These modalities and conditions shall innovation (recital 71) and regulatory
to the best extent possible support flexibility for
learning. In order to promote innovation, it
national competent authorities to establish and
is important that the interests of small-scale
operate their regulatory sandboxes, foster
providers are taken into particular accounts
innovation and regulatory learning and shall take
(recital 73). This is must be reflected in the
into account particularly the special circumstances
regulatory sandboxes’ modalities and
of participating SMEs .
conditions.
Those implementing acts shall include general
common rules common main principles general
common rules on the following issues:
a)
the eligibility and selection criteria criteria for
participation in the AI regulatory sandbox;
b)
the procedure for the application, selection
participation, monitoring, and exiting from and

The more procedural aspects of sandboxes
regarding the application, participation,
monitoring, exiting from and termination
can be laid down in common main
principles in order to ensure flexibility for
national competent authorities.

NL – Position Paper on Regulatory Sandboxes

09.2022

termination of the AI regulatory sandbox, including
templates of all relevant documents;
c)
the terms and conditions applicable to the
participants., including in relation to their
collaboration with the authorities supervising the
sandbox, as well as the conditions for suspension and
termination of the participation in the sandbox;
Those implementing acts shall also include common
main principles on the following issues:
d)
procedure for the application, participation,
monitoring, evaluation, exiting from and
termination of the AI regulatory sandbox.
e)
the modalities for the involvement in the AI
regulatory sandbox of other national authorities and
other actors within the AI ecosystem;
f)
the modalities and procedures for crossborder cooperation, including the establishment and
operation by two or more Member States of crossborder AI regulatory sandboxes.
7
1.
Testing in real world conditions supervised
within the framework of an AI regulatory sandbox
established under this article may be authorised by
the national competent authority.
2.
When authorised, providers may conduct
testing in accordance with Articles 54a(2), 54a(3),
54a(4)(c), 54a(4)(e-l), 54a(5), 54a(6), 54a(7), 54a(8)
and other terms and conditions of such testing
agreed between the participant and the national
competent authority.
When national competent authorities consider
authorising testing in real world conditions
supervised within the framework of an AI
regulatory sandbox established under this Article,
they shall specifically agree with the participants on
the terms and conditions of such testing and in
particular on the appropriate safeguards, with the
view to protect fundamental rights, health and
safety. Where appropriate, they shall cooperate with
other national competent authorities with a view to
ensure consistent practices across the Union.

We propose to use the relevant requirements
of Article 54a when allowing testing in real
world conditions in an AI regulatory
sandbox.
We feel like these are reasonable safeguards
that ensure that the rights of the persons
involved are respected.
Allowing national competent authorities and
participants to draw up their own terms and
conditions carries the risk of ‘unsafe
testing’. This risk is increased to the
undefined term of an AI regulatory sandbox,
potentially carrying on indefinitely.
An undefined term for an AI regulatory
sandbox is acceptable, since the goal will
always be to end up with a product that is
compliant with the AI Act. On the contrary,
indefinite testing can be a loophole to use an
AI-system without complying with the AIA.