tombcghtech/api_guidelines huggingface dataset:
Identification — Does the agency need to (uniquely) identify any, or all, of the API consumers? What is the appropriate mechanism for uniquely identifying the customer who is registered via an API consuming application?
Risk appetite — APIs have the potential to introduce new risks to an agency and / or remove some risks. Different API designs may have different risk profiles, which need to be understood.
Business analytics — One side effect of APIs is that more and more information is exchanged electronically, potentially creating new information stores. These new information stores provide new opportunities for amalgamating personal information and sharing information.
Terms of use — What terms of use are appropriate for your API? Consideration should be given to the purpose for the information exchanged via the API and the limitations on its use by the application developer for other purposes. Additionally, consideration should be given to how compliance to the terms of use will be enforced and monitored.
Pay-per-use — While it is possible to consider pay-per-use charging for APIs where commercial entities will be making money from the services offered, pay-per-use can have a negative impact by impeding take-up. Even requiring a one-off charge as an on boarding fee can impact take-up especially by smaller application developers and NGOs. There may also be some legal restrictions to charging for use of APIs. (Pay-per-use should be differentiated from transaction fees or official fees.)
1.9.2. Potential contractual impacts
Creating new contracts and Memorandum of Understandings (MOUs) — Commercial use of APIs may require more rapid contract and MOU development to encourage uptake and to help ensure all the intended consumers are supported. Contractual arrangements that take too long to organise or have onerous conditions will discourage commercial uptake.
Extending existing contracts and MOUs — A new API may have implications for existing contracts or MOUs. As the integration approach has the potential to change, so too does the contract or MOU.
Structure of contracts and MOUs — The structure of a contract or MOU may need to change in line with the API development approach. For example, if the development approach is agile, with frequent API releases, then the structure of the contract or MOU should have the flexibility to easily allow changes to maintain alignment with a rapidly evolving API.
Enforcement of contracts and MOUs — APIs often make information available to consumers in the context of specific terms of use. For example, personal information about the customer is made available from the API to consuming applications for a specific purpose. What are the implications if the application developer then uses the information for other purposes?
Service levels — APIs often require service level agreements (SLAs) with application developers and suppliers. SLAs are important to understand and will likely evolve over time as the API matures. MOUs and contracts should cater for this scenario.
Coverage of MOU — Existing MOUs for information sharing may be sufficient initially, to cover an application developer’s planned use of an API, but once an API is exposed new developers not covered by the MOU may request access. In this situation, careful consideration must be given to whether the MOU should be modified or access to the API denied.
Extent of sign-up — When an API is offered publicly, and application developers register for use, it is important contractually to know whether the individual is signing up to an API in their own capacity or on behalf of an organisation.
Onward information handling — It may be appropriate to contractually impose limitations on the on-going use of information exposed via APIs. Application developers can cut, amalgamate, derive metrics and present information as they choose, once they have obtained data via an API unless you impose limitations.
Responsibilities definition — Agencies may wish to clearly contract responsibilities with application developers to ensure customers of their applications are made aware of their legal obligations in regard to use of agency APIs (for example, changing tax information via their application).
Information currency — Some application developers may be tempted to use the API to update their own local caches of information. Depending on the information update period, this can be risky as it means they are working from potentially obsolete information.
1.9.3. Potential organisational impacts
New services — If this is an agency’s first foray into APIs there may be a completely new model for service delivery, potentially with the need to support new types of customers (for example, software developers) and product release cycles. It is important to consider the organisational implications of this new model.
New processes and procedures — In addition, APIs may introduce new ways of working, the implications of which need to be understood. For example, if the API is automating a manual process then the implications of this new automated process need to be considered.
New skills and techniques — An API channel for service delivery requires a new set of skills and techniques to manage the application developers and the communications with application developers, for example, incident resolution.
Service delivery partners — An API channel opens the opportunity to develop relationships with third parties who deliver services on behalf of the agency. These service delivery partners may need to be engaged and managed in new and different ways.
Availability — The availability of an API should be no less than the availability of the equivalent agency online service or website, as this will have already set an expectation from the user community. An API may need to be available 24 hours a day, 7 days a week (24 / 7), but consideration should be given as to whether support for the API also needs to be 24 / 7.
Service levels — Opening an API channel may necessitate an agency to offer a level of service, where previously there has been no such need. Commercial experience and knowledge could be required in order to define realistic and achievable service levels that serve everyone. These service level changes may require additional staff to cover increased support hours, for example. There is also a need to measure, monitor and report back on performance to ensure SLAs are met.
Support services — There may be a reduced need for agency support services, where consuming applications replace agency presentation layers, moving customer calls from agency support staff to the application developers.
Principles — In order to incorporate all-of-government thinking into API development it is useful to have a set of business principles that should influence the way APIs are built and their usefulness over time.
Governance — API governance helps save time and money because it ensures that APIs are built proactively to achieve specific goals and bring value to the agency. API governance also helps agencies make intelligent decisions regarding API programmes and establish best practices for building, deploying and consuming APIs.
2. Principles
These are a set of principles that are considered good practice when designing, implementing and operating APIs. The New Zealand Government Digital Service Design Standard provides a wider perspective and set of principles that should also influence the design of APIs and the services they support.
Digital Service Design Standard
2.1. Design for the consumer
APIs need to be developed with the consumer primarily in mind. The application developers also need to be considered because, of course, if the API does not meet their needs then they will not develop consuming applications that use it. This leads to APIs being developed as generic as possible in order to meet the basic needs of all potential consumers.
API developers should not try to predict how the consumer will interact with consuming applications and should leave the application developer to innovate. By designing APIs for the consumer, agencies are likely to build APIs that are intuitive and easy to use, thus ensuring uptake of their APIs and encouraging access to public information.
Initially, application developers will be the main users of APIs. Hence development and delivery of APIs should be geared around making it as easy as possible for developers to discover, understand and develop against those APIs. So, APIs, along with the associated onboarding and support processes, should be simple to understand and well described.
Some examples of this principle in action are:
ensuring a low barrier to entry so it is easy to start using the API
providing sandbox APIs so application developers can try out APIs and develop in parallel
being responsive to feedback and bug reports
providing automated onboarding processes, as manual processes can limit take-up
providing prototyping tools and support for development
ensuring the funding cycle does not introduce a bottleneck for uptake within government
selling the concept to the consumer early on so they create demand that industry needs to meet
creating an atmosphere of competition in industry for take-up of the API
creating a software development kit (SDK) to support an agency’s APIs, including examples.
2.2. Design-driven development
This builds on the principle of ‘Design for the consumer’ and could also be phrased as ‘specification-driven development’ because the interface specification for the API is written first. Designing the interface does not require full design of the code that will sit behind it. The interface can be in place without the functional capabilities being built and yet still be useful.
By defining the interface first, there is a known integration point, which both the application developer and API provider can refer to and build to. Potential application developers can use the specification to assess whether the API will meet their needs, or if further information or parameters are essential to them.
The process of creating a specification before development forces the API developer to properly consider the purpose and content of the interface they are providing, before coding starts. Taking a design-driven approach simply means that the API design should be performed before development begins — it does not mean that development is hindered in any way.
There are a number of tools that assist with this process, however the tool to create the specification should be of little significance, the act of creating the specification is what counts.
2.3. Co-design
This principle is the natural extension to both the ‘Design for the consumer’ and ‘Design-driven development’ principles and includes collaborating with the consuming application developer community when designing the interface. This is important to help meet the Strategy for a Digital Public Service aims to co-create with partners and make business engagement with government more efficient, not just for business as a customer of government but also where business works together with government. Co-design works as an enabler to business innovation and supports better interaction with government.
Application developers are one of the primary customers for the API and, as such, are a primary source of requirements. Working with application developers to define the requirements and to design the interface ensures the API will meet their needs. However, identifying potential application developers and understanding the demand from the market can be difficult. Careful consideration should be given as to the best approach to engaging with the market and identifying your potential application developer community.
Additionally, it is strongly encouraged that application developers start to produce API consuming applications based on the interface specification as early as possible. This agile or iterative approach helps ensure real-world feedback is incorporated into the API design as early as possible.
Completeness is not necessarily the goal, especially in initial APIs. The goal should be to get early partially complete releases out, defining the limited capability they offer, to enable consuming application uptake. Development needs to be flexible and agile to adapt to early adopters’ feedback on the identification of pitfalls and issues. However, early releases should be tested and stable so as not to impede uptake. The aim is to try, then adapt, rather than waiting to release a fully functional API.
2.4. Future-focused design
Most agencies will have a variety of legacy systems that they need to continue to support and service. It is important to remember that business, technology and application architects should be designing for the future of their organisation, and not ‘hamstring’ their APIs under development by designing them to work in the way the legacy system currently works, or to tailor APIs so that they work perfectly with all legacy systems. The aim is to be future focused (while still pragmatic) and develop APIs to meet future needs.
These needs will be agency specific but may include a drive to move to a wholesaler rather than a retailer role, where the agency reduces its web presence in favour of enabling commercial entities to create applications, or add richness to existing applications, through access to agency APIs.
2.5. Dogfooding
When building an API, there is always a danger of building the wrong thing in the wrong way for the wrong people. This is especially a risk in the absence of external developers driving or informing the development of the API.
The simplest way to ensure an API is useful and consumable is to build the API as part of delivering a larger-scale internal business support system such as a web application. In this way, the customer and application developer are internal, and the first principle ‘Design for the consumer’ applies.
The best way to design an API is in collaboration with the potential consuming application developers. The best way to understand and address the weaknesses in an API’s design and implementation is to use it in a production system. So, wherever feasible, design an API in consultation with application developers but build it in parallel with an accompanying integration of that API to ensure it is fit for purpose — in other words, build for inside use, then expose. This has the added advantage of meaning that the cost of API development is covered throughout the build of the business support system or integration mechanism, reducing the need for extra funding.
2.6. Stability and backwards compatibility
It is important that APIs have stability (are available and work consistently) and support a velocity of change that is acceptable to the application developers. Early versions of APIs should be available via pilots or on developer portals so application developers can work them and identify areas of instability before an API goes into production.
Application developers will not always be able to adapt to new capabilities or changes to existing interfaces as quickly as the API providers might wish, due to organisation priorities and funding. Hence, minor changes to APIs must always be deployed as fully backwards-compatible upgrades.
For major changes, which are not backwards compatible, the old API version should be maintained alongside the new version for an appropriate period to allow all-consuming applications to transition. By monitoring usage, it should be possible to assess when an API version can be deprecated, either because it is no longer being used or because the usage pattern does not warrant maintaining that particular version.
In any case, it is important to clearly communicate with your developer community and manage expectations as to the longevity of a particular version of an API.
2.7. Privacy and sensitivity aware
APIs are used extensively for passing information on, so it is important to consider the information privacy and sensitivity aspects of data being passed to and from APIs to ensure that data is protected adequately.
Consideration should be given as to whether a privacy impact assessment and / or a security risk assessment is appropriate for the API during each stage of development, from concept through to implementation.
Privacy Impact Assessment Toolkit — Privacy Commissioner Te Mana Mātāpono Matatapu
Privacy, security and risk
For example, if the API is providing programmatic access to publicly available information, the privacy considerations are likely to be minimal and the security considerations limited to the usual suspects, like the Open Web Application Security Project (OWASP) top 10.
However, privacy and security considerations become hugely important if the API is providing programmatic access to private, personal information. In this situation, it may be appropriate to do regular assessments, especially early in the concept phase to ensure any privacy or security constraints are understood before design.
There is also the issue that with ease of data consumption comes increased ability to combine data from different sources, which increases privacy risks and the potential for unintended information leakage.
2.8. Well managed
As API take-up increases over time, there is more likelihood of API change, redevelopment and adaptation to meet new needs. A well-managed API should handle regular change.
API management gives an agency the capability to deliver good version control capability and forewarn application developers of changes that may impact them, including failures and outages. For this reason, it is important that all application developers are identified, even if the API is considered public.
3. Key characteristics of an API
At its heart, an API is, as its acronym suggests, an application programming interface. This simply means that an API is an interface that enables software inter-communication. The purpose of these interfaces is to enable safe and reliable communication in a standards-based way.
An API should strive to deliver these functional and non-functional requirements:
Functional requirements:
usability.
Non-functional requirements:
scalability.
3.1. Reference architecture
Part A, and companion Parts B and C, are based on the reference architecture information in this section. Due to the nature of these guidelines, the reference architecture addresses numerous aspects of an API and is reasonably complex. Depending on their circumstances, agencies may choose to:
use this reference architecture
simplify the reference architecture to better suit their needs, or
develop their own reference architecture.
Figure 6 illustrates the reference architecture, with the concepts further expanded, layer by layer.
Figure 6: API reference architecture
Figure 6 illustrates the layers of the API reference architecture. When designing and developing an API, analysis is required across all layers of the reference architecture to meet the needs of all parties. From top to bottom these layers are as follows:
Consumers (consuming applications) — discover the API by using the interface specification.
Interface specification — the layer between consumers and the API. This needs to be published for consumers to discover the API.
API — provides authentication, course-grained authorisation and validation / threat protection, and may perform simple orchestration. It makes sure the interface specification is maintained and published. It is stateless.
Application / logic — performs platform authentication, fine-grained authorisation and validation, and may perform complex orchestration. It is stateful.
Data — performs platform authentication. It is stateful.
Some layers are linked by:
context — linked from:
the Consumers to the API layers, and
the API to the Application / logic layers
analytics — linked from the API to the Data layers.
Trust exists between individual layers, but decreases the further up the model you go, with a high level of trust in the lower layers (data, application / logic) but minimal trust in the upper layers (consumers).
These layers are described in more detail in sections 3.1.1. to 3.1.9.
View larger image (PNG 54KB)
3.1.1. Consumers
Relates to the usability functional requirement.
Consuming applications can be anything that uses an API. Consuming applications ‘drive’ your APIs.
Application developers build the applications that consume your APIs. When designing an API, the usability or function of the API should be aimed at this group.
Interface specification must be published for it to be discoverable. API discovery refers to the way that your APIs are understood by an application developer. Application developers like to see an API or interface specification in a format that they are familiar with. The functional and non-functional behaviours of the API should both be well described. Refer to Part C: API development 2022 for details.
The consumers component has a low level of trust.
3.1.2. Interface specification
Relates to both the usability and security functional requirements.
An API interface specification is a technical description and reference model for an API. The specification is used most commonly by application developers to understand the technical aspects of an API. However, interface specifications should also be provided in a machine-readable format. Refer to Part C: API development 2022 for details.
3.1.3. API
The API component of the reference architecture is multifunctional. It provides API authentication and authorisation, preferably using a standard approach. Refer to Part B: API security 2022 for details.
To protect an agency’s resources, the API component should perform threat protection and quality of service (QoS) functions. This could be in the form of ‘throttling’ requests, schema validation and common vector attack protection (Structured Query Language (SQL) injection, Cross Site Scripting). This functionality should not be seen as a replacement for other defences, but an augmentation to those defences.
In some cases, simple orchestration may be performed within this component. An example of simple orchestration is a ‘mashup’ of 2 backend services or applications into a single API.
The API component should be responsible for maintaining interface specifications and publishing these to the interface specification component.
In a bimodal IT model this component emphasises agility and speed.
This component has a medium level of trust.
3.1.4. Application / logic
This component is where complex business logic is handled. In many cases this will be a system or group of systems that exist at an agency which do not conform to a common standard. This component is where fine-grained authorisation should be performed based on business logic, which the API component should have no knowledge of. In a bimodal IT model this component emphasises safety and accuracy.
This component should verify the authenticity of the API component, referred to as ‘platform authentication’.
This component has a high level of trust.
3.1.5. Data
The data component represents one or more sources of data within an agency. It does not necessarily need to be a traditional data store or database — it simply represents the fact that an agency will inevitably have data that can be represented in an API.
This component traditionally verifies the authenticity of the application component, referred to as “platform authentication”.
The data component has the highest level of trust.
3.1.6. Context
Context can take many forms and it is simply a way of providing contextual information for the request to downstream components that may require it.
The main purpose of context is to prevent layers of the architecture having to look up information that has already been gathered elsewhere by something that the component trusts. A good example of this is where the API component has performed authentication and gathered role information for the customer.
If the application components need to perform fine-grained authorisation, the API component can pass customer role information as context for the request.
3.1.7. Analytics
Analytics are key to the success of an API programme. Technical analytics enable you to maintain API stability and performance, while business analytics enable you to grow your API programme.