# Packet Forwarding Control Protocol

## Summary

Packet Forwarding Control Protocol (PFCP) is a protocol used for communicating between control plane (CP) and user plane (UP) functions in 4G (Release 14 onwards) and 5G networks.  In these networks, the concept of **Control and User Plane Separation (CUPS)** separates the two planes. However, there's a need for these two planes to communicate across the newly defined interfaces. For this purpose, PFCP was defined.

PFCP sits on top on UDP/IP. It functions only in the control plane. A control plane node uses PFCP to associate with one or more user plane nodes and subsequently configure PDU sessions for the user plane. PFCP consists of node-related or session-related messages. Messages and IEs are encoded in TLV format.

Open source implementations of PFCP are available.

## Discussion

### Where does PFCP fit in?

In 4G EPC, PFCP is used on the **Sx interfaces**. In particular, these are Sxa (SGW-C and SGW-U), Sxb (PGW-C and PGW-U) and Sxc (TDF-C and TDF-U). Due to CUPS, traditional Serving Gateway (SGW) and PDN Gateway (PGW) were split into SGW-C and SGW-U, and PGW-C and PGW-U respectively. PFCP was defined to help these split entities to communicate. The Traffic Detection Function (TDF) is an optional function and it was also split. Its parts also use PFCP.

It's also possible to combine SGW-C and PGW-C into a single control plane node. Likewise, SGW-U and PGW-U can be combined. These combined entities can communicate using PFCP.

In 5G, CUPS was applied from the outset, that is, in Release 15. The equivalent control plane and user plane nodes are Session Management Function (SMF) and User Plane Function (UPF). These are connected via the **N4 interface**. PFCP is used on this interface.

In some scenarios, user plane data packets may be sent on these interfaces between CP and UP functions. However, such packets don't use PFCP. They're sent using GTP-U over UDP/IP.

### Which are the main procedures in PFCP?

The figure shows the list of PFCP messages, which follow the procedures that PFCP needs to perform:

    + **Node-related**: Heartbeat (to know if the peer node is alive); Load Control (UP sends CP its current load); Overload Control (UP informs CP that it's overloaded); Association Setup/Update/Release (relate to associating UP with CP); Packet Flow Description (PFD) Management (provision PFDs to UP); Node Report (UP reports non-session-related information to CP).
    + **Session-related**: Session Establishment/Modification/Release (configure rules in the UP to handle packets on a per-session basis); Session Report (UP reports session-specific information to CP).PFCP runs on top of UDP, which doesn't guarantee packet delivery. For this reason, PFCP **retransmits** every request for which a response is not received within a defined timeout. Timeout value and number of retransmissions are implementation specific. A request-response pair share the same sequence number.

For efficiency, many session-related messages targeting the same peer PFCP entity can be **bundled** together. If bundled messages remain unacknowledged, they may be retransmitted individually. Bundling is not applicable for node-related messages.

### What's meant by PFCP Association?

Before a UP function can carry packets, it must first be configured by a CP function. It's for this purpose that PFCP exists. However, even before CP configures UP for a session, it must first select a suitable UP. This is exactly what the association procedure achieves. A CP associates with a UP before establishing sessions on the UP. Association allows CP to subsequently use the resources of the UP.

The association setup procedure may be initiated by either CP or UP. Support for UP-initiated association is optional. Either way, CP and UP exchange their support for PFCP optional features during the association procedure.

Either CP or UP may initiate the association update procedure. Only CP may initiate the association release procedure. When UP receives a release request message, it shall delete all PFCP sessions and then delete its association with the CP.

A CP node may be associated with many UP nodes and vice versa. However, a CP-UP pair shall have only one association.

### What's the packet forwarding model in PFCP?

The CP function configures an associated UP function with Packet Detection Rules (PDRs) for each session. Each PDR contains Packet Detection Information (PDI). A PDI can include UE IP address, SDF filter, QFI, Ethernet packet filter, etc. Each PDR also has associated rules that specify how a packet should be treated, that is, if or how should the packet be forwarded, buffered, dropped, etc. No two PDRs shall have the same PDIs.

When a data packet comes into the UP function, first task is to identify the packet's PFCP session using provisioned PDRs. Then UP identifies all matching PDRs of the session. PDR with the highest precedence is selected. Finally, the associated rules of this PDR are applied on the packet.

Packets unmatched by any PDRs are dropped. However, CP may configure UP with a separate session with a PDR containing wildcarded fields. Such a PDR shall have lowest precedence. Such packets may be configured to be dropped or forwarded to CP.

### Which are the rules signalled by PFCP?

A PFCP session is configured with the following:

    + **Packet Detection Rule (PDR)**: PDIs are used to identify the PFCP session.
    + **Forwarding Action Rule (FAR)**: Apply Action IE informs UP function if packets are to be forwarded, duplicated, dropped or buffered.
    + **QoS Enforcement Rule (QER)**: QoS enforcement includes gating control, QoS control, DL flow level marking, service class indicator marking and reflective QoS.
    + **Usage Reporting Rule (URR)**: UP function sends traffic reports to CP function. Reports may be volume-based, time-based or both.
    + **Buffer Action Rule (BAR)**: Specifies buffering parameters for FARs whose Apply Action IEs indicate buffering.
    + **Multi-Access Rule (MAR)**: Applicable for multi-access PDU sessions if UP function supports the Access Traffic Steering, Switching, Splitting (ATSSS) feature.FAR, QER, URR and MAR are association with PDRs of the same PFCP session. Only PDR and FAR are mandatory for a session. These rules are configured during session establishment or modification procedures. However, to reduce signalling load, the UP function may be asked to activate pre-defined PDRs. This is permitted for a UP function that has pre-defined FAR, QER and/or URR.

### How are messages and IEs encoded in PFCP?

PFCP messages and IEs are encoded using **TLV format**. TS 29.244 defines message formats in section 7.2 and IE formats in section 8.1. A PFCP message consists of a PFCP header and zero or more IEs.

The header is of variable length with flags in the first byte signalling the presence/absence of fields. Session-related messages contain a Session Endpoint Identifier (SEID) that's omitted in node-related messages.

Message type is signalled with a single byte. Message length takes up two bytes. The length value excludes the first four bytes (which are mandatory) of the PFCP header.

For each IE, the IE type is signalled with two bytes. IE length takes up two bytes. The length value excludes the first four bytes of the IE. Such an encoding enables forward compatibility. Some IEs are extendable. A legacy receiving entity should ignore extra bytes that it doesn't understand.

Some IEs can contain other IEs. These are called **Grouped IEs**. Messages carry these grouped IEs.

### Are there open source implementations of PFCP?

The Open5GS project has C language-based implementation of PFCP. Open5GS is a well-known project with 1300+ stars on its codebase.

Less well-known implementations include free5GC's PFCP and go-pfcp, both in Go language. A C++ implementation is OpenAirInterface Software Alliance's PFCP. It's associated openair-uml project gives useful class, sequence and activity diagrams.

The Open Mobile Evolved Core (OMEC) project has Go implementations of UPF, SMF and PFCP simulator. The simulator simulates 4G SGW-C or 5G SMF. It can be used for UPF testing.

### Which are the patents concerning PFCP?

We note a few patents pertaining to PFCP (though possibly not standardized by 3GPP):

    + WO2021009166A1 (2021): CP configures UP to enable Network Address Translation (NAT) for one or more Service Data Flows (SDFs). An IE for NAT can be part of FAR.
    + 20210297535 (2021): 5G specifications allow UPF to report only traffic volume to SMF. It's foreseen that data analytics may become more important in 5G (NWDAF). By enhancing Reporting Rule, we can report latency, jitter, QoE, etc.
    + WO/2021/254645 (2021): Layer-specific volume reporting would give operators flexibility to charge based on application payload only or include L3/L4 headers.
    + EP3697171B1 (2020): UP-initiated Association Release procedure could be useful when UP functions are shut down for maintenance, prior to which usage reports need to be sent to CP. Correct reporting is needed for policy control and charging.
    + 20200059992 (2020): Using Deep Packet Inspection (DPI), a UP determines if a PDR associated with a PFCP session context needs to be modified or a new PDR created. UP then initiates this change towards CP.
    + US20210281664A1 (2021): This talks about negotiating and compressing PFCP messages on the N4 interface.

## Milestones

Jul
2015

At the SA WG2 Meeting #S2-110, a study item titled *Feasibility Study on Control and User Plane Separation of EPC nodes* is proposed. This is subsequently approved in September at the 3GPP TSG SA Meeting #69.

Jun
2016

3GPP publishes **TR 23.714** titled *Study on control and user plane separation of EPC nodes* as part of Release 14. The document identifies the different issues and possible solutions. The selected solution is a functional split of user plane and control plane functions, including the case of combined SGW/PGW.

Sep
2016

3GPP publishes **TS 23.214** that specifies the architectural changes to support CUPS in LTE EPC. This is for Release 14. This evolves for other releases: V15.0.0 (Sep 2017), V16.0.0 (Jun 2019) and V17.0.0 (Jun 2021).

Mar
2017

3GPP publishes **TS 29.244**, v1.0.0, titled *Interface between the Control Plane and the User Plane nodes*. This is the main document giving details of PFCP. In June, this is approved for Release 14 as v14.0.0.

May
2017

At the IANA (Internet Assigned Numbers Authority), **port number 8805** is assigned to PFCP over UDP. This is destination port number for a PFCP request message. The source port may be any locally assigned number.

Dec
2017

As part of 5G Release 15, 3GPP publishes PFCP document TS 29.244, v15.0.0.

Jun
2019

As part of 5G Release 16, 3GPP publishes PFCP document TS 29.244, v16.0.0. This update includes Enhanced PFCP Association Release, Deferred PDR activation/deactivation, Activation/Deactivation of pre-defined PDRs, Multi-Access Action Rule, ATSSS support, and more. With v16.1.0 (September 2019), PFCP messages bundling is introduced.

Mar
2021

As part of 5G Release 17, 3GPP publishes PFCP document TS 29.244, v17.0.0. This includes support of Radius, Diameter or DHCP communication via UPF, and partial failure handling over the N4 interface. With v17.1.0 (June 2021) there's support for multi-access PDU sessions and per QoS flow performance measurement.

Mar
2022

3GPP publishes **TR 29.820**, v17.0.0, titled *Study on Best Practice of PFCP*. These best practices lead to better interoperability such as when SMF is deployed centrally at the operator side while UPF is deployed locally at the customer side. Over the N16a interface, I-SMF and SMF have to be interoperable since PFCP IEs are sent over this interface.

Packet Forwarding Control Protocol


# CAP Theorem

## Summary

A well-design cloud-based application often stores its data across multiple servers. For faster response, data is often stored closer to clients in that geography. Due to the distributed nature of this system, it's impossible to design a perfect system. The network may be unreliable or slow at times. Therefore, there are trade-offs to be made. CAP Theorem gives system designers a method to think through and evaluate the trade-offs at the design stage.

The three parts of the CAP Theorem are **Consistency**, **Availability**, and **Partition Tolerance**. The theorem states that it's impossible to guarantee all three in a distributed data store. We can meet any two of them but not all three.

Over the years, designers have misinterpreted the CAP Theorem. To reflect read-world scenarios, modifications to the theorem have been proposed.

## Discussion

### What's the definition of CAP Theorem?

A formal definition of CAP Theorem is, "It is impossible in the asynchronous network model to implement a read/write data object that guarantees the following properties: availability, atomic consistency, in all fair executions (including those in which messages are lost)".

A simplified definition states that,

> In a network subject to communication failures, it is impossible for any web service to implement an atomic read/write shared memory that guarantees a response to every request.

The word "atomic" used above means that although it's a distributed system, requests are modelled as if they are executing on a single node. This gives us an easy model for consistency.

### Could you explain the CAP Theorem?

The parts of the CAP Theorem can be understood as follows:

    + **Consistency**: When a request is made, the server returns the right response. What is "right" depends on the service. For example, reading a value from a database might mean that the most recent write to that value should be returned.
    + **Availability**: A request always receives a response from the server. No constraint is placed on how quickly the response must be received.
    + **Partition Tolerance**: The underlying network is not reliable and servers may get partitioned into non-communicating groups. Despite this, the service should continue to work as desired.As an example, consider two nodes G1 and G2 that have been partitioned. A client changes a value from v0 to v1 on G1. However, the same value is not updated on G2 due to the partition. Hence, when G2 is queried it returns the old value v0. Thus, the service is available but not consistent.

Sometimes the terms *safety* (consistency) and *liveness* (availability) are used in the generalized sense. Safety means "nothing bad ever happens". Liveness means "eventually something good happens".

### What's the implication of the CAP Theorem when designing distributed systems?

When CAP Theorem was proposed, the understanding was that system designers had three options:

    + **CA Systems**: Sacrifice partition tolerance. Single-site or cluster databases using two-phase commit are examples.
    + **CP Systems**: Sacrifice availability. If there's a partition, for consistency we make the service unavailable: return a timeout error or lock operations.
    + **AP Systems**: Sacrifice consistency. If there's a partition, we continue accepting requests but reconcile them later (writes) or return stale values (reads).In practice, we deal with network partitions at least some of the time. The choice is really between consistency and availability. For databases, consistency can be achieved by enabling reads after completing writes on several nodes. Availability can be achieved by replicating data across nodes. In fact, permanent partitions are rare. So the choice is temporary.

Designers don't have to give up one of the three to build a distributed system. In fact, it's possible to have all three under normal network conditions. There's trade-off only when the network is partitioned. It's also helpful to think probabilistically. We can design a CA system if probability of a partition is far less than that of other systemic failures.

### Could you share real-world applications of the CAP Theorem?

Databases that follow ACID (Atomicity, Consistency, Isolation, Durability) give priority to consistency. However, NoSQL distributed databases prefer availability over consistency since availability is often part of commercial service guarantees. So caching and logging were used for *eventual consistency*. This leads to what we call BASE (Basically Available, Soft-state, Eventually consistent). As examples, Zookeeper prefers consistency while Amazon's Dynamo prefers availability.

Maintaining consistency over a wide area network increases latency. Therefore, Yahoo's PNUTS system is inconsistent because it maintains remote copies asynchronously. A particular user's data is partitioned locally and accessed with low latency. Facebook's prefers to update a non-partitioned master copy. User has a more local but potentially stale copy, until it gets updated.

A web browser can go offline if it loses connection to the server. The web app can fall back to on-client persistent storage. Hence, availability is preferred over consistency to sustain long partitions. Likewise, Akamai's web caching offers best effort consistency with high level of availability.

In Google, primary partition usually resides within one datacenter, where both availability and consistency can be maintained. Outside this partition, service becomes unavailable.

### What are some criticisms of the CAP Theorem?

Although the Theorem doesn't specify an upper bound on response time for availability, in practice, there's exists a timeout. CAP Theorem ignores **latency**, which is an important consideration in practice. Timeouts are often implemented in services. During a partition, if we cancel a request, we maintain consistency but forfeit availability. In fact, latency can be seen as another word for availability.

In NoSQL distributed databases, CAP Theorem has led to the belief that eventual consistency provides better availability than strong consistency. Some believe this is an outdated notion. It's better to factor in sensitivity to network delays.

CAP Theorem suggests a binary decision. In reality, it's a continuum. There are different degrees of consistency implemented via "read your writes, monotonic reads and causal consistency".

## Milestones

1985

Originally presented in 1983, researchers Fisher, Lynch and Paterson (FLP) show that distributed consensus is impossible in a fault-tolerant manner in an asynchronous system. Distributed consensus is related to the problem of atomic storage addressed by CAP Theorem.

1996

Before CAP Theorem is formalized, researchers have been working on similar ideas. One example is the paper titled *Trading Consistency for Availability in Distributed Systems* by two researchers at Cornell University.

2000

Eric Brewer presents the CAP Theorem at the 19th Annual ACM Symposium on Principles of Distributed Computing (PODC). It's early history can be traced to 1998 and first published in 1999. Brewer points out that distributed computing has unduly focused on computation and not on data.

2002

MIT researchers Seth Gilbert and Nancy Lynch offer a formal proof of the CAP Theorem in their paper titled *Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services*. In asynchronous systems, the impossibility result is strong. In partially synchronous systems, we can achieve a practical compromise between consistency and availability.

2010

Daniel Abadi proposes the **PACELC Theorem** as an alternative to the CAP Theorem. When data is replicated, there's a trade-off between latency and consistency. PACELC makes this explicit: during partitions (P), trade-off is AC; else, trade-off is LC. Default versions of Dynamo, Cassandra, and Riak are PA/EL systems. VoltDB/H-Store and Megastore are PC/EC. MongoDB is PA/EC.

CAP Theorem


# Microservices

## Summary

Microservice Architecture describes an approach where an application is developed using a collection of loosely coupled services. Previously, applications were based on centralized multi-tier architecture. This worked well in the age of mainframes and desktops. But with cloud computing and mobile devices, backend must be available at all times for a wide range of devices. Bug fixes and features must be delivered quickly without downtime or deploying the entire application.

Microservices are independently deployable, communicating through web APIs or messaging queues to respond to incoming events. They work together to deliver various capabilities, such as user interface frontend, recommendation, logistics, billing, etc.

Microservices are commonly run inside containers. Containers simplify deployment of microservices but microservices can run even without containers.

## Discussion

### What are Microservices?

A microservice is an **autonomous independent service** that encapsulates a business scenario. It **contains the code and state**. Usually a microservice even contains its own data store. This makes it independently versionable, scalable and deployable. A microservice is loosely coupled and **interacts** with other microservices through well-defined interfaces using protocols like http. They remain **consistent and available** in the presence of failure. A microservice is independently releasable. Each microservice can be scaled on its own without having to scale the entire app.

### What are the types of Microservices?

Broadly, there are two types of microservices:

    + **Stateless**: Has either no state or it can be retrieved from an external store (cache/database). It can be scaled out without affecting the state. There can be N instances. Examples: web frontends, protocol gateways, etc. A stateless service is not a cache or a database. It has frequently accessed metadata, no instance affinity and loss of a node is non-evident.
    + **Stateful**: Maintains a hard, authoritative state. For large hyper-scale application the state is kept close to compute. N consistent copies achieved through replication and local persistence. Example: database, documents, workflow, user profile, shopping cart, etc. A stateful service consists of databases and caches and the loss of a node is a notable event. It is sometimes a custom app that holds large amounts of data.As a variation, one author has identified three types: stateless (compute), persistence (storage), aggregation (choreography). Aggregation microservices depend on other microservices, thereby have network and disk I/O dependence.

### What are the types of microservices in a layered architecture?

When we look at microservices as a layered architecture, we can identify the following types:

    + **Core/Atomic services**: Fine-grained self-contained services. No external service dependencies. Mostly business logic. Often no network calls.
    + **Composite/Integration Services**: Business functionality composed from multiple core services. Include business logic and network calls. Implement routing, transformations, orchestration, resiliency and stability patterns. Often interface to legacy or proprietary systems.
    + **API/Edge Services**: A selected set of integration services and core services offered as managed APIs for consumers. Implement routing, API versioning, API security and throttling.

### How are microservices different from APIs?

APIs are not microservices and microservices are not the implementation of an API. API is an interface. A microservice is a component. The term "micro" refers to the component and not the granularity of the exposed interfaces. Microservices can be used to expose one or more APIs. However, not all microservice components expose APIs.

### What's the relationship between Microservices Architecture and Service Oriented Architecture (SOA)?

SOA and Microservice architecture relate to different scopes. While SOA relates to enterprise service exposure, microservice architecture relates to application architecture. Both try to achieve many of the same things (creation of business functions as isolated components) but at a different scale. They differ in maintainability, granularity, agility, etc. SOA is a very broad term and microservices is a subset of that usage. Netflix noted that microservices is "fine-grained SOA". Microservices have been recognized as "SOA done right".

Some microservices principles are really different to SOA:

    + Reuse is not the goal: Reuse of common components is discouraged due to the dependencies it creates. Reuse by copy is preferred.
    + Synchronous is bad: Making synchronous calls such as API or web services creates real-time dependencies. Messaging is used whenever possible between microservices.
    + Service Discovery at run time: Components are assumed to be volatile. So it's often the clients’ responsibility to find and even load balance across instances.
    + Data Duplication is embraced: Techniques such as event sourcing result in multiple independent "views" of the data, ensuring that microservices are truly decoupled.

### What are the important principles to keep in mind while designing a Microservices Architecture?

Broadly speaking the following principles are good to know while designing a microservices architecture: Modelling around a Business Domain, Culture of Automation, Hide Implementation Details, Highly Observable, Decentralise all things, Isolate Failure, Consumer First, Deploy Independently.

### What are the advantages of using a microservices architecture?

Advantages span both development and operations. Briefly we note the following advantages: Build and operate service at scale, Improved resource utilisation to reduce costs, Fault isolation, Continuous Innovation, Small Focused Teams, Use of any language or framework.

Scalability comes because of the modularity that microservices enable. Because of containers, microservices have become easier deploy in various environments. Because of the isolation of services, there's also a security advantage: an attack on one service will not affect others.

Microservices are about code and development. Containers are about deployment. Containers enable microservices. Containers offer isolated environments, thus making them an ideal choice for deploying microservices. However, it's possible to deploy microservices without containers. For example, microservices can be deployed independently on Amazon EC2. Each microservice can be in its own auto-scaling group and scaled independently.

### Why should I probably not adopt microservices for my app?

Since application is distributed across microservices, such distributed systems are harder to manage and monitor. Operational complexity increases as the number microservices and their instances increases, particularly when they are dynamically created and destroyed. Network calls can be slow and even fail at times. Being distributed, maintaining strong consistency is hard and application has to ensure eventual consistency.

Microservices require more time for planning and partitioning the app. They should be designed with failure in mind. When you are building a Minimum Viable Product (MVP) or experimenting to assess what works or can add value to business, then a monolithic approach is faster and easier. Adopt microservices only when you need to scale after your idea and its business value is proven.

If you're managing a legacy application, the effort to migrate to microservices might involve considerable cost. The technology stack may be considerably larger than a monolithic app. Applications have a great dependence on networking and its performance. Microservices can be tested independently but to test the entire application is more difficult.

## Milestones

1970

Although not directly influencing the birth of microservices, it's been noted that from an architectural/philosophical perspective similar ideas existed in the 1970s: Carl Hewitt's Actor Model; or pipes in UNIX that connect many small programs, each of which did something specific rather than having a large complex program that did many things.

1990

This is the decade when **Service-Oriented Architecture (SOA)** starts to gain wider adoption. The term itself is first described by Gartner in 1996. The idea is to use services as isolated and independent building blocks that collectively make up an application. Services are loosely bound together by a standard communications framework.

2000

To enable applications to communicate with one another over a network, W3C standardizes SOAP. Over the next few years, WSDL and UDDI gets standardized as well. XML is the preferred format of communication. Through this decade, **Web Services**, a web-based implementation of SOA, becomes popular over proprietary methods such as CORBA or DCOM.

2009

Netflix redefines the way it develops apps and manages operations by relying completely on APIs. This later develops into what we today call microservices.

May
2011

The term **microservices** gets discussed at a workshop of software architects near Venice, to describe a common architectural style that several of them were exploring at the time.

2012

**REST** and **JSON** become the de facto standard to consume backend data. These would later prove to be foundational for inter-service communication when building microservices-based apps.

Mar
2013

Closely related to microservices, **Docker**, a computer program that performs operating-system-level virtualization (containerization), gets open sourced.

Jun
2014

**Kubernetes**, a container orchestration system for automating deployment, scaling and managing containerized applications, gets open sourced by Google.

Nov
2015

An app developer survey from NGINX shows that microservices are entering mainstream with about two-thirds either using or investigating them. Adoption is higher with medium and small companies.

Microservices


# Promises

## Summary

In asynchronous programming, it's common practice to use callbacks so that the main thread can continue with other processing rather than wait for current function to complete. When that function completes, a relevant callback function is called. Writing and maintaining asynchronous code can be difficult. Promises offer a syntax that enables better code structure and flow.

A promise is an object that's returned immediately when an asynchronous call is made even when such a call has not completed. This object is constructed and returned "with the promise" that its contents will be filled in at a later point when the asynchronous call completes. Formally,

> A promise represents the eventual result of an asynchronous operation.

Promises have become popular since mid-2010s in the world of JavaScript and web development, although promises are not exclusive to JavaScript.

## Discussion

### Why do we need promises?

Asynchronous code often ends up with deeply nested callbacks. Programmers often call this *callback hell* or *pyramid of doom*. This happens because in async code we can't return values because such values are not yet ready. Likewise, we can't throw exceptions because there's no one to catch them.

Promises solve this by flattening the program flow. Because each asynchronous call immediately returns with a promise object, this object can then be used to specify the callback functions for both success and failure cases. In particular, `then` method of a promise object allows us to specify the callbacks. This method also returns another promise object, thus facilitating a chain or composition of promises.

Asynchronous code written with promises is closer in spirit to how we write synchronous code. In synchronous code, we are used to `return`, `throw` and `catch` statements. This functionality was lost in the world of asynchronous callbacks. Essentially,

> The point of promises is to give us back functional composition and error bubbling in the async world.

### What are the essentials of a promise?

A promise is a first-class object, meaning that it can be copied, passed as arguments or returned from functions. Moreover, a promise is returned even before the result of an async call is ready. This allows us to call methods of the object (such as `then` method) while the async call is still in progress.

Callbacks can be specified via the `then` method. Two things are possible when the `then` method is called:

    + If the async call has already completed (or it could even be a synchronous call), the promise object will invoke the relevant callback immediately.
    + If the async call is still pending, the promise object will register the callback but call it later.In either case, `then` method returns a new promise object and this is important to enable chaining. Multiple callbacks can be added by calling `then` multiple times.

Promises also simplify the handling of exceptions. Anytime an exception is thrown it can be handled by a single `catch` method. In fact, the `catch` method is a simplification of `then` method for exception handling.

### What are the states and rules of transition of a promise object?

A promise can be in one of three states:

    + **pending**: This is the initial state.
    + **fulfilled**: This is entered if the execution succeeds. The promise is fulfilled with a *value*.
    + **rejected**: This is entered if execution fails. Rejection comes with a *reason*.A promise that's either fulfilled or rejected is said to be *settled*. This is an apt term since a promise that's settled is an immutable object. It's value (when fulfilled) or reason (when rejected) cannot change. Immutability is important so that consumers of the promise are prevented from introducing side effects.

A promise is said to be *resolved* if it's either settled or "locked in" to the state of another promise. Once resolved, any attempt to resolve it again will have no effect on the promise. An unresolved promise, a promise that's not resolved, is in pending state.

### What are the methods of a promise object?

The Promises/A+ standard specifies `then` method to access the eventual value or reason. For interoperability, no other method needs to be specified.

However, it's common for other standards or implementations to have other methods. For example, ECMAScript 2015 specifies:

    + Methods `all`, `race`, `reject` and `resolve` of `Promise`
    + Methods `then`, `catch` and `finally` of a `Promise.prototype`Among the non-standard methods are `Promise.denodeify`, `Promise.prototype.done`, `Promise.prototype.finally` and `Promise.prototype.nodeify`. GuzzleHttp's Promise implementation in PHP provides methods `otherwise`, `wait`, `getState` and `cancel`. Bluebird adds useful methods such as `map`, `some`, `any`, `filter`, `reduce`, `each` and `props`.

### Could you share some details of the then method?

A promise must have a `then` method that accepts two arguments and returns another promise: `q = p.then(onFulfilled, onRejected)`, where the following hold true for promises `p` and `q`:

    + Arguments `onFulfilled` and `onRejected` are both optional
    + If `onFulfilled` is a function, it's called once when `p` is fulfilled
    + If `onRejected` is a function, it's called once when `p` is rejected
    + Promise `q` is resolved with value `x` if either function `onFulfilled` or `onRejected` returns `x`
    + Promise `q` is rejected with reason `e` if either function `onFulfilled` or `onRejected` throws an exception `e`
    + If `onFulfilled` is not a function, `q` will be fulfilled with the value of `p` when fulfilled
    + If `onRejected` is not a function, `q` will be rejected with the reason of `p` when rejectedFor interoperability, non-promises can be treated as promises if they provide the `then` method, for example, using duck typing. Such an object is called a *thenable*.

### Can you give some use cases where promises might simplify my code?

Promises can enable us to sequentially call a bunch of async functions. We can handle all errors in a single code block if we wish to do so. We can trigger multiple async calls and do further processing only when all of them have completed; or exit if any one of them throws an exception; or handle the first one that completes and ignore the rest. Promises enable us to retry async calls more easily.

### What are some tools for working with promises?

If your browser doesn't support promises, a polyfill will be needed. One option is to use promise-polyfill. Another polyfill is called es6-promise.

Bluebird is a JS library for promises. It can be used in Node.js and browsers. Using "promisification", it can convert old API into promise-aware API. Another useful library is Q. Alternatives include promise, lie, when and RSVP. A comparison of these promise libraries by size and speed is available.

Mocha, Chai and Sinon.JS are three suggested tools for testing asynchronous program flow. Mocha and Chai in combination work nicely for testing promises.

### Can you give some tips for developers coding promises?

Here are some tips, for beginners in particular:

    + It's possible to write promise-based code in the manner of callback-style nested code. This is bad practice. Instead use a promise chain.
    + Always handle errors using either `then(null, onRejected)` or `catch()`.
    + Avoid using the old-style `deferred` pattern that used to be common with jQuery and AngularJS.
    + Always return a value or throw an exception from inside `then` and `catch` methods. This is also handy for converting synchronous code into promisey code.
    + `Promise.resolve()` can wrap errors that occur in synchronous code. Be sure to use a `catch` to handle all errors.
    + Note that `then(onFulfilled).catch(onRejected)` is different from `then(onFulfilled, onRejected)`. The latter code will not catch exceptions that may occur inside `onFulfilled` function.

### I have a bunch of promises I wish to call sequentially. Can I use a for loop?

This is an interesting case where each promise invokes (presumably) some asynchronous code and yet we wish to wait for each asynchronous call to complete before we start the next promise. A promise executes as soon as it's created. If you have a bunch of promises to be called in sequence, any loop construct such as `forEach` will not work. Instead, use a chain of promises, that is, each promise is constructed within the `then` method of the previously fulfilled promise.

## Milestones

1976

The concept of promises is born in the domain of parallel computing. It's called by different names: futures, promises, eventuals.

Jun
1988

MIT researchers present a paper that defines and explains promises. They describe how promises can aid asynchronous programming in distributed systems, including sequences of async calls. It references an async communication mechanism called *call-streams* and illustrates the concepts in Argus programming language.

Mar
2009

Discussion starts within a CommonJS Google Group for specifying an API for promises. An initial version of *Promises/A* is published in February 2010.

Dec
2012

*Promises/A+* specification version 1.0 is released. Versions 1.1 and 1.1.1 are subsequently released in 2013 and 2014 respectively. Promises/A+ is based on the earlier Promises/A but has some omissions, clarifications and additions. Specifically, progress handling and interactive promises are omitted. The focus has been to arrive at a minimal API for interoperability. For interoperability, the specification details the behaviour of the `then` method of a promise object. The specification doesn't talk about how to create, fulfill or reject promises.

Jun
2015

Promise is introduced into ECMA Script 2015, 6th Edition.

Promises


# JavaScript

## Summary

JavaScript (JS) is a programming language and a core technology of the World Wide Web (WWW). It complements HTML and CSS. HTML provides structured static content on the web. CSS does the styling on the content including animations. JavaScript enables rich interactions and dynamic content.

JS started for the web and executed within web browsers. With the coming of Node.js in 2009, it became possible to use JS for server-side execution. It's now possible to build entire web apps with JS for both frontend and backend code. Today, the JS ecosystem is rich and mature with many libraries, frameworks, package managers, bundlers, transpilers, runtimes, engines and IDEs.

Technically, JavaScript is a loosely typed, multi-paradigm, interpreted high-level language that conforms to the **ECMAScript** specifications.

## Discussion

### What's unique about JavaScript that one would want to learn it?

JavaScript is the language of web. It's one of the three core technologies of the World Wide Web along with HTML and CSS. JS enables dynamic effects and interactions on webpages. Hence it's become an essential part of web applications. Static web pages are almost obsolete nowadays.

With the advent of NodeJS, JS was quickly adopted for server-side scripting. JS can be used for both frontend (client-side) and backend (server-side) programming of web apps. A developer therefore needs to learn only one language. This is particularly important when there's a demand for fullstack (frontend + backend) developers.

JavaScript is no longer limited to just client-side code. It's being used for both client-side and server-side code, in mobile devices, for IoT applications, desktop applications and more.

The importance of JavaScript is also highlighted by this quote from Eric Elliott,

> Software is eating the world, the web is eating software, and JavaScript rules the web.

### How does JavaScript work on the web?

JavaScript code can be embedded into HTML files or stored in separate `*.js` files that can be referenced from HTML files. When browsers encounter the JS code, they will parse it and execute it on the fly. JavaScript is an **interpreted language**, meaning that it's not compiled in advance to machine code.

By mid-2000s, **AJAX** became a common approach to use JS more effectively for richer user experience. With AJAX, user interactivity is continued while new content is fetched asynchronously in the background.

Traditionally, content on the web was served as individual HTML pages and each page had some interactivity enabled via JS and AJAX. In the late 2010s, there was a shift towards building the entire app in JavaScript and updating the contents via AJAX. Such an app is called a **Single Page Application (SPA)**.

Since we can now run JS on both client and server, this gives us the flexibility to render part of the web page on server and let the client complete the rest. This approach leads to what we call **Isomorphic or Universal Application**.

### Is JavaScript related to Java, VBScript, JScript and ECMAScript?

Java and JavaScript are two different programming languages. The name *JavaScript* itself was coined in 1995 because of a partnership between Netscape Communications (that invented JavaScript) and Sun Microsystems (that invented Java). Java syntax was introduced into JavaScript. While Java was reserved for enterprise apps, JavaScript was positioned as a web companion to Java. About the same time, Java was also made available within web browsers as **applets**. Today, Java and JavaScript are both popular languages.

When Microsoft got into web technology in the 1990s, its Internet Explorer (IE) browser needed an equivalent to what Netscape and Sun had with JavaScript and Java. JScript and VBScript are therefore Microsoft scripting languages. JScript is said to be very similar to JavaScript and enabled dynamic content on IE.

This fragmentation meant that a web page written for Netscape would not work well on IE, and vice versa. There was no standard and JavaScript was evolving far too quickly. In this context, **ECMAScript** was born in 1997 as a standard for JavaScript.

### Why is JavaScript called a multi-paradigm language?

JavaScript is an **object-oriented** language. It has objects, with properties and methods. Functions are first-class objects that can be passed into other functions, returned from functions, bound to variables or even thrown as exceptions. However, object inheritance is not done in the classical manner of C++ or Java. It uses **prototypal inheritance**, inspired by Self language. This makes the language flexible. A prototype can be cloned and modified to make a new prototype without affecting all child instances. It's also possible to do **classical inheritance** in JavaScript.

Though being object-oriented, there are no classes in JavaScript but constructors are available. Object systems can be built using either inheritance or aggregation. Variables and methods can be private, public or privileged to the object.

Eric Elliott explains why he regards *prototypal inheritance* and *functional programming* as two pillars of JavaScript. *Closures* provide encapsulation and the means to avoid side effects. Thus, understanding closures is important for **functional programming** in JavaScript. JavaScript can also be used in an **imperative style** using `if-else` conditions, `for` loops, and module-level variables.

Thus, JavaScript is multi-paradigm because it's object-oriented, functional and imperative.

### What are some important features of JavaScript that I should learn?

Beginners should first learn the basics of JS, in addition of HTML and CSS. The Modern JavaScript Tutorial is great place to start learning.

Some useful things to learn include effective console logging, destructuring, template literals, and spread syntax. Explicit loops can be avoided by taking the functional programming approach with `reduce`, `map` and `filter` methods. For asynchronous programming, you should learn about Promises and adopt the modern `async/await` syntax.

Get a good understanding of closures and partial application. Use the arrow syntax of ES6 for more compact and readable code. Other useful ES6 features to learn are `const`, multiline strings, default parameters, and module import/export.

### What's the difference between a JS engine and a JS runtime?

A **JavaScript Engine** parses JS code and converts it to a form that the machine can execute. It contains the memory heap and the call stack. Examples include Google's V8 Engine (Chrome), SpiderMonkey (Firefox), JScript (IE), and Chakra (Microsoft Edge).

A **JavaScript Runtime** is an environment in which JS runs. Web browsers are environments. Node.js provides a runtime for server-side execution. In a typical web app, the program may access the DOM, make AJAX calls, and call Web APIs. These are not part of the language. They are provided by the browser as part of the JS runtime.

Another important distinction is that the engine does **synchronous** processing. It can process only one function at a time via the call stack. On the other hand, the runtime can maintain a number of items in the callback queue. A new item can be added anytime to the queue and processed later when the call stack is free. Thus, the runtime enables **asynchronous** processing.

### As a beginner, what should I know about the JavaScript ecosystem?

Consider the following:

    + **Frameworks**: These simplify development by providing a structure, including design patterns such as MVC. Examples include React, Angular, Vue, Backbone and Ember.
    + **Libraries**: Many developers release their code as JS packages/modules/libraries so that others can reuse them. In December 2018, about 836,000 libraries were available on NPM.
    + **Package Managers**: Developers use them to install and manage packages they require for their projects. Examples include NPM and Yarn. Packages are downloaded from known repositories on the web.
    + **Linters**: These catch errors or bad practices early. Examples include Prettier and ESLint.
    + **Module Bundlers**: A project may have many dependent JS packages. Bundlers combine them into far fewer files so that these can be delivered more efficiently to browsers and other runtimes. Examples include Webpack and Browserify.
    + **Task Runners**: These enable automated development workflows such as minifying or concatenating files. Examples include Gulp and Grunt.
    + **Transpilers**: Developers can write code in other languages (CoffeeScript, TypeScript) and then convert them into JavaScript. Another use case is to write in modern ES6 syntax and transpile to syntax supported by older browsers. Examples include Babel.

### What are some concerns that developers have about JavaScript?

Developers coming from languages such as C++ and Java, find JavaScript annoying in many ways: automatic semicolon insertion, automatic type coercion (loose typing), lack of block scoping, lack of classes, unusual (prototypal) inheritance. Another quirk is type mismatches between objects that are otherwise similar: `"Hello world"` of type "string" versus `new String("Hello world")` of type "object". Some of these are attributed to the hurried development of JavaScript.

Because there are many JS frameworks, libraries and tools, the choice of which one to adopt can be daunting. In one example, it was shown that Vue.js was hardcoded to rely on `yarn` and this broke the workflow because the developer had only installed `npm`. It's also possible in the JS world to have a small project of just few lines of code download lots of dependencies. An average web app has over 1000 modules.

Because the language was designed in a hurry, it has design errors. Developers can write bad JS code. This prompted Douglas Crockford to write his now famous book titled *JavaScript: The Good Parts*.

## Milestones

1993

In the early days, the history of JavaScript is somewhat tied to the history of web browsers. Among the early browsers are WorldWideWeb, ViolaWWW, Erwise and Midas. In 1993, **NCSA Mosaic** appears on the scene. It supports multiple networking protocols and is able to display images inline. It becomes popular and people begin to see the potential of the web. There's no JavaScript at this point and content shown on browsers are all static.

1994

Marc Andreessen, one of the creators of Mosaic, has the vision that the web should be lot more dynamic with animations and interactions. He starts **Netscape Communications**.

May
1995

At Netscape, the idea is to invent a dynamic scripting language for the browser that would have simple syntax and appeal to non-programmers. Brendan Eich is contracted by Netscape to develop "Scheme for the browser", Scheme being a language with simple syntax, dynamic and powerful. Thus is born **Mocha**, which becomes part of Netscape Communicator browser in May 1995.

Nov
1995

With competition from Microsoft and its Internet Explorer, Netscape partners with Sun Microsystems to make Java available in the browser. At the same time, they recognize that Java is a beast not suited for the browser. Together they announce **JavaScript**, which they describe as "an open, cross-platform object-scripting language designed for creating and customizing applications on the Net." To ride on the hype surrounding Java, JavaScript is positioned as a companion to Java, even based on Java. JS becomes available in beta version of Netscape Navigator 2.0. Mocha becomes JS but less Scheme-like and more Java-like.

1996

Microsoft introduces **JScript**, it's alternative to JavaScript. It's included in Internet Explorer 3.0. JScript can also be used for server-side scripting in Internet Information Services (IIS) web server. Server-side scripting is also possible with JavaScript in Netscape Enterprise Server. However, only in 2009 does Node.js make server-side scripting with JS popular.

Jun
1997

ECMA organization publishes **ECMAScript**, documented as the ECMA-262 standard. Web pages that conform to ECMAScript are guaranteed to work on all browsers, provided browsers support the standard. JavaScript conforms to ECMA, while providing additional features. ActionScript and JScript are other well-known implementations of ECMAScript.

2005

Jesse James Garrett of Adaptive Path mentions a new term, **AJAX**, which expands to *Asynchronous JavaScript + XML*. AJAX eliminates the need to reload an entire webpage to update small details. It gives developers more flexibility to make their web apps more responsive and interactive. Many projects in Google already use AJAX. AJAX goes on to change the way web interactions are built with JS.

2009

As a standard library, **CommonJS** is founded, mainly for JS development outside the browser. This include the use of JS for web server, desktop and command line applications. This is also the year when **NodeJS** is released so that JS can be used for server-side scripting.

Jun
2015

As a standard, **ECMAScript 2015** (ES2015 or ES6) is released. ES6 is seen as a major update to the core language specifications, which hadn't changed since ES5 of 2009. ES6 is expected to make JS coding easier, although it may take a while for browsers to support it fully.

2017

JavaScript has highest number of pull requests on GitHub, an open site for sharing code, particularly on open source projects. A pull request is a way for developers to submit their contributions to a project.

2018

With more than 9.7 million developers using JavaScript worldwide, JS is declared as the most popular programming language. This is based on a survey of 21,700 developers from 169 countries. JS has 2.4 million more developers than the next popular language. However, some predict that JS may decline due to **WebAssembly**. For interactions on the web, JS has been the only language of choice so far. With WebAssembly, developers can potentially migrate to other languages.

Mar
2018

TensorFlow project releases **TensorFlow.js** to train and run Machine Learning (ML) models within a web browser. This becomes an easy path for JS developers to get started with ML. This is just one example to illustrate the versatility of JS.

JavaScript


# 5G Core RESTful APIs

## Summary

The 5G Core separates the control plane from the user plane. The control plane is a designed as a set of Network Functions (NFs). Each NF exposes one or more services. Each service offers one or more operations, implemented as RESTful APIs over HTTPS. For this reason, NFs are said to use Service-Based Interfaces (SBIs).

Disaggregating the 5G Core as independent (but interworking) NFs gives Communication Service Providers (CSPs) the ability to mix and match NFs from various software vendors. Benefits include automation, cost reduction, customization, and less vendor lock-in.

3GPP has defined these APIs in conformance with the OpenAPI Specification. APIs have been published as YAML files. Using OpenAPI tools, developers can autogenerate client/server code from these YAML files. Such autogenerated code can become a starting point for implementing 5G Core.

## Discussion

### Could you give an overview of 5G Core NFs?

There are dozens of NFs (and their services) for which APIs have been defined. The figure highlights that Service-Based Architecture (SBA) and Service-Based Interfaces (SBIs) are defined only for the control plane. SBA is based on web technologies. Services communicate via HTTP/2 over TLS over TCP/IP, and exchange data in JSON format.

Non-SBA interfaces use telecom-specific protocols: NAS-MM (on N1), NGAP over SCTP (on N2), GTP-U over UDP (on N3), PFCP over UDP (on N4), and so on.

An NF Service Producer exposes its services that are then invoked by an authorized NF Service Consumer. Producers and consumers can communicate directly or indirectly via Service Communication Proxy (SCP). A consumer can discover a producer via local configuration, via NRF (direct communication) or via SCP (indirect communication).

Each service is self-contained, reusable and independent of another service, even if they're part of the same NF. Services may share some context data. A system procedure is basically invoking a sequence of NF services.

### Where can I obtain the 5G Core APIs?

A descriptive overview of NFs is in section 6 of **TS 23.501**. Section 7 lists the services of each NF. In Release 17 (Oct 2023), there are a total of 28 NFs containing 102 services.   These are listed in the figure.

Section 5 of **TS 23.502** details all operations of each service. An operation can take the form of request/response or subscribe/notify. The document also gives the known consumers of each operation. Section 4 describes system procedures that make use of these operations.

Stage 3 specifications (including API definitions in YAML) of each NF is the **TS 29.5xx series** of documents. An easier way to obtain the API definitions is via the 5G APIs repository at 3GPP Forge. The README file in this repository includes links to view/edit/validate/test each API via Swagger Editor UI.

### Which are the REST principles adopted by 5G SBA?

Code on Demand and HATEOAS are two REST principles currently not used in 5G SBA. Otherwise, the following have been adopted:

    + **Resource, Serialization and Representation**: Resources/objects are accessed via unique URIs. They're serialized as JSON documents.
    + **Client/Server**: Clients are service consumers while servers are service producers. The same NF can take on different roles in different contexts. For example, when registering with NRF, SMF is the client and NRF is the server. When creating a session, SMF is the server and AMF is the client.
    + **Stateless**: Servers don't maintain any state about clients. Clients include all necessary information with each request. This allows for load balancing, scalability and resilience in a distributed environment. UDSF offers a centralized storage and thereby facilitates stateless implementations. If an NF instance goes down, another instance can obtain context data (such as Service Profile used at SMF) from UDSF. UDR and UDSF can't be stateless.
    + **Cacheable**: Servers indicate if clients can cache the information.
    + **Layered System**: Clients need not be aware exactly which server provides a service. NRF allows this decoupling of clients and servers.
    + **HTTP Methods**: GET, POST, PUT, PATCH and DELETE are used.

### What communication types does 5GC SBA adopt?

For CRUD operations, **Request/Response** communication is used. An NF consumer requests. An NF producer responds. POST method is used on a parent resource to create a new child resource. The latter's URI is returned in the response. Another way to create a resource is to use PUT method on the resource itself. In this case, the NF consumer selects the resource identifier and URI. Resource can be read with GET method accompanied with necessary query parameters. Resource updates can be done with PUT (full update) or PATCH (partial update). Resource deletion is via DELETE method.

There's also **Subscribe/Notify** communication. An NF consumer subscribes by providing a callback URI and an optional filter. An NF producer notifies the consumer when data changes and matches the filter. Subscriptions are created with POST, updated with PUT or PATCH, and deleted with DELETE. Notifications use POST. Subscriptions can be of explicit or implicit. For example, UDR is implicitly subscribed to provisioned subscriber data at the UDR. Another example is an AMF instance that creates an SM Context for a PDU Session. That instance is implicitly subscribed to SM Context Status at the SMF.

### What's the structure of URIs used by 5GC SBA?

The figure shows the structure and an example of the `nf-instances` resource from NRF NFManagement API. API Root, API Name and API Version are collectively called API URI. API URI doesn't include a trailing forward slash. Variable `nfInstanceID` is delimited with `{}`. In the example, this is replaced with UUID to indicate the specific instance.

API Version follows semantic versioning of the form `vx.y.z` where `x`, `y` and `z` are positive integers. Only the major version (`x` part) is indicated in the URI. A suffix starting with hyphen indicates pre-release version (eg. `1.0.0-alpha1`). A suffix starting with plus sign indicates build metadata after release freeze (eg. `3.0.1+orange.2020-09`). An NF consumer can use NRF to discover supported versions of NF producer instances.

### What are some challenges in implementing 5GC?

Multi-vendor deployments are complex with respect to interoperability, provisioning, dependencies, Operations Support Systems (OSSs), etc. AI-driven approaches may help avoid manual work. Engineers need to skill themselves in system integration. Operators remain undecided if they should get into system integration themselves or adopt a pre-packaged solution from a consortium.

3GPP permits custom operations using POST method. It also permits custom RPC-style API operations. However, these will adversely impact interoperability.

A service mesh such as Istio isn't aligned with the 3GPP approach of using NRF for service discovery or UPF selection by SMF. However, 3GPP does suggest using a service mesh for SCP.

OpenAPI is meant to simplifying implementation due to readily available validators, code generators and other tools. The open source OpenAPI Generator is able to generate code in many languages. When run against 5G specifications, it shows its limitations. As on December 2023, the project has close to 4,000 open issues.

## Milestones

2000

Fielding describes **Representational State Transfer (REST)** in his PhD dissertation. He proposes REST as a guide to enhancing HTTP, not building web APIs per se. He notes that any architectural style (such as REST) is not a silver bullet to solve all types of problems. REST itself is meant for "large-grain hypermedia data transfer".

Jun
2018

3GPP publishes **TS 29.501, v15.0.0**, titled *Principles and Guidelines for Services Definition; Stage 3*. In v15.1.0 (September 2018), security requirements of API design and JSON structures in query parameters are specified. Subsequent versions include v16.0.0 (June 2019) and v17.0.0 (December 2020).

Aug
2020

5G Core APIs specified in YAML files are hosted at 3GPP Forge. While this is a GitLab environment, GitLab tools and processes can't fully be used. This is because changes can happen only via 3GPP mandated Change Requests (CRs).

May
2023

At a **multi-vendor trial**, Enea demonstrates fast network slicing. A network slice can be instantiated quickly. Software comes from Enea, Oracle and Casa Systems. Enea supplied UDM, UDR and AUSF. Hardware included those from HPE, Intel and Nokia. However, achieving this in a commercial network is not expected to be straightforward.

5G Core RESTful APIs


# Cryptography

## Summary

**Cryptography** is a set of techniques for encrypting data using specified algorithms that make the data unreadable to the third party computer systems, unless decrypted using predefined procedures by the sender. Messages between the sender and receiver are passing through a medium, which may be attacked and the information can be stolen. So the sender encrypts and the receiver decrypts. In the presence of third parties, cryptography is the practise of secure communication. **Data encryption** is known for its ability to keep information safe from prying eyes. It uses an encryption key to convert data from one format, called plaintext, to another, called ciphertext. Modern cryptography is largely based on computer science's mathematical theory and practise.

## Discussion

### What is the purpose of cryptography?

A system's ability to verify the sender's identity is **Authentication**. The sender and recipient can verify each other's identities as well as the information's origin and destination.

Information transmitted should only be accessed by legal parties and not by anyone else for the purpose of **Confidentiality**. Anyone who was not intended to get the information will be unable to comprehend it.

Only authorised parties are allowed to make changes to transmitted data to maintain **Integrity**. Information cannot be manipulated with while in storage or in transmission between the sender and the intended recipient without being detected.

**Non-repudiation** is the guarantee that no one can refute something's legitimacy. The information creator/sender cannot later contradict their intentions in the development or transmission of the material.

The information provided is only accessible to those who have been given permission. This gives **Access Control** only to the involved parties.

### What are some well-known applications of cryptography?

**Protected Communication:** Encrypting communications between ourselves and another system is the most common application of cryptography, and one we all use on a regular basis. A web browser and a web server are two examples, as are an email client and an email server. Modern switching networks make interception more difficult.

**Storage Integrity:** The emergence of computer viruses has led to the adoption of cryptographic checksums for data storage. A checksum is generated and compared to expectations, just as it is for transmission integrity. Transmitted information is often available for a shorter amount of time and is used for a smaller volume of data, and is retrieved at a slower rate than stored data.

**Electronic Money:** Today, there are patents in existence around the world that allow electronic information to substitute cash money in individual financial transactions. Cryptography is used in this system to keep national assets in electronic form.

### Which are some commonly used terms in cryptography?

**Plaintext** refers to any communication or data that has to be protected for various reasons.

**Ciphertext** is the unreadable form of data that is generated at the end of the encryption process.

**Encryption** refers to the process of encoding a message with the use of a key. The basic text is turned into illegible text in this way.

**Decryption** is the process of deciphering an encoded communication with the use of a key. It's the inverse of the encryption method.

A **key** is a parameter that dictates what a cryptographic process's final output will be. The length of the key is important in the encryption process.

### What techniques can be used for encrypting/decrypting data?

When **Symmetric Encryption** is performed, the same cryptographic keys are utilised for plaintext crypting and degradation of the figure materials. Symmetrical key encryption is less complicated. One key is used to encrypt and decrypt both data sets. Types of symmetric keys: Stream or Block cyphers, can be used in symmetric-key encryption. Stream cyphers encrypt the digits of a message one by one (typically bytes). To modify the component's measurement, Block figures employ distinct sections and encrypt them with the plaintext as a single component unit.

**Asymmetric Encryption** is a collection of keys that are used to encrypt and decrypt public and private-key information.

Because the user utilises two keys as asymmetrical encoding employs two keys, it's also known as the Cryptography Public Key.

In **Encryption with public keys** the messages are encrypted using the recipient's public key. The post cannot be deciphered by anyone who is not the private coordinator, does not own the key, or is not connected to the general public key.

**Digital Signature** uses a personalised transmitter key that can be verified by anyone with a personal key, ensuring network security.

### Could you share examples of cryptographic algorithms?

**Data Encryption Standard (DES)** developed by IBM in the 1970s, approved for commercial usage by the National Bureau of Standards(NBS) in 1977 uses a 56-bit key and 8 rounds to work on 64-bit blocks.

**Advanced Encryption Standard (AES)** is a fast and safe algorithm released in 1998, created by Vincent Rijmen and Joan Daemen works with variable key and block lengths. The key length can be 128, 192, or 256 bits, and the block length can be 128, 192, or 256 bits.

**Rivest Cipher (RC)** was created by Ronald Rivest, and named after him. RC1, RC2, RC3, RC4, RC5, and RC6 are available.

Bruce Schneie created **Blowfish** in 1993 and published it in 1994. It contains 8 rounds, with a 64-bit block size and a key length of 32 to 448 bits. Blowfish is considered as a replacement for DES as it is substantially faster than others with a good key strength.

**RSA** was named after it's inventors Ron Rivest, Adi Shamir and Leonard Adleman, in 1997. A variable-size key and encryption block are employed in it. It provides increased security and convenience, and uses Public Key Encryption.

### How do I evaluate cryptographic algorithms?

Each encryption algorithm has strengths and disadvantages, that affect encryption performance based on these parameters:

**Encryption time** is measured in milliseconds and depends on the data block's length and key. When the encryption time is short, an algorithm's performance is considered advanced.

**Decryption time** is the amount of time it takes to recover the original text from ciphertext, measured in milliseconds. When the decryption time is short, an algorithm's performance is considered superior.

**Memory used** should be minimal because it has an impact on system costs.

**Throughput** is calculated by dividing the total encoded block size by the whole encode time. If the throughput cost rises, the algorithm's power consumption will fall.

**Avalanche effect** predicts that if the plaintext changes, the ciphertext will also change dramatically, by calculating the difference between plaintext and ciphertext modifications.

**Entropy** is a statistical measure of data randomness and uncertainty..

**The number of bits required for encoding optimally** defines the bandwidth required for transmission. When an encrypted character or bit is encoded with fewer bits, it uses less storage and bandwidth, directly impacting the system's cost.

### What is the role of computational and energy costs in implementing Cryptography?

Networks are developing towards a ubiquitous model in which heterogeneous devices are networked (offering easy network interconnections anytime and anyplace). Cryptographic algorithms are necessary for the network security solutions' development. However, network devices' computational and energy restrictions make real implementation of such processes difficult. As a result, a thorough examination of the costs of launching symmetric and asymmetric cryptographic algorithms, hash chain functions, elliptic curve cryptography, and pairing-based cryptography on personal agendas is carried out, and the results are compared to the costs of basic operating system functions. The studies reveal that, while cryptographic power costs are considerable and such operations must be time limited, they are not the primary limiting factor in a device's autonomy.

The technological advancement has resulted in the expansion of personal portable computers and the emergence of new forms of networks. Security solutions are required to secure heterogeneous ubiquitous networks generated by small and restricted devices, which are being explored or are already in use. The inherent nature of ubiquitous networks requires the safeguards' implementation to assure proper protocol execution at all layers, from networking operations to collaborative enforcement protocols or privacy-protecting mechanisms.

## Milestones

1932

Polish cryptographer Marian Rejewski discovers how Enigma works.

1939

Poland shares the information on how Enigma works with the French and British intelligence services, allowing cryptographers like Alan Turing to figure out how to crack the key, which changes daily. It proves crucial to the Allies' World War II victory.

1945

Claude E. Shannon of Bell Labs publishes an article called "A mathematical theory of cryptography." It's the starting point of modern cryptography. For centuries, governments have controlled secret codes: applied to diplomacy, employed in wars, and used in espionage. But with modern technologies, the use of codes by individuals is exploding.

1976

Whitfield Diffie and Martin Hellman publish a research paper, New Directions in Cryptography, on what would be defined as the Diffie-Hellman key exchange. For the first time, the code key is no longer pre-arranged, but a pair of keys (one public, one private but mathematically linked) is dynamically created for every correspondent.

1977

RSA public key encryption is invented.

1978

Robert McEliece invents the McEliece cryptosystem, the first asymmetric encryption algorithm to use randomization in the encryption process.

2000

The Advanced Encryption Standard replaces DES, or AES (asymmetric key - the user and sender must know the same secret key), found through a competition open to the public. Today, AES is available royalty-free worldwide and is approved for use in classified US government information. PKI (Public Key Infrastructure) is a generic term used to define solutions creating and managing public-key encryption. It is activated by browsers for the Internet but also by public and private organizations to secure communications.

2001

Belgian Rijndael algorithm selected as the U.S. Advanced Encryption Standard (AES) after a five-year public search process by National Institute of Standards and Technology (NIST).

2004

The first commercial quantum cryptography system becomes available from id Quantique.

2005

Elliptic-curve cryptography (ECC) is an advanced public-key cryptography scheme and allows shorter encryption keys. Elliptic curve cryptosystems are more challenging to break than RSA and Diffie-Hellman.

2007

Users swamp Digg.com with copies of a 128-bit key to the AACS system used to protect HD DVD and Blu-ray video discs. The user revolt is a response to Digg's decision, subsequently reversed, to remove the keys, per demands from the motion picture industry that cited the U.S. DMCA anti-circumvention provisions. NIST hash function competition announced.

2013

Edward Snowden discloses a vast trove of classified documents from NSA. Dual\_EC\_DRBG is discovered to have a NSA backdoor. NSA publishes Simon and Speck lightweight block ciphers.

Cryptography


# Xamarin

## Summary

Xamarin is a cross-platform application development framework. It allows you to develop a mobile app using C# and reuse most of the codebase across multiple platforms including Android, iOS and Windows. Among the advantages of Xamarin are native user interfaces, native API access, native performance and developer productivity.

Xamarin itself leverages .NET development platform that comes with C# language, its compilers, base libraries, editors and tools. Xamarin extends .NET so that developers can build mobile apps and target multiple platforms.

## Discussion

### When should I use Xamarin?

There are many hybrid and cross-platform frameworks to develop apps across multiple platforms, but these require skills in JavaScript and are best suited for Web developers. For them, Apache Cordova is one possible framework but do keep in mind that this is a hybrid approach that will not give native performance.

Where performance is desired along with faster time to market via multiple mobile platforms, Xamarin must be preferred. Xamarin is best suited for developers coming from .NET, C# or Java backgrounds.

As a note, Xamarin should not be confused with **.NET Core**, though both are cross platform and open source. Xamarin is for cross-platform mobile (though it can be used for MacOS) whereas .NET Core is for creating cross-platform web apps, microservices, libraries and console apps that can run on Windows, Linus or MacOS.

### Why should I use Xamarin?

Xamarin helps to expedite native mobile app development targeting multiple platforms. It's been said that for informational apps, 85% of code can be reused across Android and iOS. For more intensive apps, code reuse of 50% is possible. This code reuse comes from using shared C# app logic across platforms.

User interface code itself is not shared by Xamarin but that's where **Xamarin.Forms** comes in. Using **Xamarin.Forms** 96% code reuse has been reported.

For developers, this means that you get to build native Android, iOS and Windows Phone apps concurrently without having to build them one after another or having multiple teams with multiple skillsets and tools.

**Xamarin.Android** and **Xamarin.iOS** provide further customization possibilities for developers who are looking at tweaking the app's look and feel achieved with Xamarin.Forms. There are several other benefits like seamless API integration, easy collaboration and sharing, etc.

### Could you describe some useful Xamarin-specific terms that developers should know?

The term **managed code** refers to code that's managed by .NET Framework Common Language Runtime. In the case of Xamarin, this is the Mono Runtime. In contrast, **native code** is code that runs natively on the platform. Managed code could be in C# or F#. Java/Android code are native to Android; or Objective-C code is native to iOS/MAC.

During compilation, C# or F# code is converted to **Microsoft Intermediate Language (MSIL)**. In Android and Mac platforms, MSIL is compiled to native at runtime using **just-in-time (JIT)** compilation. In iOS, due to security restrictions, this is not allowed. Hence **ahead-of-time (AOT)** compilation is performed.

### What's the architecture of Xamarin?

Architecture is described in the documentation in three places:

    + Xamarin.Android Architecture
    + Xamarin.Mac Architecture
    + Xamarin.iOS ArchitectureIn general, C# code and .NET APIs sit on top of Mono runtime, which in turn sits on top of the OS kernel. Architecture allows for managed code to invoke native APIs via bindings. In Android, **Managed Callable Wrappers (MCW)** are used when managed code needs to invoke Android code; **Android Callable Wrappers (ACW)** are used when Android runtime needs to invoke managed code. There's also the **Java Native Interface (JNI)** for one-off use of unbound Java types and members.

### How much does it cost to use Xamarin?

Xamarin was initially available for a license. After it was acquired by Microsoft in 2016, it's now bundled with the Visual Studio suite of tools for free. While Visual Studio is not completely free, there is a Community Edition, which is free for eligible companies and developers.

The .NET platform itself open source and Xamarin in part of it. This means there are no fees or licensing costs, even for commercial use.

### What are the pre-requisites for Xamarin app development?

To develop the app, one should have good programming expertise in C#, assuming that the RESTful APIs required for the application are already available. A Windows PC will be required for development on Android and Windows platforms. iOS development is done on Windows PC but to build the app, a Mac will be required to be connected in the same network as per Apple's requirements.

### How does Xamarin compare against other cross-platform frameworks?

Among the cross-platform mobile frameworks are React Native, Ionic 2, Flutter, NativeScript, and Xamarin. Ionic 2, React Native and NativeScript rely on web technologies (HTML/CSS/JavaScript/TypeScript). Ionic 2 can suffer from performance problems. While Ionic 2 uses WebView (basically HTML wrapped in a native app), React Native and NativeScript use native UI components. Ionic 2 can also allow access to native API but will require the use of Apache Cordova. Xamarin uses native UI components and offers good performance.

For development, Xamarin lacks automatic restarting and hot/cold swapping. In this aspect, React Native is one of the easiest for developer productivity. All frameworks allow native code bindings but this process is easiest on Xamarin. Xamarin offers full one-to-one bindings to native libraries whereas in React Native or Ionic, support is partial and done via an abstraction layer.

A blog post from AltexSoft compares the performance of Xamarin apps against native apps. It concludes that performance drop due to Xamarin is acceptable for most cases but the use of Xamarin.Forms is not recommended when apps are CPU intensive.

### What are some criticisms of Xamarin?

It's been said that Xamarin support for latest platform updates (iOS/Android) are usually delayed. While iOS and Android developers can tap into an active ecosystem plus use open source technologies, similar following is limited in Xamarin. Xamarin is also not suited for apps that are heavy for graphics, where code reuse will be limited. For that matter, Xamarin developers still need to learn platform-specific languages for the UI, unless they also adopt Xamarin.Forms for their apps.

Xamarin apps are also larger in size when compared to equivalent native apps. This is because the app has to include the Mono runtime and associated components. A simple "hello world" Android app written in Xamarin requires about 16 MB.

### Where can I learn more about Xamarin?

Xamarin has excellent documentation and code samples at their website. Specific official resources from Microsoft include:

    + Documentation
    + Courses, tutorials and videos
    + Community forums
    + The Xamarin Show on MSDN
    + Xamarin tutorials on Microsoft Learn
    + Visual Studio tools for XamarinThere used to be Xamarin University to train and certify professional developers. Since June 2019, this is now part of Microsoft Learn platform.

## Milestones

1999

Miguel de Icaza and Nat Friedman start **Helix Code** for GNOME desktop application development for the Linux platform. In 2001, this is renamed to **Ximian**, which is acquired in 2003 by Novell.

2000

Microsoft releases .NET Framework and Visual Studio .NET along with introducing a new language named C#.

2001

**Mono** project is launched for supporting .NET applications on non-Windows platforms. Open source and based on .NET Framework, Mono enables developers to build cross-platform applications.

May
2011

**Xamarin** as a company is incorporated with the idea of building commercial .NET offerings for iOS and Android. In July, Novell gives Xamarin the rights to use MonoTouch and Mono for Android.

2016

Microsoft acquires Xamarin and releases Xamarin as part of Visual Studio suite of tools.

Xamarin


# Natural Language Toolkit

## Summary

Natural Language Toolkit (NLTK) is a Python package to perform natural language processing (NLP). It was created mainly as a tool for learning NLP via a hands-on approach. It was not designed to be used in production.

The growth of unstructured data via social media, online reviews, blogs, and voice-based human-computer interaction are some reasons why NLP has become important in the late 2010s. NLTK is a useful toolkit for many of these NLP applications.

NLTK is composed of sub-packages and modules. A typical processing pipeline will call modules in sequence. Python data structures are passed from one module to another. Beyond the algorithms, NLTK gives quick access to many text corpora and datasets.

## Discussion

### Which are the fundamental NLP tasks that can be performed using NLTK?

NLTK can be used in wide range of applications for NLP. For basic understanding, let's try to analyze a paragraph using NLTK. It can be pre-processed using sentence segmentation, removing stopwords, removing punctuation and special symbols, and word tokenization. After pre-processing the corpus, it can be analyzed sentence-wise using parts of speech (POS) to extract nouns and adjectives. Subsequent tasks can include named entity recognition (NER), coreference resolution, constituency parsing and dependency parsing. The goal is to find insights and context about the corpus.

Further downstream tasks, more pertaining to application areas, could be emotion detection, sentiment analysis or text summarization. Tasks such as text classification and topic modeling typically require large amounts of text for better results.

### Which are the modules available in NLTK?

NLTK's architecture is modular. Functionality is organized into sub-packages and modules. NLTK is used for its simplicity, consistency and extensibility of its modules and functions. It's better explained in the tabular list of modules.

A complete module index is available as part of NLTK documentation.

### How is NLTK package split into sub-packages and modules?

NLTK is divided into different sub-packages and modules for text analysis using various methods. Figure depicts an example of `text` sub-package and the modules within it. Each module fulfils a specific function.

### Which are the natural languages supported in NLTK?

Languages supported by NLTK depends on the task being implemented. For **stemming**, we have RSLPStemmer (Portuguese), ISRIStemmer (Arabic), and SnowballStemmer (Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish).

For **sentence tokenization**, PunktSentenceTokenizer is capable of multilingual processing.

**Stopwords** are also available in multiple languages. After importing `stopwords`, we can obtain a list of languages by running `print(stopwords.fileids())`.

Although most **taggers** in NLTK support only English, `nltk.tag.stanford` module allows us to use StanfordPOSTagger, which has multilingual support. This is only an interface to the Stanford tagger, which must be running on the machine.

### What datasets are available in NLTK for practice?

NLTK corpus is a natural dump for all kinds of NLP datasets that can be used for practice or maybe combined for generating models. For example, to import the Inaugural Address address, the statement to execute is `from nltk.corpus import inaugural`.

Out of dozens of corpora, some popular ones are Brown, Name Genders, Penn Treebank, and Inaugural Address. NLTK makes it easy to read a corpus via the package `nltk.corpus`. This package has a reader object for each corpus.

### What are the disadvantages or limitations of NLTK?

It's been mentioned that NLTK is "a complicated solution with a harsh learning curve and a maze of internal limitations".

For sentence tokenization, NLTK doesn't apply semantic analysis. Unlike *Gensim*, NLTK lacks neural network models or word embeddings.

NLTK is slow, whereas *spaCy* is said to be the fastest alternative. In fact, since NLTK was created educational purpose, optimized runtime performance was never a goal. However, it's possible to speed up execution using Python's `multiprocessing` module.

Matthew Honnibal, the creator of *spaCy*, noted that NTLK has lots of modules but very few (tokenization, stemming, visualization) are actually useful. Often NLTK has wrappers to external libraries and this leads to slow execution. The POS tagger was terrible, until Honnibal's averaged perceptron tagger was merged into NLTK in September 2015.

In general, NLP is evolving so fast that maintainers need to curate often and throw away old things.

### For beginners, what are some useful resources to learn NLTK?

The official website includes documentation, Wiki, and index of all modules. There are Google Groups for users and developers.

For basic usage of NLTK, you can read a tutorial by Bill Chambers. This also shows some text classification examples using *Scikit-learn*. Another basic tutorial from Harry Howard includes examples from *Pattern* library as well.

Often specific processing is implemented in external libraries. Benjamin Bengfort shows in a blog post how to call CoreNLP from inside NLTK for syntactic parsing.

There's a handy cheat sheet by murenei. Another one from 2017 is published at Northwestern University.

A list of recommended NLTK books appears on BookAuthority. You can start by reading Natural Language Processing with Python (Bird et al. 2009). Those who wish to learn via videos can look up a playlist of 21 videos from sentdex.

## Milestones

Jul
2001

The first downloadable version of NLTK appears on SourceForge. Created at the University of Pennsylvania, the aim is to have a set of open source software, tutorials and problem sets to aid the teaching of computational linguistics. Before NLTK, a project might require students to learn multiple programming languages and toolkits. Lack of visualizations also made it difficult to have class demonstrations. NLTK is meant to solve these problems.

Jul
2005

**NLTK-Lite 0.1** is released. Steven Bird, one of the creators of NLTK, explains that NLTK 1.4 introduced Python's dictionary-based architecture for storing tokens. This created overhead for programmers. With NLTK-Lite, programmers can use simpler data structures. For better performance, iterators are used instead of lists. Taggers use backoff by default. Method names are shorter. Since then, regular releases are made till NLTK-Lite 0.9 in October 2007. NLTK-Lite eventually becomes NLTK.

Apr
2008

Two NLTK projects are accepted for **Google Summer of Code**: dependency parsing and natural language generation. The dependency parser becomes part of NLTK version 0.9.6 (December 2008).

Jun
2009

Book titled *Natural Language Processing with Python* by Bird et al. is published by O'Reilly Media. Since October 2013, the authors release online revised versions of the book updated for Python 3 and NLTK 3.

Apr
2011

**Version 2.0.1rc1** becomes the first release available via GitHub, although till July 2014 releases are also made via SourceForge.

Jul
2013

Over a five-year period from January 2008 to July 2013, NLTK gets more than half a million downloads. This excludes downloads via GitHub.

Sep
2014

**NLTK 3.0.0** is released, making this the first stable release supporting Python 3. Alpha release, version 3.0a0 (alpha), supporting Python 3 can be traced to January 2013.

Natural Language Toolkit